A while ago I wrote a post about how I ended up with Savage Worlds. The post got some critique for me judging games just by reading rules; it's true, you can't tell how a game plays without actually trying it. But there are other factors that may matter when choosing a game system than just how it plays.
New system for old players
These were all things I had to consider. I wanted to find one system I like, which is fun and easy for the GM and would still have solid rules system and preferably supported use of miniatures. I wanted a game that I could easily use for random one-shots, some mostly improvised, and I wanted good support for various genres and game worlds.
My players tend to share a dice pool and everyone is looking for the dice they want once in a while, and I don't think buying new dice would help this. They tend to find any system flaws and exploit them, and character generation tends to take a good amount of time. While I hadn't actually tested Cortex, mainly these facts that closed that system (using RAW) out.
All groups have some distinctive features, that will affect how a system will be used and how will it play in practice. Think about your group, what kind of gaming habits do they have?
Similar systems or mechanics
I first tried to venture away from D&D by using Iron Heroes, Star Wars, Wheel of Time and True20, which were d20 games so my players could jump in easily. But when 4E was announced I felt relief as D20 system has many cumbersome special rules, and after getting into it I felt there was no going back to 3E -based games. But as I wrote before, 4E wasn't a dream come true either, even if we got (and still get) a good amount of fun from it.
Trying all those d20 games was an easy solution at the time. By being very familiar with the core system allowed to get pretty good picture about how the game plays, and there was little learning. Other game systems may also have rules or features similar to a game you know; this helps you understand how the rules work, but be aware that it is always possible that the rule in question interacts differently with different core rules.
Matter of tastes
Tastes can't be ignored. Sometimes you just don't like something, and you can't get over it. If you just don't like it on paper then try it - it's always possible that it actually works in the game. Good example of tastes getting on the way is when I tried D20. Rolling different dice is something I really missed in it, and realized I really want a game where different dice can be used. I was ok with those dice being used just for damage, but ended up using different dice for all rolls.
Some things work well on paper, but not in practice
This isn't really surprising. Have you ever tried to write your own rpg system and had an idea of a great rule? Often this kind of innovations won't work. The idea may get so stuck in your mind that you become obsessed of it's goodness. Or perhaps the rule actually is good, but it may be in conflict with the rest of the system, and therefore should be discarded. And one of the hardest things in game design is being able to let go of some ideas you think are great. If conflicting rules are left in the game, they may do the game a lot of harm.
Of course non-working rules may be personal issues - people's brains work in different ways; generally a rule should work for a majority of people.
As a personal example of rules that don't work in practice is True20's damage system. I liked it on paper; you have static damage from weapon (plus strength in case of melee), and the target makes a toughness save after being hit; How well or badly this save goes tells how much damage you get - a scratch, wound, serious wound or worse. The system has it's fans, and I think this is a matter of tastes too - there were damage house rulings on the forum though, so I'm not the only one who didn't like the damage in practice.
Some things work well in practice, but not on paper.
Many rules may sound strange, especially if you're used to a specific formula how rules should work. In well-designed games, rules have been iterated repeatedly to get a streamlined result that takes most things into account, one way or another. When looking at a rule for the first time, it may look really strange. I realized Savage Worlds was full of this kind of small details - something that couldn't be told at first sight.
Mongoose Traveller had a few things that look odd on the paper; especially strange to me was auto-fire rules. You roll amount of dice and then pair them to get attack rolls you can assign to wanted targets. In addition to generic 'oddity' feel, the problem I saw with this is that it can be a bit of a bottleneck in practice, with players trying to think which number is enough to hit which target. The rule may simulate concentrating the spray more on some target(s), and I'm curious how it works in practice.
I also felt it's strange in MT to give difficulty modifiers for easy or difficult tasks, and always trying to get 8 or more - assigning difficulty as target number would remove need for one operation which would make playing faster.
If you have experience with MT I'd be curious to hear how these rules work in practice, how they feel. If you think I'm wrong with my doubts go ahead and prove me wrong!