Magic is pretty much it, for me, although imaginary races will do it too. There's a Finnish novel translated as Not Before Sundown and Troll: A Love Story that had no fantasy element aside from including trolls in the world - classes as animals - which I instantly recognized as fantasy, but that might have something to do with the fact that trolls are still a fantasy trope. (Great book to read if you're Finnish, btw; not sure how well it translates.)

There are novels that have a certain artsy feel to them that I get the feeling wouldn't want to be classed as fantasy but still have a fantasy element in it. I usually class them as fantasy only after I've had a think about it. A certain style doesn't really give the author allowance to say, "it's not just a fantasy novel" as if there was something wrong with being a fantasy novel.

Not really sure how to class Alternate Universe novels... There I do think it depends on the style. If they use magic to open portals between worlds (His Dark Materials) I call it fantasy; if they use any pretension of science to do so (Star Trek) I call it scifi. If they don't travel through the worlds but it's set in a slightly different world without magic, then I'm just confused.