well the function in flame is as easy as in after effects, you only need to understand the basics
of compositing. you only need to premultiply your image with your alfa before you use the key.
flame has no premultiply button, so what you need to do is use logic ops and multiply.
premultiply is exactly what it says first multiply it with your alfa before you key it.

the fun thing is, that in your case it will not work coz the shaddow is not matching your alpha channel. where ever you are on your alf the result of alpha plus front should be 1 in luma.
when you check your values it is not. it is slightly off. thats why you see this halo.
smoke/flame/nuke or shake are doing everything correctly when they display a halo.
i cant try in after effects coz i dont care about ae, but i ve tried in all the other software packages.

so as usual - if there is a problem with flame it is mostly sitting in front of it.
;-)