dunno which kernel you use, but I was able to boot (almost stock)fremantle since 3.14(?). Pali's tree should have SGX driver (didn't really check, but it should be there). You need userspace bits from fremantle ofc. No idea if those will work with hardfp though.