BB5 security is implemented inside the OMAP rom bootloader(and some parts on the CMT).
A few years back, when i was in the unlocking phone business, i think i saw some gsmcity thread talking about flashing unsigned symbian images, but it required a expensive BB5 unlocking/flashing box... But i don't know if that technique works with the latest BB5 SL3 security...

Anyway, something like HaRet(which is used to boot a linux kernel on windows mobile) which is basically a tool that loads linux kernel into memory under an already running OS and then jumps into kernel entry point, should work, however you have to use some tool like HelloOX2(i dont know if it works on n8) to give your app "manufacturer access" to symbian API...