If you want to boot that device you need either a patched preinit or a patched getbootstate. If you had U-boot/recovery console/backup menu/etc installed you wouldn't be asking

So, how to patch a binary on a device that is not bootable?

I assume USB works (because you tried setting R&D mode), so you can flash it. One option would be to create a customized firmware (I always wanted to do this), with U-boot, backupmenu, recovery, patched preinit, patched getbootstate, ssh, etc. all built-in.

Also note that, assuming Pali's getbootstate matches Nokia's getbootstate behaviour, if the boot mode is set to either FLASH, LOCAL or TEST the bootstate will be set to the bootmode without even looking at BSI or R&D.

AFAIK LOCAL and TEST is effectively the same as USER (i.e. should be enough to boot), so you could use the flasher to boot with a customized kernel commandline.

Sorry no details for now (severe lack of time..) but hopefully this will get you going. Otherwise I think it's time (for me) to someday start with the patched, flashable, firmware.