I've not tried it myself, but the OS2007 Hacker Edition probably gives you more of the "thumb control" features of the 800 (large menus when you use a finger, etc.) Beware that you'll lose some existing functionality (e.g. video codec support).
Some individual apps are built to stylus-free, such as the Canola media player. The Quiver image viewer (recently mentioned here) is pretty cool to play with, too, if you just want to see a smooth, clean UI (although it does include the standard tiny toolbar).
The iphone has a hell of a lot more software running it as well as nearly 3x the processor power- there's no way you're going to get its UI and finger inputs (which is where a part of all that software and extra horsepower come in) in the 770-ever.
With that said, you're never going to install applications on the iPhone- unless apple does a reversal on it's current stand.
They're two different products aimed at two different markets with an overlap of a few features.
I do think(hope?) that in time there will be at least one truly open source solution for the 770's Operating System(nothing iPhone-ish, though) While I don't expect Nokia necessarily to give the community total freedom in developing the OS further, I can't see how it'd hurt them to open it up quite a bit more than it is right now.
I also have these crazy dreams that sooner or later someone with an approximate ton-load of coding knowledge will take an interest to break open just exactly what it is that needs to be done to work out an entirely new system for the 770.
