I bought PocketDVD as well, the best of a bunch of software for doing essentially the same thing. Having said that, it is a pretty good piece of software and excellent value for Money.
Have you had a chance to try it yet? If so, do the settings go down to the recommended spec for playback?
From my experimentation it appears that the 770 does not support resolutions beyond 352x208 - higher horizontal resolutions are possible, eg. 360x208 - but any vertical resolution greater than 208 will result in "unsupported format" errors on the 770 when trying to play the video. 352x208 is the closest match to the aspect ratio of the 770 screen.
DVDCatalyst allows you to override the resolution so you can try your own experimentation and determine the upper limits but from my playing around it appears that 352x208 is the best we can achieve with the current player.
The author of DVDCatalsyst has added specific support for the Nokia 770 which will default to 352x208 resolution - select "NOKIA" as the output format and "NOKIA770" for "Maximum Screensize" and away you go...
When i get my hands on one for an hour or two, i can debug it a bit to see why a osx version I made is not working on mac. I whould be able to make the modifications thereafter quite easily to get the mac version up and running. Alas, I'm still home trying to get rid of a common cold, so can't go anywhere (like to the office) to do it. I hope I'm in a better shape tomorrow.