I have looked around for a good solution to convert video for N800 without to much work.

So I started with DVDshrink to rip my DVD movie that I got. I used the "re-author" in DVDshrink to just rip the main movie.
Compression rate to max (46.1% in Xmen)
Have the 5.1 audio I want and just the subtitle I wanted.
Then I set Start-End to trim the start (don`t need all that info about fancy sound and intro with who produced the movie)
Remember to go in Preferences and make it not split Vob in 1gb size.
Stream selection could be set to audio English + AC3 or LPCM and Subpicture language to your prefered language.

When DVDshrink is finished I got a file VTS_01_1.VOB that is the main movie. the size is 2,4gb.
Then I use Handbrake.

I used the preset for Iphone/Ipod touch and set the width/hight to 480x240, encoder to MP4, Audio to AAC, Framerate 25, Audio/subtitle to what I want, track mix to auto.

I ended up with a file that is called M4v that I renamed to MP4 to make it recognize it in N800 and started it in the default Media Player.
It started with a tad lag in the intro but after a second or so it went to normal speed without lagging.
I had the wifi enabled, filemanager open and the default jabber client enabled when I tested the movie.

uhm.. After trying to actually watching the movie I noticed more lagging.. So i need to tweak the settings in handbrake to make it more smooth.

After checking the default AVI commercial video on the N800 I found out these settings that I am now trying on the Handbrake.. i tweaked it first with MP4 with success but since the original AVI N800 video looked so good I am trying with this also.

Hi

Quit making things hard for yourself

http://xoomer.alice.it/sepaolo/n800vc/

The answer is very easy: MPEG4 video at 400x240 (or best match for the aspect ratio of the source) and no more than 1500Kbps, and MP3 audio at around 128Kbps.