I use handbrake and I set it at 1500 and 416 x 190 give or take 10-20 each way.
Here is a little trick I discovered that has made video playback seemless.
I have dual boot with PB Easy Dual boot and I put a 1gb swap partition. Now I can leave my wifi, bluetooth, weather app, and email all on while running a video on my N800.