My current settings (so far) for generating h264/avc videos that works pretty well.

I use 2-pass encoding.

Profile: Level 1.2 Baseline (No CABAC, No BFrames, Ref Frames=2, Mixed R-Frames)
(Note: If you are still having choppy video, you could try 1 Ref Frame with no mixed R-frames to trade image quality for faster decoding speed)

Resolution: For HD I use 400:224, for SD I use 360x240 (I crop to maintain AR)

Frame Rate: 24 fps (For 30 fps source, TDecimate to 24 fps)

Video Bitrate: Ave 128 Kbps

Audio: 64kb/s HE-AAC or AAC-LC (Can't get mp3 to play inside .mp4 container, otherwise, I think I can play 128kb/s mp3 audio with video, as mp3 is far simpler to decode)

On vixy.net/rawvideo you can download Youtube videos without converting at all. They come in H.264 format+AAC for audio, with resolution mostly 320x240 and bitrate ~320kbps. All you need then is to rename the file from getvideo.php to *.mp4. The quality is way better compared to "old" FLV videos. However, some are a bit choppy in default Mediaplayer (but playable anyway), and completely not watchable in Mplayer (though it tries to start playback). For now it's a comfortable method to get Youtube videos on N8*0 without additional players.

I have no problem converting videos for n800 on my mac. Used QTP for a while now switched to Visual Hub, nice, quick and relatively flexible. The quality is somewhat better than what Nokia PC converter generates, the advantage is much much smaller files. For example I'm watching now "No country for old men" 30fps, 380kbps h264, 400x174, AAC stereo 128k, 2+ hours, 450MB and it plays fine. Much nicer experience watching it here than on iPod Video. Don't know about Touch. Converting it further down to 24fps would obviously be a better option.

IME generally 380kbs is the max n800 will handle with 400h resolution. Minor note to Canola2 people, some of my h264 files will not play with your player, while Nokia player has no problem whatsoever. Can provide a sample on request.

Yep, the N810 will start to get choppy when the bit rate goes higher than 400Kbps. So I have modified my encoding settings again. I am reducing to 1 ref-frame and no mixed ref-frame to speed up the decoding (on the N810) a little bit more, the quality hasn't suffer too much, so it's worth it. Audio is changed down to AAC-LC to speed up decoding as well.

Video:
x264 2-pass encoding

Framerate = 23.976 fps (I use avisynth TDecimate() to reducing the frame rate)

x264 settings:
Profile: Baseline Level 1.2
(No CABAC, No B-Frames, Ref Frames=1, No Mixed R-Frames)
(Partitions: i4x4,p4x4,p8x8,b8x8)

Max Bitrate = 384Kbps (try to constraint the bitrate to below 400kbps, N810 can deal with occassional peak to above 400 with this baseline profile)
Ave Bitrate = 192KBps (This will determine the final file size)

Audio:
AAC-LC 96/128 kbps

dchao,

What tool are you using to do your h.264 coversions? thanks

After further experimenting I'm not longer so sure. Here is what I think the current limits are (n800, os2008) for h264 encoded video in mp4 enclosures.

Max size limits, 400 by 240.

(that means your 3:2 video should be encoded as 360x240, while 16:9 400x224 and 2.35:1 400x176). Whatever you do make sure that both dimensions are divisible by 16, any other number dramatically degrades codec performance). To be on the safe side you could use 32 but I have not tested it.

With 16-divisable media size, I see no stutter* on n800 even up to 600kbps. n810 should be the same I told below. Also with 600kbps your file will be smaller than what the best quality in Nokia PC converter generates (800kbps), but this being h264 (as opposed to MPEG4 in Nokia converter) will give you perceivably better video, don't expect miracles though.

600kbps above is for 2.35:1, 24/25fps media, 30fps or fuller aspect ratios will likely stutter like hell at 600kbps. Just throttle it down a bit and see. 300kbps should always work fine.

----

* with non-16-divisible media Nokia codec starts stumbling already above 300kbps regardless of anything and you will see some artifacts.

Uh, why? :\

faster clock I understand.

Yeah—no.

For all intents and purposes, the guts of the N800 and the N810 are identical.

I use RipBot264, under the hood, it uses x264 encoder, ffdshow and avisynth.... Most files are included in the zip file, except ffdshow and avisynth

http://forum.doom9.org/showthread.php?t=127611