Thanks! Just converted some videos to "x264 baseline profile and 6 channel ac3 audio" and they work fine on N9. Just I found command line ffmpeg to difficult to use and used WinFF user interface: http://winff.org/html_new/downloads.html where I added custom preset as follows:

Preset Name: NokiaN9
Preset Label: Nokia N9
Preset command line parameters: $ -s 854x480 -r 25 -g 25 -vcodec libx264 -vb 1536k -vprofile baseline -acodec ac3 -ab 192k -ac 6
Output file extension: mp4