Because it doesn't have the horsepower or bandwidth to handle videos of that size (besides, how much content do you actually have that's over 800x480?). Use mplayer if you want good codec support and good performance. There's really not a good reason to port VLC over mplayer right now, as mplayer has already been incredibly, painstakingly optimized for the N800 and the 770 by our own Serge.