I had the same experience, but I did not exclude the mupdf binary from the control file, because someone might find it useful anyway.
Because the mupdf developpers wanted to keep it small and portable (I guess) they didn't use any major toolkit, so modifying/patching the GUI is not a very pleasant job. That's why I wrote my own frontend using qt.

Flatboat now supports previous/next page (or scroll down/next column in column mode) via right/down/volume-, but I'll add keys for zoom and rotation.