This will not work as some drivers have not been ported upstream and are available only in the 2.6.28 kernel-source.deb file.
Download it, extract the source and use this as base for your changes.

BTW: you need to use a cross-compiler for arm.