maacruz | # 1 | 2011-01-11, 00:07 | Report

This thead is deprecated. Please go here: http://talk.maemo.org/showthread.php?p=944241

Hi all!

What do I mean with that thread title?
During this Christmas holidays, I've been working on porting what a I could of the latest improvements in the linux kernel with respect to desktops, commented in Phoronix on the second half of this year. The success, beyond my expectations, in the results I obtained, pushed me to try something on the same line for my little N810.

In the past, there have been some attempts at improving the user experience in the N8x0 by turning some sysctl knobs in the kernel, without much success.
So, what could be done with a such old kernel? Attempts by others to get a newer kernel seem to have failed, may be because it is too much work due to the big divergence between kernel releases, the need to port modifications made by Nokia in the Diablo kernel needed by the dammed closed source propietary system utilities, and the dammed closed source propietary kernel modules in the wifi stack.

After a first look at it I noticed the following:
- Kernel 2.6.21 uses the old O(1) scheduler, which was replaced in 2.6.23 with the CFS for performance reasons (there was a notable controversy).
- It is built with a 128 HZ system timer (not very interactive)
- It is built without preemption (even less interactive - there were a few attempts at building a preempted kernel by a few community members, but seem to have been quickly abandoned).
and of course, the problem that the mmc flash is a slow medium for a swap file/partition, already solved in part with the ramzswap (ramzez_init package) modules.

I wanted to try to apply Con Kolivas BFS, but the kernels are too far apart and that was impossible for me.
Then, I found that Ingo had published a patch to apply the CFS scheduler in the 2.6.21 kernel. Also, Con Kolivas had a set of interactivity (desktop) related patches with his own scheduler SD.

So, I have decided to appropriate the name "DiabloTurbo" from the other thread, to name this effort to improve the N8x0 responsivity.

What have I done so far?
I have built a set of experimental kernels and kernel modules, based on the Diablo+Community SSU kernel, with the following features/patches:

For all kernels:

• DSPOP: One of those patches floating around. When the DSP is in usage the ARM cpu is set to 333 MHZ, with this patch, you can select to set it to 400 MHZ (at the cost of underclocking the DSP, which seems to work fine otherwise). Comes handy when you need all processing power while listening audio (like watching h264 movies with maemo Media Player). It is disabled with "echo 1 >/sys/power/op_dsp" and enabled with "echo 0 >/sys/power/op_dsp"
• SWAP_PREFETCH: When the virtual memory subsystem is idle for more than 5 seconds, it will start copying pages from swap into the swapcache while keeping the copy in swap. This means the following: actually, when you use a memory hog, much of the system is swapped out. If you close the memory hog and leave the tablet for a while, when you come back to use it again it will be sluggish because it has to move the pages to memory back from swap. With this feature, when you use the tablet again, it will be very snappy because the needed pages are already in ram, and if what you use is a memory hog, it will be snappy because the pages are already in swap.
• All other memory management improvements from Con Kolivas patches
• Swap notifier for ramzswap (see below)

Depending on the kernel:

• Old O(1) scheduler
• Plain CFS
• CFS with User Grouping
• SD scheduler
• Preemption
• 256 HZ timer

I also tried to port the CGROUPS wonderful feature for CFS from the 2.6.24 kernel, but, not being a kernel hacker myself, I only achieved a very partial success: the kernel builds but it doesn't boot. Unfortunately, it is a lot of work and my time is over, so I won't pursue this anymore.

The change of scheduler leaded to the need of rebuilding modules (otherwise subtle problems arise). Since I use (and like) ramzswap (compcache), and ramzez has no sources available for the modules, I got the latest compcache (0.5.4) sources for old kernels from its homepage, fixed the sources so they can be built, and then went on and backported the backing swap to file from the 0.6.2 sources (only to be disappointed later because this feature limits the size of the ramzswap device to the backing swap size), and applied to the kernel the swap notifier patch included with compcache. This patch allows ramzswap to free memory again when it is not needed anymore (ramzez modules won't free much memory, so once it grows to its maximum size, although you close the programs consuming memory, it won't shrink).

Since I was rebuilding modules, I also applied the WPA-EAP fix for the cx3110x modules, too.

To make the testing easier, I provide you with a installation script included in the package. This script will make a backup of your current kernel and modules before installing the selected one from the package, and allows you to restore the backup. It will also detect if you have ramzswap (ramzez) installed and update its modules, until you restore the backup.

Since the package size is about 20 MB, I have designed it to be able to be run from one of the mmc cards. It contains 12 different kernel configurations.
Untar the package where you want, and run the script as "sh install.sh" to get help.
For example, to install the kernel with CFS scheduler with user grouping, preempted and 256 HZ timer:
While I have taken every step I could to make sure this is a safe operation, I make no claims regarding the safety.

DISCLAIMER: This is experimental software and may crash your device, burn your house and kill your dog. You may have to end reflashing your device. In particular, extra modules not provided in the package inserted with boot scripts can crash at boot rendering your device unbootable (until you boot in an alternate system to fix it or reflash kernel and initfs). If you have any such modules, I recommend disabling their load at boot and loading them manually and testing them thoroughly before reenabling them.

If you are afraid of having to reflash and/or reinstall or do not know how to do it, then this is not for you.
If you do not understand what I've been talking about, this is not for you.
If you do not know how to use the command line, this is not for you.
If you think you can test this, but you prefer to be on the safe side, you should do a full backup (see 0xFFFF - and here is a binary for N8x0 0xFFFF.static - to get flashable image backups) .

I've done all the hard work, now it is your turn, to test these kernels and choose among all of us the best one.
Once we have a winner, and if it isn't the stock kernel, I intend to prepare a proper deb packaging for extras-devel.

To help building synthetic tests I've uploaded to extras-devel the command line program stress. You can use it to create cpu, disk and memory hogs to stress the system and see how it responds. I recommend using the "-t" option to avoid hanging your device (this options sets a time limit for stress to run).
You may try all kernels, but it's better if you choose one and test it for long. Take note of any problem that arises and, if possible, try to reproduce it and double check it doesn't happen with the stock kernel.
It is a good idea to have ksyslogd enabled, so in case of any crash we can know where it comes from.
There are differences among the three schedulers and the concept of "fairness" and the interpretation of priority. The stock configuration of users (see below) and priorities may not be the best for a different scheduler.
These kernels have been built for improving interactivity and responsivity, more ticks and preemption mean more context changes, so more overhead, so less throughput.

One of the best improvements in my desktop came from using CGROUPS to do the same that the famous 200 lines kernel patch (search in Phoronix) does but with just 4 lines bash.
CFS groups allow to manage how much cpu a group of processes get in relation with other, much much better than with priorities. CGROUPS are very flexible, but anyway with User Grouping the same results could be achieved by using different users to run the programs (already some programs are run as different users: root, messagebus, daemon, haldaemon, systemui, and nobody).
You can change the cpu shares a user has using "echo #cpushares > /sys/kernel/uids/#uid/cpu_share" where #cpushares is the number of shares this user will get (so that %cpu_max_with_all_users_competing_for_cpu = #cpushares/sum(#cpushares) and #uid is the user id number).

Old testing packages have been deleted, as they are no longer useful

Happy testing!

---

Last edited by maacruz; 2011-02-12 at 18:31. Reason: Move into beta stage.

Dancairo | # 2 | 2011-01-11, 00:45 | Report

This is way beyond anything I can understand at the moment but i've thanked you because you've obviously put a lot of time and effort into this, and it made for an interesting read none the less.
Dan

debernardis | # 3 | 2011-01-11, 06:50 | Report

It's very interesting, and I'll try them as soon as I come back home from vacation.
One question: how could we implement the swap_prefetch you explained above in the n900? It would be a great addition to the swappolube package, possibly. Thanks

maacruz | # 4 | 2011-01-11, 08:07 | Report

Here: http://kernel.org/pub/linux/kernel/p...k/patches/2.6/ you'll find Con's patches. Last set in the pre-2.6.30 series is 2.6.22, you'll need to port the swap-prefetch patch from there to N900 2.6.28. For such task, you may use linuxhq.org diff engine to follow changes in a given kernel file, and the linux crossreference webpages (for example http://lxr.free-electrons.com/) to search functions, and "grep -R" in the kernel sources a lot. Depending on the divergence in the vm subsystem between those two releases, the task can go from easy to almost impossible unless you are a kernel hacker.

maacruz | # 5 | 2011-01-11, 15:57 | Report

Garage is working now. The package has been uploaded and is available for download now.

gerbick | # 6 | 2011-01-11, 17:23 | Report

Any reports on how well this works for anybody? This seems pretty cool.

cstryon | # 7 | 2011-01-11, 19:36 | Report

Will test as soon as I can get my baby to sleep ^_^. Awesome work maacruz. Feels good to see Diablo work being done!

Straycat | # 8 | 2011-01-12, 12:42 | Report

News, please!!!

News about, please!

maacruz | # 9 | 2011-01-12, 14:41 | Report

What news can I give? All kernels work, that's for sure. And I think some are better.
Long term? Which's better? Problems? I don't know or I'm not sure, that's why I call for testers.
Now it's your turn!

Only three downloads!
There are only 4 people in the whole N8x0 community able/not afraid to test this???
No wonder the N8x0 is stagnating.
C'mon, you can do better!

wnd | # 10 | 2011-01-12, 16:25 | Report

I just installed and uninstalled cfs-ugroup-preemptnovermagic-256. Some extremely hasty and pessimistic observations. I think I'll try other configurations later; tonight I'm going to need a working device.

- Applications may have reacted faster to mouse events.

- Applications seem to receive mouse release events when they shouldn't. As strange as it sounds, panning in Maemo Mapper is much more difficult; half of the time panning stops after a fraction of a second. I don't remember any issues like with with SSU kernel.

- When I power on, I can see Nokia splash screen gradually replaced by Hildon desktop. I've seen this before, but it's been months and months since the last time. This seems to happen every time I boot up. In related news, boot procedure (to desktop) seems to take significantly longer than before.

- During by second and third power-on sessions, ksoftirqd/0 would consume most of the CPU time. Battery life was projected to be hour hours instead of the usual seven. This may have been related to (usually harmless) JFFS warnings about JFFS internal storage space in kernel log.

- During my third session I could not run sudo gainroot; all four tries simply stopped after I pressed enter.

- Hildon UI clearly receives less CPU time. When the screen is powered after th
e device being idle, some screen elements may be missing at first. This happens nearly every single time with battely level status bar applet.

- Installer script expects to find all (currently) installed modules in kbuilds
/foo. This means that if you have additional kernels such as those in kernel-diablo-modules-extra installed, you get very uninformative warning for every and each one of those.

- WLAN/WiFi didn't work properly. I could scan available networks, but actually connecting one failed.

---

Last edited by wnd; 2011-01-12 at 16:42.