Striping swap to increase performance under memory contention - maemo.org

nightfire | # 1 | 2011-02-18, 18:18 | Report

If anyone's interested, I've been striping my swap across SD and internal storage for the last few days, and it's increased performance under high memory contention.

How's it work? It simply distributes the writes (and sometimes reads) across the two storage areas. In theory, this should increase (possibly double) the available bandwidth.

Implementation would be trivial, except that busybox's swapon command doesn't support priority assignment. So, you need the proper, full utility.

I've attached it to this thread; if you'd prefer, you can grab it from the Debian repository here:

http://http.us.debian.org/debian/poo....2-9_armel.deb

.. and unpack it with:

Code:

dpkg-deb -x mount_2.17.2-9_armel.deb .

You're looking for sbin/swapon.

Now, it depends on version info from libblkid.so.1 (inexplicably missing):

Code:

1 root@glamb-n900 [~]# /sbin/deb/swapon
/sbin/deb/swapon: /lib/libblkid.so.1: no version information available (required by /sbin/deb/swapon)
/sbin/deb/swapon: /lib/libblkid.so.1: no version information available (required by /sbin/deb/swapon)
/sbin/deb/swapon: /lib/libblkid.so.1: no version information available (required by /sbin/deb/swapon)
(...)

.. but you can ignore the warning. It works fine.

I deployed mine to /sbin/deb/swapon.

If you're not sure how to make a swap partition on your SD card, see this thread:

Using Micro SD Card as Virtual Ram on Nokia N900?

If you want to test it, as root execute something like:

Code:

/sbin/deb/swapon -p 1 /dev/mmcblk1p2
/sbin/swapoff /dev/mmcblk0p3
/sbin/deb/swapon -p 1 /dev/mmcblk0p3

That will add the SD swap area, remove the onboard swap, and then re-add it with the right priority. Of course, you'll have to change /dev/mmcblk1p2 to whatever your SD swap partition is.

If you want to make the changes permanent, it's just a matter of updating your /etc/event.d/rcS-late. Look for the swapon line, and change it to something like:

Code:

/sbin/deb/swapon -p 1 /dev/mmcblk0p3
/sbin/deb/swapon -p 1 /dev/mmcblk1p2

.. again, setting /dev/mmcblk1p2 to whatever your SD swap area is.

Of course... standard warning: it's very easy to "brick" (not really brick.. but require a re-flash) your phone by breaking the boot-up process, so be careful! If you don't feel comfortable updating boot scripts, it's safer to just put the test commands into a script, and execute it each time you reboot.

Update

I wrote a quick program to allocate 225mb and fill it with junk; here are some performance numbers, after running "crud" 10 times each (+/- 1 second).

Striped swap areas:

Code:

29 root@glamb-n900 [~]# time ./crud

real    0m5.852s
user    0m3.641s
sys     0m0.586s

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 1  1 131664   4160     88  12828    0 2752     0  2752 1109  547 45  9  0 46
 0  2 150052   4040     88  12828 3896 18468  3896 18468 4601  753 36 18  0 45
 1  1 168208   4940     88  12828  748 18156   748 18156 4811  822 29 23  0 49
 1  1 187480   4940     88  12828    0 19280     0 19280 4760  675 35 19  0 47
 0  0 132456 181124     88  12836    0   36     0    36  463  734  3 11 86  0

Internal swap only:

Code:

39 root@glamb-n900 [~]# time ./crud

real    0m10.014s
user    0m3.500s
sys     0m0.672s

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 1  1 137756   4868     88  12928    0 4612     0  4612  823  520 31  9  0 60
 3  3 151156   4328     88  12928 2664 13400  2664 13400 1865  901 18 14  0 68
 0  4 156600   3068     88  12928 5932 5444  5932  5444 1533 1041  0  6  0 94
 0  2 170648   4868     88  12928 1536 14048  1536 14048 1142  894 19 13  0 68
 0  2 188184   4088     88  12928    0 17536     0 17536  793  709 35 14  0 51
 0  0 133488 177632     88  12928  392 4780   392  4780  410  822  8 13 59 19
 1  0 133416 151988     88  12928 1024    0  1024     0  614  960 46  3 48  4

Not a scientific test by any means, but the swap I/O bandwidth is clearly improved.

Attached Files

swapon.gz (9.2 KB, 172 views)

Last edited by nightfire; 2011-02-18 at 22:47.

The Following 14 Users Say Thank You to nightfire For This Useful Post:
bobbydoedoe, D4rKlar, dhanurdhar, drangina, ericsson, gabby131, handaxe, humble, lma, Mentalist Traceur, noipv4, Tiboric, Tigerite, Trestry

lma | # 2 | 2011-02-18, 18:52 | Report

Reading between the lines a bit, but I assume the mount package is just for "swapon -p" support? If so then http://git.busybox.net/busybox/commi...978b0864699419 might be of interest for the Fremantle CSSU (I'm backporting it to the Diablo one).

nightfire | # 3 | 2011-02-18, 18:55 | Report

Originally Posted by lma View Post

Reading between the lines a bit, but I assume the mount package is just for "swapon -p" support? If so then http://git.busybox.net/busybox/commi...978b0864699419 might be of interest for the Fremantle CSSU (I'm backporting it to the Diablo one).

Yup. I saw that patch and was a little miffed we didn't have it.

nightfire | # 4 | 2011-02-18, 21:58 | Report

Added some performance numbers to the first post.

atilla | # 5 | 2011-02-18, 22:44 | Report

there is already a thread about this : http://talk.maemo.org/showthread.php...highlight=swap

and most of us know that adding more swap to a sd cards brings absolutely nothing.
another quote:
quote by TA-t3:

Be careful with terminology, folks.

o RAM: RAM is always physical RAM. In the N900 there's 256MB of it.
o There's no such thing as 'virtual RAM'.
o Swap space: Resides in the built-in flash storage (there's 768MB of it on the N900), or could be set up on microSD as well).
o Virtual memory: This is (on Linux) the sum of swap and RAM. On the N900 it's 1GB (768+256).

Virtual memory is what the applications can potentially use (not considering some limitations added by the kernel). Virtual memory is the only thing applications know about. If you have no swap space allocated, only the 256MB physical RAM, then your virtual memory size would be 256MB. There's no such thing as 'not having virtual memory set up'. You mean _swap_ here.

o Swap space is always slower than RAM. Maybe in the future this could change, when/if some of the stuff being worked on in the labs manifests itself. Then maybe we'll have RAM, SD cards, USB sticks, other mass storage, all built from the same magical, non-volatile, super-fast stuff.

o Adding swap will not make for a faster device, only for a device which can sustain more applications running at the same time, or using more data, without going down in flames. There was a myth, for a while, that the N800 and N810 would be faster if you enabled swap space. This was only in the imagination - I tested this. And there's no rational reason it should be faster, except in very carefully thought out scenarios that won't happen with normal use.

o Adding excessive amounts of swap _will_ make your device slower, eventually. Particularly if you keep your applications (e.g. browser) running at all times.
__________________

nightfire | # 6 | 2011-02-18, 22:50 | Report

Originally Posted by atilla View Post

there is already a thread about this : http://talk.maemo.org/showthread.php...highlight=swap

and most of us know that adding more swap to a sd cards brings absolutely nothing.
another quote:
quote by TA-t3:

Be careful with terminology, folks.

o RAM: RAM is always physical RAM. In the N900 there's 256MB of it.
o There's no such thing as 'virtual RAM'.
o Swap space: Resides in the built-in flash storage (there's 768MB of it on the N900), or could be set up on microSD as well).
o Virtual memory: This is (on Linux) the sum of swap and RAM. On the N900 it's 1GB (768+256).

Virtual memory is what the applications can potentially use (not considering some limitations added by the kernel). Virtual memory is the only thing applications know about. If you have no swap space allocated, only the 256MB physical RAM, then your virtual memory size would be 256MB. There's no such thing as 'not having virtual memory set up'. You mean _swap_ here.

o Swap space is always slower than RAM. Maybe in the future this could change, when/if some of the stuff being worked on in the labs manifests itself. Then maybe we'll have RAM, SD cards, USB sticks, other mass storage, all built from the same magical, non-volatile, super-fast stuff.

o Adding swap will not make for a faster device, only for a device which can sustain more applications running at the same time, or using more data, without going down in flames. There was a myth, for a while, that the N800 and N810 would be faster if you enabled swap space. This was only in the imagination - I tested this. And there's no rational reason it should be faster, except in very carefully thought out scenarios that won't happen with normal use.

o Adding excessive amounts of swap _will_ make your device slower, eventually. Particularly if you keep your applications (e.g. browser) running at all times.
__________________

My post is not about adding more swap, it's about adding more swap bandwidth.

Last edited by nightfire; 2011-02-18 at 22:53.

The Following 7 Users Say Thank You to nightfire For This Useful Post:
drangina, ericsson, MartinK, noipv4, RobbieThe1st, Switch_, Tiboric

Tigerite | # 7 | 2011-02-21, 10:41 | Report

Thanks for sharing the swapon.gz, I've been trying to get the priority option to work with no success (even went to the trouble of downloading the Easy Debian ext2 and mounting it, but that one didn't have a -p option either).

However, I find it easier (and probably less risky, as it's possible that the SD card isn't ready when rcS-late is run) to edit /etc/event.d/rcS-late to use /sbin/deb/swapon -a, i.e.

Code:

 /sbin/swapon -a || echo "Failed to enable paging partition."

(NB, /sbin/ might not be there, I'm working from memory as I've already changed it) to:

Code:

 /sbin/deb/swapon -a || echo "Failed to enable paging partition."

..and then edit /usr/lib/genfstab.awk so that the swap partition is mounted with pri=1, i.e.

Code:

start == 1 && $6 == 82 {
 printf "%s none swap sw 0 0 \n", $1
}

becomes

Code:

start == 1 && $6 == 82 {
 printf "%s none swap sw,pri=1 0 0 \n", $1
}

.. then /sbin/deb/swapon -p 1 /dev/mmcblk1p2 (or whichever partition the swap is on) in a custom event.d script which is run at a later time.

I've been experimenting with compcache, using disksize of 128M (rather than memlimit) and then setting the ramzswap priority to 2 (so it is used first), leaving the striped file-based paging file for when it is used up. Unfortunately I think the swap free notify patch that I modified to work with Nokia's swap_remap isn't completely right, as I'm getting lots of "BUG: scheduling while atomic: kswapd0/15/0x00000000" in dmesg - unless that is a direct result of the striping, of course..

shadowjk | # 8 | 2011-02-22, 15:38 | Report

I myself use swap on microsd only, on the theory that offloading swap from emmc completely is better as the emmc is loaded with 2 other intensive tasks: demand paging apps and hosting the various databases.

The databases are often sqlite, which has dreadful performance on ext3 on flash.

On the topic of swap size, bigger swap actually makes the device faster. Some of you might have noticed that after a certain amount of time, 2-3 days (or a few hours if you use fennec), the N900 performance drops. Stutter and jitter increases. I tracked down one issue to swap fragmentaion. Basically it works like this: swap writing algorithm picks largest consecutive free are in swap and writes to it sequentially. This is good. However, after some time, when apps open and close and modify memory of their that they had in swap, the largest free area becomes progressively smaller and smaller. This means smaller and smaller stretches of sequential writes. Our emmc and usd storage really hates small writes. If you write 4k in one place, and 4k somewhere else, the emmc/usd has to internally read 256k and write back 256k, with 4k of that modified. This is a slowdown of factor 64.
Bigger swap makes it take longer before swap fragmentation starts affecting performance.

Myself I tporarily switch swap back to emmc, and then to usd again when swap fragmentation becomes an issue.

The Following 3 Users Say Thank You to shadowjk For This Useful Post:
mehulrajput, nightfire, Tigerite

nightfire | # 9 | 2011-02-25, 03:23 | Report

Originally Posted by Tigerite View Post

Thanks for sharing the swapon.gz, I've been trying to get the priority option to work with no success (even went to the trouble of downloading the Easy Debian ext2 and mounting it, but that one didn't have a -p option either).

However, I find it easier (and probably less risky, as it's possible that the SD card isn't ready when rcS-late is run) to edit /etc/event.d/rcS-late to use /sbin/deb/swapon -a, i.e.

Code:

 /sbin/swapon -a || echo "Failed to enable paging partition."

(NB, /sbin/ might not be there, I'm working from memory as I've already changed it) to:

Code:

 /sbin/deb/swapon -a || echo "Failed to enable paging partition."

..and then edit /usr/lib/genfstab.awk so that the swap partition is mounted with pri=1, i.e.

Code:

start == 1 && $6 == 82 {
 printf "%s none swap sw 0 0 \n", $1
}

becomes

Code:

start == 1 && $6 == 82 {
 printf "%s none swap sw,pri=1 0 0 \n", $1
}

.. then /sbin/deb/swapon -p 1 /dev/mmcblk1p2 (or whichever partition the swap is on) in a custom event.d script which is run at a later time.

I've been experimenting with compcache, using disksize of 128M (rather than memlimit) and then setting the ramzswap priority to 2 (so it is used first), leaving the striped file-based paging file for when it is used up. Unfortunately I think the swap free notify patch that I modified to work with Nokia's swap_remap isn't completely right, as I'm getting lots of "BUG: scheduling while atomic: kswapd0/15/0x00000000" in dmesg - unless that is a direct result of the striping, of course..

I actually just read about ramzswap and I'm trying it on my laptop (512mb of 2gb)... neat idea.

Where'd you get the n900 kernel module?

Tigerite | # 10 | 2011-02-25, 16:44 | Report

I built it myself, having compiled the BFS kernel first, and used the patch for 2.6.29 (found in the older compcache zip) after modifying it. However it's unstable, because Nokia patched the life out of swapfile.c - they are using swap_remap, I believe to try to find the largest gap of empty pages before writing any out, but ramzswap just doesn't seem to like it much. I haven't really had time to research fully as to why that is..

The Following User Says Thank You to Tigerite For This Useful Post:
nightfire