Massive interactivity improvement under high I/O load!
 

Hey everyone,

I've been using my phone as a wireless media server for friends (with Samba + wifi hotspot). While it worked brilliantly for reads, writes were causing all kinds of weird issues, and absolutely destroyed interactivity on the phone itself.

Later, while copying several 400mb files from internal storage to my SD card, it actually froze, and rebooted. I suspect dsme was swapped out and couldn't respond quickly enough to avoid the watchdog.

Anyway, I sat down and thought about the problem, and set about to improve interactivity under I/O load. This is my current configuration, and I find it vastly improves responsiveness:

Quick explanation:

Generally speaking, Linux is tuned for rotating media, not flash memory. The VM and I/O schedulers try to group I/Os together in a sensible manner, to reduce disk thrashing.

Of course, it's meaningless on the n900; there is no penalty for discontinous I/O.

So what I've done is:

- Reduce the time dirty buffers and pages are allowed to remain dirty; they should be eligible for flushing almost immediately.

- Reduce the amount of VM allowed to stay dirty; I/Os block once this threshold is met.

- Increase the minimum amount of free physical memory. The kernel will always try to leave 4mb free (yes, wasted) so that large allocations can take place even under high memory pressure (without swapping). When used, the kernel prioritizes its replacement by freeing other memory (through swapping or flushing writes).

- Reduce the willingness of the system to needlessly swap. This one is somewhat contentious (no pun intended); if your phone doesn't favor maintaining cache and free memory, swap-outs can be exceptionally long, when required, to make room for things like the Phone app. However, due to the overall improvement in interactivity, I find it acceptable. I've never come close to missing a call.

- Prefer to free disk cache over program data. The onboard storage (and even microsd) is quite fast for reads, and has no rotational latency. Since we're almost always under pressure for RAM, let's prefer to keep data we know we'll need over VFS cache. Helps to prevent large files (ie. videos) from overwhelming program data.

- Set the swap page size to 256k to match the SD card or internal flash device.

- .. and finally:

The most important change was reducing (essentially eliminating) the device I/O queues. The main purpose of I/O queueing is to allow the system to make intelligent read/write decisions based on a large number of elements requested (nearly) concurrently, such as contiguous streaming of data collected across multiple writes.

This doesn't really provide any benefit for us though! There is no additional latency associated with discontinuous writes, so why let the I/O queues build up?

Reducing them to 4 causes I/Os to block, rather than queue. This means you're less likely to encounter a situation where the queue is slammed by a large background process (ie. copying a 400mb file), and something you do in the foreground requires immediate I/O to continue, because the longest you'll have to wait is 4 I/Os!

Anyway, if you want to give it a try, just paste those commands into your terminal as root.

I'd advise against making them permanent until more people try it out.

And if you're a VFS maintainer and notice an error in my logic please point it out as well! :)

Re: Massive interactivity improvement under high I/O load!
 

what do you mean by causing the I/O to block, does it return a failure status on the request?

Re: Massive interactivity improvement under high I/O load!
 

No, it forces I/O attempting processes to write out the dirty pages during their time slot, as I understand it.

So during the milliseconds-to-seconds that there's too many dirty pages, the CPU ignores I/O requests and instead spends those time slots writing out the dirty pages. If the gap between dirty_ratio and dirty_background_ratio is somewhat small, and the gap between flushes is reasonably small, it shouldn't cause too many issues on a human-perceptibility level.

I am not that knowledgeable on kernel parameters on the VM system though, so feel free to correct me, anyone who knows better.

Re: Massive interactivity improvement under high I/O load!
 

How does this compare to Swappolube?

Re: Massive interactivity improvement under high I/O load!
 

Never heard of swappolube... :)

In any case, like Mentalist Traceur said, blocking I/O doesn't result in a failure, it just means the process read()s, write()s, etc.. won't return immediately, but only once there's sufficient room in the (now very small) queue.

The benefit is that some desktop application that needs to read a config file doesn't risk ended up at queue position 100, waiting for 99 I/Os to complete, because cp, tar, or samba wasn't able to burst all those writes through (write() blocked quickly).

Re: Massive interactivity improvement under high I/O load!
 

I've written tune-up tools. I've other parametres. U may find it interesting. What are the original values?

Re: Massive interactivity improvement under high I/O load!
 

Thanks.. I'll check em out.

I actually don't have the original values anymore.. :p They were quite different, though. If anyone wants to check, just run this (root not required):

Re: Massive interactivity improvement under high I/O load!
 

U should try ur settings with cutetube and avatar trailer hd. Looks very promising. I want to try it later.

The min_free_kbytes value is not that different from the original.

Re: Massive interactivity improvement under high I/O load!
 

As requested. As a side point. If this does what we think it does - then the issues with transmission essentially locking up the phone (ok really slowing it down) will be mitigated to some extent.

Re: Massive interactivity improvement under high I/O load!
 

Exactly. I'm actually able to place phone calls, browse, and launch programs while writing at 8mb/sec! It was totally unusable before.

Re: Massive interactivity improvement under high I/O load!
 

you can adjust most of these settings from swappolube, but its missing the
why couldn't you look yourself?

Re: Massive interactivity improvement under high I/O load!
 

Swappolube came from Hawaii's optimizations. Most of them are similar to yours, however they were more along the lines of reducing occurances of I/O in general, and reducing swapping when not necessary. Swappiness was set to 30, and a few other changes were made. dirty and dirty_background ratios were put really high, and he didn't reduce the queues... Eventually someone packaged those changes as "swappolube" as a script, then it got made into a GUI program. Anyway, the difference is hawaii's original mods aimed for a I/O less, I/O in efficient bursts approach, where as you seem to go for a I/O often, so that I/O doesn't build up.

That said, I personally have been fiddling to get my own combination of settings based on those original mods.What I personally want to see is as little swap as possible while I still have ram space, but I don't want it to zealously abandon swapping the way it would if swapping is made zero. I'll fiddle with this myself, let you know if it seems better than what I'm doing.

It seems more in line with what I've been trying to do, which is keep as little memory as possible from being forced into swap, while still allowing for swapping when it is 'needed', since the smaller queues and the smaller dirty ratios would help keep it from accumulating dirty pages.

I'm having stellarium downloading one of the more massive (8 of 9 I believe) catalogs right now over wifi, so I don't want to fiddle too much now, but I'll get back to you when I get that over with and have had time to write the changes to where I want them, and then rebooted.

Re: Massive interactivity improvement under high I/O load!
 

I would, but I'm still trying to figure out how to workaround Samba's unwavering desire to pre-allocate writes... which don't cooperate with non-sparse-file enabled filesystems like the lowly fat32.

I'll take a look at swappolube later. :)

Re: Massive interactivity improvement under high I/O load!
 

This isn't the original values. I have others. Also OP is right it makes a huge difference but i strongly doubt that it will work with transmission which is strange enough.

Re: Massive interactivity improvement under high I/O load!
 

Can u try this with my tools? I'm just curious! In the original kernel swappiness is 100.

Re: Massive interactivity improvement under high I/O load!
 

Yeah.. for myself, performance is less of an issue than stability and predictability. I don't mind if file transfers drop from 8mb/sec to 7mb/sec, but I can't stand waiting 5 seconds for a UI response or crashes due to deadlocks, so I'll eliminate burstiness at the cost of efficiency any day.

And for swap, it's one of those frustrating balances. We want to swap http://talk.maemo.org/images/icons/icon8.gifas much unused data out as we can (ie. X init and bootup code, shells hosting blocked scripts, etc) to free memory for stuff we care about yet don't want to swap out stuff we might want again (and in a hurry). My argument for a low swappiness factor is.. since we're under so much pressure anyway the system will not have trouble finding excuses to page out. :)

I wonder if it would be worth setting some binaries unswappable (ie. phone).

Re: Massive interactivity improvement under high I/O load!
 

My tools is using mlock to lock x into ram. It seems to work. U should try it. And also hawaii is a troll.

Re: Massive interactivity improvement under high I/O load!
 

Personally, I didn't even know so much useful-only-at-boot crap sat in the ram indefinitely. If these things sit in ram after boot, can't a script be written to simply get rid of all the stuff like that that wastes memory space?

Re: Massive interactivity improvement under high I/O load!
 

nooooooooooooo not you, I'm mad at the Tiago guy because he posts like 10 useless things per day, I'm suggesting if he REALLY wanted to know, he could have looked.

Re: Massive interactivity improvement under high I/O load!
 

I wonder if this would fix the problems people have been having trying to decompress the Easy Debian image. Decompressing a 2GB image in MyDocs basically brings the system to its knees and takes over an hour.

I filed a bug and nobody had any idea what was wrong...

Re: Massive interactivity improvement under high I/O load!
 

so far so good, copying a 1.2gb file from mydocs to mydocs and system is still perfectly usable. Will try to stress it more tomorrow. Thanks

Re: Massive interactivity improvement under high I/O load!
 

I was thinking the same thing. Also, while we're on the subject, has anyone tried testing if the noop I/O scheduler improves things?

Re: Massive interactivity improvement under high I/O load!
 

Noop scheduler is activated by default and also u can't change a thing.

Re: Massive interactivity improvement under high I/O load!
 

I'm not even in this thread and you're calling me out? Seriously? "Your tool" is cobbled together scripts. Your tool uses libraries that OTHER people wrote and you give no credit.

You can drop free caches however AFAIK you'll notice higher load and RAM usage because there are no more caches. When a process needs memory dirty caches are dropped. Best to let the kernel handle what it does best?

I've off-loaded swap to my microSD. There's a noticeable decrease in IO "collision" and my device is much more responsive during those times.

Re: Massive interactivity improvement under high I/O load!
 

Im with hawaii on this one, this epitaph guy seems a bigger tool then his tool.
A bit insecure too.

Re: Massive interactivity improvement under high I/O load!
 

Offloading to SD card makes sense - wouldn't that mean hotswapping is only possible between SD cards with swap partitions already written into them? If so, perhaps a swap-enabling script is in order - check for x bytes of space free on card, run one of the partition resizing/making programs to shrink the partition containing the free space, then create your swap one. Then mount the entire thing.

One or two more scripts and you could probably make it capable of dumping between the on-device swap partition (assuming you leave it there) and the SD-card ones... Would be a pain in the ***, but seems doable enough, at least in theory.

As for hawaii and trolling - he's not a troll. He's just sometimes quite hostile when it comes to newbs. Which is okay, most of the time. Also, see that "Know Nokia" link in his signature? You know, the one he runs? The one where the swappolube mods were born from, the one where he shows how to enable locked wifi frequencies, etc? Know that pwnitter app in the repos? He kinda made that too, if I recall correctly. He's also ported NetDiscover onto the N900 I believe, and has posted quite a lot of useful advice on here while he's been around.

Just because he doesn't regularly sugar-coat his opinions doesn't mean he's a troll.

Now, onto productive things: So I tested this with just the two queue changes - immediately attempting to download the Stellarium catalogue I was trying to download worked faster (I had given up after multiple attempts, and decided to see if your changes would make it better. Seems that they actually did).

So far, I'm leaving swappiness at 20 (I left it there a while ago, don't really significantly notice the difference between 30 and 20, but meh), and within a little bit I'll also see what effect dropping the dirty ratios will have. For now they're at the very high swappolube values.

Hawaii: About the drop caches - I know how it's used, I believe - echo 1 to it to make the caches get emptied, but then it continues caching as normal, from what I know. But would it get rid of the stuff mentioned earlier - the stuff that gets used in bootup and then doesn't get touched? If so, then it should be reasonably harmless to drop caches once late in the boot process, and then let things go naturally from there, to clean out all the unneeded after boot stuff, correct, as I understand it, and running it only once should prevent the higher loads later on, right?

Re: Massive interactivity improvement under high I/O load!
 

Be darned interested to know that myself. I am doing some other stuff with my phone at the moment but will try and give that a shot after. If you try it in the meanwhile - can you post an update here as well.

The other thing I am wondering is performance of E.D could that possibly be improved by improved IO?

I plead guilty to not knowing much about this stuff nor how much IO E.D does.

Re: Massive interactivity improvement under high I/O load!
 

It's not a question of cache pollution during bootup; none of that will survive long. It's that many programs/scripts (ie. bash and init) fork off the rest of the system and need to survive. And, programs are almost always completely loaded into memory and never free()'d, even though much of the code may never be called again after startup (ie. Xorg's config reading code).

Re: Massive interactivity improvement under high I/O load!
 

What are disadvantages of using microSD as swap memory? What happens if back cover is removed intentionally or by accident during heavy swapping?

Re: Massive interactivity improvement under high I/O load!
 

well, it should be slower memory, so there might not be any increase in performance, or it might even be slower. It could reduce the life of the SD card. Partitioning it is a pain, you probably have to wipe it clean. I read the whole thread on that and I didn't see what happens when you force it to unmount...

Re: Massive interactivity improvement under high I/O load!
 

Well, you lose access to whatever is swapped. Which is why you may just be very screwed. :D If/when I invest in that 32 GB class 10 MicroSD card I am hoping to acquire eventually, I will be happy to test this unmount-during-swapping scenario. Too lazy to do it with my current somewhat-small-capacity, slower ones.

Also, the above test I mentioned above while downloading catalog 8/9 was with Laptop mode on. I turned laptop mode off for this next test (catalog 9/9, significantly bigger) - while leaving the massive dirty (background) ratio values... So that combination was a bit less than ideal. Download speed was nice, but UI responsiveness ground to a halt in some moments... Hmmm, specifically, during the actual finishing of the download. It had the same horrible unresponsiveness during the previous test for about a minute or so, when it finished downloading that catalog.

So what does that tell us? Laptop mode on-off doesn't seem to effect either sustained downloading or the finalizing of the write of that file, that I could notice. By itself, however, it seems that just lowering the I/O queues, but not changing the ratios or the vfs_cache_pressure, doesn't solve the unresponsiveness problem.

On the other hand, for all I know, it does, and had this been an N900 with all values at stock, the same slowness/UI-paralysis could have lasted much longer, when downloading that big of a file and saving it, all the while having Stellarium running in the background, with real time and a decently wide viewing angle, plus twinkle for the stars enabled.

So, to sumrise, when using Stellarium's built in catalogue download, tweakings JUST the queue sizes down seems to increase download speed, but, if your dirty ratio and dirty background ratio are high, swappiness is low, you do NOT have regular pdf flush daemon wakeups enables (mine don't wake up automatically, I have both centisecs values at 0), there is still a massive slow-down when finishing the download.

My GUESS is that had I had the centisecs values set to something low, or the ratio values set to something low, both would have lessened the final slow-down as well, probably significantly, because with dirty ratio at 95 and dirty background ratio at 65, and no routine flush daemons waking up, the dirty pages built up, both in RAM and in swap, I suspect.

Speaking of which, Stellarium runs great with all star cataloges downloaded. This is the non-mobile one. The mobile one presumably does even better.

Re: Massive interactivity improvement under high I/O load!
 

According to this thread (from post 21 onwards) the default is cfq.

It should be a simple matter of "echo noop > /sys/block/mmcblk0/queue/scheduler".

Re: Massive interactivity improvement under high I/O load!
 

On the subject of swap, do take a look at the ramzez and Diablo Turbo threads. On Diablo at least, ramzswap is the single most effective way to improve responsiveness when memory-starved (which is pretty much always if you're actually using the device).

Re: Massive interactivity improvement under high I/O load!
 

i just figured out how to do this yesterday, using the tablet mode hack.
http://talk.maemo.org/showthread.php?t=69912

do you have a more elegant way of doing this or is this the same method youre using?

Re: Massive interactivity improvement under high I/O load!
 

I'm giving it a try, seems noticeably quicker right off the bat :cool:

Edit: Multitasking performance has REALLY improved. The performance increase from this is better than a 700Mhz overclock! Note: I'm also running swap-on-microSD.

Re: Massive interactivity improvement under high I/O load!
 

That's not entirely correct. flash memory has massive penalty on writing randomly to different erase pages. I planned to share a link here, but have to realize I can't find the one good one. Google might help, or ask shadowJK on #maemo IRC. Basically the point is - depending on number of buffers in MMC controller - the writing to a different erase page (wich are sized like e.g 128kb) makes the controller erase a flash-page and write the dirty internal buffer to flash, then read back to buffer the new page addressed, and finally modifies the content of buffer according to the pending IO. So for a sequence like write to 0x10000, 0x50000, 0xB0000 it makes almost no difference if the individual write IO is for 1 byte or 50kB - it's the erasing and rewriting of a whole page that takes xx milliseconds, while IO to the same erase page actually only modifies the RAM buffer in controller and takes no time at all.

AFAIK a lot of elementary stuff already is mlocked, e.g rtcom-dialer-ui

What we really need is maybe a scheduler (plus kswapd) tailored to fit the exact operation mode and capabilities of the particular flash controller

/j

Re: Massive interactivity improvement under high I/O load!
 

Tried it a few times (twice by accident, forgetting that I had swap on SD, once on purpose to test the effect), every time the system either hung or reset (can't remember which one). I don't think the system actually needs to be swapping at the moment, it's probably enough that the pages get lost.

It would be interesting to know whether the unmounting/disconnection of the SD card is done by SW or HW, and whether it can be prevented.

Re: Massive interactivity improvement under high I/O load!
 

I'd guess it's done by the MMC-driver in kernel. There's definitely no hw switching off uSD slot the hard way, but given the fact you can ruin a card by inadvertently removing it, I won't suggest to override this mechanism.

/j

Re: Massive interactivity improvement under high I/O load!
 

Yes, because we want our SD cards corrupted if yanked out before unmounting.

It IS a good idea, but only in conjunction with swap relocating
scripts, in my opinion.

Re: Massive interactivity improvement under high I/O load!
 

You answered your own question :)

I've been using SD for swap for a long time now and it makes a big improvement since it frees up eMMC i/o for other things besides swapping.

For testing the settings from this thread I moved swap back to eMMC to get a more fair comparison. If I can avoid using SD for swap I will.