maemo.org - Talk - Massive interactivity improvement under high I/O load!

Page 4 of 5

« First

< Prev

Next >

Show 40 post(s) from this thread on one page

maemo.org - Talk (https://talk.maemo.org/index.php)

- Maemo 5 / Fremantle (https://talk.maemo.org/forumdisplay.php?f=40)

- - Massive interactivity improvement under high I/O load! (https://talk.maemo.org/showthread.php?t=69973)

MyNokiaN900

2011-03-13 15:12

Re: Massive interactivity improvement under high I/O load!

Quote:

Originally Posted by Alfred (Post 966838)

Hello everybody! Thanks for another great changable stuff. I'm noob right here, so could You provide me with the number for these new fields, which would make my beast faster? i don't really copy a lot of files, so just for the system and stuff...
Thanks in advance

Alfred, why don't you start reading this thread from page 1. All the answers are here as well as suggestions and the Swappolube application to try. It will make your life a lot easier if you are not comfortable with X-term scripts. ;)

AliasXZ

2011-04-01 12:15

Re: Massive interactivity improvement under high I/O load!

Just thought id add my 2 cents, even though this post is a little old...

I have just done a full backup of my rootfs using tar to my MMC and the phone stayed respsonsive for the whole process, before it would kill it.

I'm using the settings from page 1 :)

TiagoTiago

2011-04-01 15:04

Re: Massive interactivity improvement under high I/O load!

I must be doing somthing wrong, haven't noticed any improvements in the performance, much less a massive one :(

qole	2011-04-23 16:36

Re: Massive interactivity improvement under high I/O load!

Quote:

Originally Posted by freemangordon (Post 965563)

The last thing to do was to play with swap - bingo, at the moment I swapoff EMMC swap, extracting speed raise to about 10 MB/s with no lagging at all. And the strange thing is that performance raise at the moment I did swapoff, even before swap was transferred to sdcard swap partition. Swap priorities do not matter - as long as EMMC swap is turned on, performance degrades severely.

I'm wondering if there is any way to disable swap on a specific process or partition? I would like to disable swapping during the file extraction phase, just on the lzma extraction process.

Another option would be to use the same trick they used on the old Maemo tablets; make a swap file on the SD card and make that the temporary swap during the extraction. That way, the user doesn't need to know how to repartition the SD card, and the N900 doesn't bog down and possibly reboot...

misiak

2011-04-23 17:22

Re: Massive interactivity improvement under high I/O load!

Quote:

Originally Posted by qole (Post 994205)

I'm wondering if there is any way to disable swap on a specific process or partition? I would like to disable swapping during the file extraction phase, just on the lzma extraction process.

I don't know if it can help you, but by reading file "/proc/iomem" you can see what addresses of memory are mapped to which devices (I understand it like that (source) and it makes sense, because two last options, both named "System RAM" make 256 MB alltogether). In process (although it would look horribly ugly) you could allocate all space with malloc() (if you use C or C++) and check if returned pointer address is inside this area. But I have no idea if it would work properly and how to force system to alloc a memory from this area... Just wanted to point it. Your Linux and programming skills are much greater than mine (I suppose), maybe you even knew what I wrote, but even if this idea is stupid, could you explain why it can or cannot be done (if you know)? Thanks in advance.

PS. Simple code example of my idea how to check memory addres:

Code:

int *ptr = malloc(100);

if((int)(ptr)>=0x80000000 && (int)(ptr)<=0x8fffffff)

   printf("Yay! Address %x is in physical RAM :)\n", (int)(ptr));

else

   printf("Boo! Addres %x is not in physical RAM :(\n", (int)(ptr));

javispedro

2011-04-23 17:28

Re: Massive interactivity improvement under high I/O load!

Quote:

Originally Posted by qole (Post 994205)

I'm wondering if there is any way to disable swap on a specific process or partition? I would like to disable swapping during the file extraction phase, just on the lzma extraction process.

Have a look at the madvise man page.

However, considering that you mentioned lzma, I believe that what you'd like is for it not to cache any files it access so that other cached files or processes' pages are not evicted from memory; in which case, I suggest you look at the fadvise man page.

qole	2011-04-23 20:41

Re: Massive interactivity improvement under high I/O load!

I was hoping for something from the command line...

misiak

2011-04-23 21:05

Re: Massive interactivity improvement under high I/O load!

Quote:

Originally Posted by qole (Post 994278)

I was hoping for something from the command line...

I thought so... :P

Why not just:

Code:

# read current values

VAL01=`cat /proc/sys/vm/dirty_ratio`

VAL02=`cat /proc/sys/vm/dirty_background_ratio`

VAL03=`cat /proc/sys/vm/dirty_writeback_centisecs`

VAL04=`cat /proc/sys/vm/dirty_expire_centisecs`

VAL05=`cat /proc/sys/vm/min_free_kbytes`

VAL06=`cat /proc/sys/vm/swappiness`

VAL07=`cat /proc/sys/vm/vfs_cache_pressure`

VAL08=`cat /proc/sys/vm/page-cluster`

VAL09=`cat /sys/block/mmcblk0/queue/nr_requests`

VAL10=`cat /sys/block/mmcblk1/queue/nr_requests`

# use vaules from page 1 of this tread

echo 3 > /proc/sys/vm/dirty_ratio

echo 3 > /proc/sys/vm/dirty_background_ratio

echo 100 > /proc/sys/vm/dirty_writeback_centisecs 

echo 100 > /proc/sys/vm/dirty_expire_centisecs 

echo 4096 > /proc/sys/vm/min_free_kbytes 

echo 50 > /proc/sys/vm/swappiness 

echo 200 > /proc/sys/vm/vfs_cache_pressure 

echo 8 > /proc/sys/vm/page-cluster

echo 4 > /sys/block/mmcblk0/queue/nr_requests

echo 4 > /sys/block/mmcblk1/queue/nr_requests

#

#DO SOME STUFF HERE, EG. RUN YOUR PROCESS

#

# set previous values

echo $VAL01 > /proc/sys/vm/dirty_ratio

echo $VAL02 > /proc/sys/vm/dirty_background_ratio

echo $VAL03 > /proc/sys/vm/dirty_writeback_centisecs 

echo $VAL04 > /proc/sys/vm/dirty_expire_centisecs 

echo $VAL05 > /proc/sys/vm/min_free_kbytes 

echo $VAL06 > /proc/sys/vm/swappiness 

echo $VAL07 > /proc/sys/vm/vfs_cache_pressure 

echo $VAL08 > /proc/sys/vm/page-cluster

echo $VAL09 > /sys/block/mmcblk0/queue/nr_requests

echo $VAL10 > /sys/block/mmcblk1/queue/nr_requests

Well, it won't use these values for just one process, but for all processes during execution time of this process... But basically that could be sufficient for tasks like EasyDebian image extraction, am I right? ;)

EDIT: After reading more posts from this thread, maybe using values from first post isn't BEST option... but there are other settings posted ITT, so you can pick some other ones

misiak

2011-04-23 21:12

Re: Massive interactivity improvement under high I/O load!

Quote:

Originally Posted by freemangordon (Post 965563)

They last thing to do was to play with swap - bingo, at the moment I swapoff EMMC swap, extracting speed raise to about 10 MB/s with no lagging at all. And the strange thing is that performance raise at the moment I did swapoff, even before swap was transferred to sdcard swap partition. Swap priorities do not matter - as long as EMMC swap is turned on, performance degrades severely.

Could that possibly be used also in new nicolai's camera-ui, so recorded videos won't lag (the lag somtimes because of slow i/o operations)?

qole	2011-04-24 01:03

Re: Massive interactivity improvement under high I/O load!

I tried making a swap file on the SD card, so I could get the benefit of an external swap file without the complexity of repartitioning the card.

Sadly, I ran across this bug: https://bugs.maemo.org/show_bug.cgi?id=7165

Maybe if it were fixed in Titan's power kernel, I could do it... but then that defeats my purpose of trying to find a way to get this working for people who don't want to futz with custom kernels etc.

qole	2011-04-24 04:03

Re: Massive interactivity improvement under high I/O load!

1 Attachment(s)

Ok so I was able to decompress ubuntu-m5-v1.img.ext2.lzma in less than 10 minutes (rather than the 45 minutes or more that it usually takes) using the included script, taken from freemangordon here and other places. I have no idea what piece is the "magic piece", I'll have to experiment some more...

Code:

#!/bin/sh

echo 75 > /sys/devices/system/cpu/cpu0/cpufreq/ondemand/up_threshold

echo 150000 > /sys/devices/system/cpu/cpu0/cpufreq/ondemand/sampling_rate



echo 30 > /proc/sys/vm/swappiness

echo 0 > /proc/sys/vm/page-cluster



echo 512 > /sys/block/mmcblk0/queue/nr_requests

echo 512 > /sys/block/mmcblk1/queue/nr_requests

echo 0 > /sys/block/mmcblk0/queue/iosched/slice_idle

echo 0 > /sys/block/mmcblk1/queue/iosched/slice_idle

echo 32 > /sys/block/mmcblk0/queue/iosched/quantum

echo 32 > /sys/block/mmcblk1/queue/iosched/quantum

echo 80 > /sys/block/mmcblk0/queue/iosched/fifo_expire_sync

echo 80 > /sys/block/mmcblk1/queue/iosched/fifo_expire_sync

echo 180 > /sys/block/mmcblk0/queue/iosched/fifo_expire_async

echo 180 > /sys/block/mmcblk1/queue/iosched/fifo_expire_async



echo noop > /sys/block/mmcblk0/queue/scheduler



swapoff /dev/mmcblk0p3

swapon /dev/mmcblk0p3

qole	2011-04-24 05:55

Re: Massive interactivity improvement under high I/O load!

As a test, I rebooted and ran the same decompression with stock settings. It is still going, over an hour later...

UPDATE: Looking at my "stock" settings, I must have them already tweaked (maybe swappolube?) because I already have swappiness set to 30 and page-cluster set to 0. So those weren't the magic settings that sped up the decompression. And it isn't the noop > scheduler line either, that doesn't seem to have any effect.

I wonder if it is just the swapoff / swapon that I do just before decompressing? I'll try that next.

freemangordon

2011-04-24 09:46

Re: Massive interactivity improvement under high I/O load!

Quote:

Originally Posted by qole (Post 994405)

No, you need all of "mine" settings, moving swap to sd card helps, but does not do magic alone. I am running my n900 with those settings since I posted them here, mo single hang because of the io.

humble

2011-04-24 11:05

Re: Massive interactivity improvement under high I/O load!

@qole

i can confirm that with a

Code:

swapoff /dev/mmcblk0p3

swapon /dev/mmcblk0p3

lets lzma decompress faster and no reboots:) as for the other stuff in the script idk.
im going to incorporate that to DON so that we dont have to turn off watchdogs just for decompression.

qole	2011-04-24 16:35

Re: Massive interactivity improvement under high I/O load!

Yep. 8 minutes with only swapon/swapoff. The other stuff seems to have a small effect (-2 minutes) but the major speed gain seems to come from the cache reset...

casketizer

2011-04-24 18:39

Re: Massive interactivity improvement under high I/O load!

just tweaking the two nr_requests after fresh boot let me extract latest easy debian in ~5min (@1ghz)

stlpaul

2011-04-24 20:03

Re: Massive interactivity improvement under high I/O load!

What about the danger of disabling swap (even temporarily) if there is high memory usage at the time? Will OOM killer wreak havoc?

qole	2011-04-24 21:21

Re: Massive interactivity improvement under high I/O load!

I tried the cache reset while apps were open, it wouldn't let me. But definitely tell users to close all other apps during decompression.

I'll see if the nr_request lines help even more...

freemangordon

2011-04-24 23:16

Re: Massive interactivity improvement under high I/O load!

Quote:

Originally Posted by qole (Post 994749)

I tried the cache reset while apps were open, it wouldn't let me. But definitely tell users to close all other apps during decompression.

I'll see if the nr_request lines help even more...

The other stuff is to keep UI responsive during heavy io, torrent download for example.

gabby131

2011-04-25 01:57

Re: Massive interactivity improvement under high I/O load!

i run the commands at the first post and saved it as a custom setting on swappolube.

i noticed improvement on using the multi-task breaking and microSD card corrupting Transmission app. multi-tasking is like im doing nothing, with transmission open with 3 other apps (web[3], conky, qmltube). although Transmission still lags (not responding) the multi-tasking and other stuff is not affected, or if affected, not fully,

before post 1, my device is not usable when transmission is checking for files, but after post 1, this issue is resolved.

thanks.......:)

qole	2011-04-25 16:39

Re: Massive interactivity improvement under high I/O load!

So the swapoff/swapon works to speed up decompression, but only when Swappolube is enabled. I still need to find the magic setting(s).

epitaph

2011-04-25 16:50

Re: Massive interactivity improvement under high I/O load!

Quote:

Originally Posted by qole (Post 995106)

So the swapoff/swapon works to speed up decompression, but only when Swappolube is enabled. I still need to find the magic setting(s).

I'm not using the recommend value of 30 from Swappolube. It didn't worked for me. I'm using something > 45 and I can confirm that swapoff/swappon works wonders and also a low nr_requests but not so low as OP suggested.

qole	2011-04-25 17:21

Re: Massive interactivity improvement under high I/O load!

I disabled Swappolube, and tried just swapoff/swapon. Decompression took TWO HOURS.

I rebooted, did mmblk0 and mmblk1 nr_requests = 512, and swapoff/swapon, and the same decompression took only 9 minutes.

So that seems to be the magic.

gabby131

2011-04-25 18:32

Re: Massive interactivity improvement under high I/O load!

Quote:

Originally Posted by epitaph (Post 995110)

i saw a thread re: I/O nr_request and according to that, the lower the value might be better.....please correct me....... :)

thanks.

EDIT: i hope this thread be merged here....

stlpaul

2011-04-25 21:46

Re: Massive interactivity improvement under high I/O load!

Also FWIW there are threads on Android forums about this exact same topic, tweaking the settings for better I/O performance. One I read recently suggested setting nr_requests to 1024 (along with a slew of other settings, like we're doing here).

humble

2011-04-25 21:54

Re: Massive interactivity improvement under high I/O load!

it kinda late but i forgot to add that i have Swappolube installed. i only tweaked

Code:

min.Free kbytes from 2039 default to 512

gabby131

2011-04-26 04:26

Re: Massive interactivity improvement under high I/O load!

i kinda not getting the hang on the wiki, is there a noob way on setting the priority of swaps equal? :o

thanks........google-ing

EDIT: found one!!! google and the search bar rules!!!

gabby131

2011-04-26 05:46

Re: Massive interactivity improvement under high I/O load!

i cant set the swap priorities to equal.....>_<

i need light!!! :D

stlpaul

2011-04-26 05:52

Re: Massive interactivity improvement under high I/O load!

Quote:

Originally Posted by gabby131 (Post 995403)

i cant set the swap priorities to equal.....>_<

i need light!!! :D

Default swapon program doesn't allow setting priority. You need to use a different swapon (for example the one from easy debian works from outside of the chroot). Though I have better performance just using SD for swap and disabling the eMMC swap entirely.

sr00t

2011-04-28 17:44

Re: Massive interactivity improvement under high I/O load!

qole, I just tought you wanna listen some positive review here. I don't have any problems at all with easy debian, and I use extensively almost all day(s).
Installed it in MicroSD Class6, sharing with NITDroid. No problems at all.
Using last power user kernel and SSU, only kernel setting changed is swappiness, to 30. No OC at all, stock @ 600mhz.

Estel

2011-04-29 15:31

Re: Massive interactivity improvement under high I/O load!

Nice to hear, sr00t, but as i understand it, the problem is not with general Easy Debian use, but with initial unpacking Easy Debian image. That shouldn't take more than 15 minutes, but many time last for 2 hours, and you may not see happy ending, because N900 tend to reboot after long time of 0 responsiveness.

It apply to any big file unpacking (or even copying!) in general, not only Easy Debian.

My stone to the garden - i use swap on SD class 2 only, and even with such low class, i see massive improvement, don't even need to use swapoff/swapon. Have in mind that i also tweaked nr_request to 256. That seems quite opposite to setting proposed in first post of this topic - maybe it need update?

freemangordon

2011-04-29 16:18

Re: Massive interactivity improvement under high I/O load!

Quote:

Originally Posted by Estel (Post 997417)

My findings are exactly the same, see my settings here http://talk.maemo.org/showpost.php?p...&postcount=112, proved to work. I have nor swapolube installed neither other tweaks (KP46 of course).

To OP - please either remove or update first post to reflect latest findings. Thanks.

gabby131

2011-04-29 19:22

Re: Massive interactivity improvement under high I/O load!

Quote:

Originally Posted by stlpaul (Post 995407)

oooohhh.....awright! fdisk-ing microSD to a 1gb swap partition. :)

shadowjk

2011-04-30 01:03

Re: Massive interactivity improvement under high I/O load!

Swapoff/swapon has the side effect of defragmenting the swap area, which makes future swap writes faster.

mehulrajput

2011-04-30 03:12

Re: Massive interactivity improvement under high I/O load!

install swapon from debian package called as mount. unzip using archive manager and copy it to where every you want. there is a thread for creating swap on mmc.

i also have swapon and off script which basically switches one off and second on and otherway round so that all swap gets defragmented. i run generally in 3 to 4 days to cleanup swap space.

shadowjk

2011-04-30 18:58

Re: Massive interactivity improvement under high I/O load!

I have a script which looks at the output of 'iostat -m' command. When the amount of megabytes written to a swap partition exceeds the size of the swap partition, the fragmentation effects start. I store the megabytes written in a file after reswap(swapon temp, swapoff main, swapon main, swapoff temp) and keep track of bytes written after last reswap.

hawaii

2011-04-30 20:53

Re: Massive interactivity improvement under high I/O load!

I also run a re-swap and chunk out to temp and then back to primary every 3 days at 4 AM with fcron.

Helps a ton.

Mentalist Traceur

2011-04-30 22:06

Re: Massive interactivity improvement under high I/O load!

@shadowk & hawaii:

Do you both do this using the internal swap? I'm curious how much of the slowness people regularly report comes from I/O collisions with the eMMC and how much comes from the swap partition fragmentation - since not that many people move swap off-device onto a microSD and even less run scrips like you two do to make sure swap stays fresh and defragmented I can't help but wonder which of these actually helps more; and if you have experience with both methods seperately, and/or one method seperately and both methods combined, if you perceived one as no longer useful once the other one was implemented. I.E do you feel moving swap to the sd card helps even if you're already doing the swap-space cleaning thing (and vice versa, though I suspect preventing fragmentation is always going to be an improvement in the long run - it's only the disk I/O thing I think could be negligible, given the discoveries in this thread that swapoff/swapon seems to speed up things that were originally believed to be the fault of I/O bottlenecks)?

shadowjk

2011-05-01 20:41

Re: Massive interactivity improvement under high I/O load!

I first moved swap to microsd. This helped a little with some I/O heavy things. but found that "stutter" increased significantly after 3-4 days uptime.

I then made a script to dump info (sector numbers) of all write accesses to microsd, plotted that on a graph on my PC. Discovered that it was a linear line (mostly) until reaching the end of the swap area, after which the dots indicating which sector was written to became more and more random. This is how I discovered swap fragmentation issue and the fix.

As I still need a temporary swap for the swapoff/swapon cycle I just kept swap mainly on microsd, and use the emmc swap temporarily when refreshing the microsd swap.

hawaii

2011-05-01 22:51

Re: Massive interactivity improvement under high I/O load!

I only use swap on external microsd. I never bothered to do anything intelligent :P I just set a 2 day nightly "reswap" provided my device was locked, plugged in and it was 3 AM in the morning.

I still reboot my device every now and then to clean out cobwebs.

All times are GMT. The time now is 15:00.

Page 4 of 5

« First

< Prev

Next >

Show 40 post(s) from this thread on one page

vBulletin® Version 3.8.8