[Under consideration] Remove 256MB limitation of the rootfs partition in the N900

[stefanmohl]

The root directory of Maemo5 is limited to 256MB on the N900. This is a serious limitation, and users often and easily fill it up (especially if using the extras-devel repository). It is unfortunate in other ways too. For example, the standard Debian distro officially supports Arm7, so with a better partitioning, much of Debian would be directly usable through apt-get. We could use the full gamut of GNU/Linux commands instead of the settling for the Busybox replacements - imagine, "less" and "man" included by default!

The current solution is "optification". However, that solution is brittle since it requires active adaptation of all packages so that they are placed in /opt instead of in their usual locations. This means added burden on developers and makes packages slower to move into stable repositories.

As far as I know, the total set of mass storage available for the N900 is:


* 256MB Root partition (in a fast 256MB flash)
* 768MB Swap
* 2GB /home
* 27GB FAT32 USB storage
(Swap, /home and USB storage are all in a 32GB slow flash)

If I understood this correctly, the main reason for the small root partition is that it is in a faster flash than the home partition. To keep the device responsive, the speed sensitive stuff is placed in that flash, and other stuff is placed in /opt (/opt in turn is mounted in /home).

There are a few more constraints, in summary:

1) There is a fast 256MB flash and a slow 32GB flash
2) The device needs to be capable of getting to a guaranteed stable functioning state from a reflash
3) A reflash should not delete the user's document data (i.e. pics, music, etc) in the large flash storage
4) A reflash will always replace the entire contents of a flash memory.

Right now I am using a home grown solution, but it breaks each time the distro has an upgrade, forcing me to spend days rebuilding and moving everything again. And newbies end up having to reflash just because they tried out a few beta-level apps and filled the root partition to a point where booting fails. We need a better standard solution!

The brainstorm for this thread is here:
https://maemo.org/community/brainstorm/view/remove_256mb_limitation_of_the_rootfs_partition_in _the_n900/

There is a related Brainstorm about re-partitioning the 32GB internal flash here:
https://maemo.org/community/brainstorm/view/more_efficient_and_flexible_use_of_internal_flash/
though that doesn't solve the root directory problem.

There is also a previous discussion of the issue in the wiki:
http://wiki.maemo.org/Opt_Problem

[stefanmohl]

Does anyone know if there are other reasons for the small root partition apart from flash memory speed? Am I even right about the fast flash reason (can someone verify)?

Assuming that the 256MB partition really is in fast flash, it seems reasonable that the 768MB swap partition also is in fast flash. Out of curiosity: Does anyone know if there is an underlying 1GB fast flash partitioned into the root and swap partitions? If so, there might be other partitioning possibilities...

[Laughing Man]

Yes you are correct about the OS being located on a small root partition to take advantage of the faster flash memory speed. But I believe the swap is part of the other flash memory, not the 256 MB flash.

[GeneralAntilles]

Assuming that the 256MB partition really is in fast flash, it seems reasonable that the 768MB swap partition also is in fast flash. Out of curiosity: Does anyone know if there is an underlying 1GB fast flash partitioned into the root and swap partitions? If so, there might be other partitioning possibilities...

There are two flash storage areas in the N900. The 256MB of OneNAND in the PoP on the OMAP3 contains the rootfs, the kernel and the bootloader. The second flash unit is the 32GB eMMC which contains home, swap and the large 27GB storage partition.

There's no easy way to make this better. Either you optify your packages (slow) or boot directly from eMMC (slower).

[j.s]

As far as I know, the total set of mass storage available for the N900 is:

* 256MB Root partition
* 768MB Swap
* 2GB /home
* 32GB FAT32 USB storage



The 768MB swap, 2GB /home, and FAT 32 partitions all share the same 32GB flash memory. So swap is not on the "fast" flash memory.

The 32GB partition is 32 billion bytes, not 32*1024*1024*1024 bytes.

[stefanmohl]

Thanks for making the partitionings clearer to me. I have edited the brainstorm accordingly.

@GeneralAntilles: I understand that the solution isn't very easy, but I have a suggestion (also to be found in the brainstorm itself as solution #1):

Solution #1: Place root in the large flash and link critical files in the fast flash

Posted on 2010-02-09 01:46 UTC by Stefan Möhl.

It seems to me that the current "optification" solution is close to right but the wrong way around. Instead of placing everything in root, and changing big slow stuff to go in /opt, we should do it the other way around. Put the root directory in the bigger flash, and for files that really need speed, link those specifically from the fast small flash. I.e. some sort of "speedification". The reason this is better is that it is more robust. If a package needs to have a file in the fast flash, it can be placed there. But if nothing is done, the package will at least work, just not at optimal performance. In contrast, a package that isn't optified will, if it can be installed at all, have a good chance of making the device unusable by completely filling the root partition.

So, if I understand GeneralAntilles right, the problem is the files used at boot time (and probably a few others). Well, in a nutshell, my suggestion is to "speedify" just those files, instead of (quite literally) optifying everything else in the world.

[GeneralAntilles]

Solution #1: Place root in the large flash and link critical files in the fast flash

Go for it. This isn't, however, an acceptable solution for Nokia.

[stefanmohl]

Hmmm, do you have any information about why this isn't acceptable for Nokia? Obviously, I can change my device around any way I like, but that breaks with each upgrade (and doesn't help newbies). I am looking for a default distribution solution in this brainstorm.

Why not treat the few files that need to be fast as special (i.e. link them from the fast flash) and all other files as the normal case (i.e. root default in slow, large flash), instead of the other way around?

[maxximuscool]

agree but how? it wouldn't be reliable when things broke ever time the new update arrived!

[stefanmohl]

If Nokia would change the default Maemo5 distribution to have root directory files in the large slow flash by default, and only the files that really need it in the fast flash, then it would be distribution policy. Then it wouldn't break at each upgrade any more.

[geneven]

As I understand, it's acceptable to Nokia to sell innocent users non-optified files that may screw up their system, but this solution isn't acceptable? This would rescue the minefield Nokia created with OVI and the users gullible enough to purchase from it.

Correct me if I'm wrong...

[Thesandlord]

Is eMMC really that much slower than flash? I am booting of SD card on my n810 and I see zero speed difference. The file-system is no longer compressed, so it is less taxing on the system and takes care of the speed.

[Martin Holz]

Traversing a file path and resolving a link does not come for free. Putting / on a slow file system could slow down access to any file, even if it is on the fast file system itself.

[jcompagner]

Is eMMC really that much slower than flash? I am booting of SD card on my n810 and I see zero speed difference. The file-system is no longer compressed, so it is less taxing on the system and takes care of the speed.

i am also wondering about this. even if it is fast how much cpu usage (and battery) is lost again by the compressing/decompressing?
we have a fast cpu but that fast that we really dont notice that? if it was a dual core i guess it would be better but now we have to wait for decomprssing.

[twaelti]

/opt is the official way (http://www.linuxtopia.org/online_books/linux_beginner_books/linux_filesystem/opt.html)to go in the linux world anyway, so stop making a fuss about it.
We developers should stop messing in root (I catch myself falling into that trap again and again, last time with shutter install lirc configs into root even altough /opt works as well :-)

[406NotAcceptable]

Is eMMC really that much slower than flash? I am booting of SD card on my n810 and I see zero speed difference. The file-system is no longer compressed, so it is less taxing on the system and takes care of the speed.

I doubt it. Android users have been booting off class 6 cards for more storage for a long time. The iPhone uses a similar board as the N900 and Apple get away without using the 256MB. Likewise Beagleboard users boot from memory cards.

It may be easier to put everything on the mmc and the swap on the internal flash. This way linux would deal with putting important files in faster storage for you. If the mmc is so slow I don't really see the point of using it for swap.

Getting Maemo 5 running from a good class 6 memory card should be the first step to removing this limitation.

I actually believe this 256mb partition being the reason why the N900 won't see Maemo 6.

[johnel]

What if you loop-mounted a filesystem using aufs & chroot - similar to slax (http://www.slax.org/) and linux-live (http://www.linux-live.org/)?

E.g. My laptop at home does not have linux installed to the hard-drive but I have a custom slax install on my usb stick.

Of course the usb stick is read-only but a mounted overlayed filesystem is used to persist any changes.

For example when slax boots of the usb stick it looks for a file called 'slaxsave.dat' (a 3gb xfs-formatted file), mounts it and then persists all changes to this file.

This enables me to have a system that behaves like a "native" installation.

Is this feasible on the n900?

[Rob1n]

Is eMMC really that much slower than flash? I am booting of SD card on my n810 and I see zero speed difference. The file-system is no longer compressed, so it is less taxing on the system and takes care of the speed.

I don't think the N810 uses OneNAND flash, so the MMC will be about the same speed as the internal flash. With the N900 there's significant speed differences (see http://wiki.maemo.org/Opt_Problem for some performance details, as well as details on previous discussions about this issue).

[Rob1n]

i am also wondering about this. even if it is fast how much cpu usage (and battery) is lost again by the compressing/decompressing?
we have a fast cpu but that fast that we really dont notice that? if it was a dual core i guess it would be better but now we have to wait for decomprssing.

Generally you come out about even on these things - you're losing time & power compressing/decompressing but gaining by reading/writing less data. The compression algorithm is chosen to balance the compression ratio & compression/decompression times.

[TA-t3]

/opt is the official way (http://www.linuxtopia.org/online_books/linux_beginner_books/linux_filesystem/opt.html)to go in the linux world anyway, so stop making a fuss about it.
First, it's not used by all distros (e.g. not by Debian), and secondly, the "optify" method is very different from traditional use of /opt, which stems back from older Unix systems and is simply just another directory, and additional paths in $PATH. There's none of the bind-mounting and other tricks used by the "optify" method. I find "optify" very messy indeed.

[jcompagner]

Generally you come out about even on these things - you're losing time & power compressing/decompressing but gaining by reading/writing less data. The compression algorithm is chosen to balance the compression ratio & compression/decompression times.


yes back in the early days, in pc's you could get even better performance with compressing because HD's where really slow compared to the processor.

But with the way flash works.. i dont know if this is really a gain..

For example reading 1KB will result in a read of 1 Page (mostly 4KB)

and writing is especially with not so performing i/o controllers is done in Blocks and those are X number of pages of total sometimes 512KB at total.

So to write a 1KB block 512KB needs to be read in. 1KB updated in that block and then 512KB written again.

But this all depends on what kind of flash controller is in the N900...
But i cant believe that with flash we really gain much in read speed and write speed is really a bit slower..

[titan]

Note that we have done some benchmarks on the real filesystem performance
for different media and filesystem http://talk.maemo.org/showpost.php?p=504332&postcount=76
The NAND results are the best case scenario, as only a highly compressable stream is written.
Real life scenarios may be much worse and are on my todo list.

You can also find my ideas for a "full optification" solution on
http://wiki.maemo.org/User:Tanner#full_optification

[GeneralAntilles]

If Nokia would change the default Maemo5 distribution to have root directory files in the large slow flash by default, and only the files that really need it in the fast flash, then it would be distribution policy. Then it wouldn't break at each upgrade any more.

Like I said, this isn't a workable solution. Then each reflash would require overwriting user data.

Not. An. Option.

[ruskie]

/opt is the official way (http://www.linuxtopia.org/online_books/linux_beginner_books/linux_filesystem/opt.html)to go in the linux world anyway, so stop making a fuss about it.
We developers should stop messing in root (I catch myself falling into that trap again and again, last time with shutter install lirc configs into root even altough /opt works as well :-)


Please pleas lern about the FHS before posting such things.

/opt is there for things that AREN'T packaged by the distribution. And the format is generally:
/opt/$VENDOR and then whatever layout under that.

Here's the relevant link to FHS:
http://www.pathname.com/fhs/pub/fhs-2.3.html#OPTADDONAPPLICATIONSOFTWAREPACKAGES

Some alternate path like /usr/local would probably be a better idea in this case. Or even using unionfs-fuse on top of /usr infact that would be nice. Be always able to just unmount all the modifications.

[egoshin]

There is one an effective way to solve this problem. It can be not easy -

put all root (besides boot) into 32GB flash and use fast 256MB space as a cache for disk blocks. Initially it would take some additional time and battery to replicate blocks into new 256MB cache but later it would give a fast access to frequently used data with virtually unlimited space.

It is possible just to extend kernel cache interface to include 256MB into it. But to get the result permanent via next reboot it may requires some additional development in FS.

[titan]

no, /opt is for add-on software. Everything not shipped by Nokia (i.e. software from extras or Ovi) could be considered as add-on packages. Packages can be installed under /opt/<PACKAGE> and symlinked to /opt/bin or /opt/maemo/bin.
/usr/local is reversed for local installations, i.e. self compiled programs.


/opt is there for things that AREN'T packaged by the distribution. And the format is generally:
/opt/$VENDOR and then whatever layout under that.
Some alternate path like /usr/local would probably be a better idea in this case.

[javispedro]

No desktop GNU/Linux distro follows the FHS to the letter.

(oh, and btw: yet another /opt thread >:) ).

[stefanmohl]

Hmmm, do you have any information about why this isn't acceptable for Nokia?

I am sorry GeneralAntilles, I seem to have missed your earlier reply to this. From your latest response:

Like I said, this isn't a workable solution. Then each reflash would require overwriting user data.

Not. An. Option.

From this, I take it that there are two basic problems:

1) The device needs to be capable of getting to a guaranteed stable functioning state from a reflash
2) The reflash should not delete the user's data in the eMMC

At least those two seem like the reasonable requirements.

Well, that obviously means that the most straight-forward method; just installing root in eMMC directly, will not work. However, this is a brainstorming forum, and finding solutions when the most naive approach fails is the whole point of brainstorming. Let's not get disheartened quite yet!

I think that most agree that optification is not good as a long term solution. As such, we really do need to tackle the issue as soon as we can! Until we have it fixed, we will continue to see newbies getting into trouble such as this (http://talk.maemo.org/showthread.php?t=43825) (the guy ended up re-flashing his device). I would like to argue that a solution will have to be very convoluted before it is worse than optification, so we do have quite a bit of leeway for ideas.

[@SR]

I've been experimenting with aufs, unionfs and mini_fo for a while to mount ext3 partition on top of internal memory. All of them have problems.
One common problem was that both of the file system partitions needed to be writable. It is because of the complex algorithm when renaming non empty folder from read only file system (aufs page has a good explanation on that). For example application managed did not work because of this problem. So mini_fo did not work at all.
Aufs is most flexible in configuration. But for some reason with it phone stopped playing video. And the battery did not last long with it (~6-8 hours). And I could not get the file system to umount during shutdown so I had a lot of lost data and sometimes after shutdown phone was unusable.
Unionfs was more stable. I did not have the problems aufs had. But it was impossible to set priorities on where to write new files if there are two writable file systems. Everything was written in root file system as it already contained folders where the files were written. Unionfs has awful documentation. It was not easy to find that it was the default behavior for new files. And I could not find how to change that. So very quickly I ended up with 0 bytes left in root file system.

Right now I moved everything to the card and change the root file system to the card during boot. But I had to enable R&D mode and disable watchdog for it to work. While I was experimenting watchdog was only casing trouble. Right now I am searching for a way to remove it from the system.
With disabled watchdog system boots fine from card and everything works fast. When I moved only /usr to the card without disabling watchdog the system was slow and unusable. And it was impossible to move the whole root to the card with it enabled.
I can upload my modified firmware if someone wants to test it. It will write root file system image to the internal mmc card at the first boot and after that it will be booting from the card.

[stefanmohl]

@titan: I was just going to suggest that you add your ideas as a solution, but noticed that you have already done so. Thanks, we need to examine some solutions!

I would also like to concur with javispedro: There are many other discussions about /opt and the FHS around the 'net. Lets keep this discussion to the problem of finding a more robust way of using our two flash drives in the N900, rather than discus how standards should be adhered to or interpreted.

[stefanmohl]

I have updated the brainstorm with the issue GeneralAntilles hinted at, and added a link to the Wiki discussion too.

[ruskie]

no, /opt is for add-on software. Everything not shipped by Nokia (i.e. software from extras or Ovi) could be considered as add-on packages. Packages can be installed under /opt/<PACKAGE> and symlinked to /opt/bin or /opt/maemo/bin.
/usr/local is reversed for local installations, i.e. self compiled programs.

Please learn what FHS is and how it fits in here. Seriously. Anything that is packaged for a distribution on a regular desktop GNU/Linux system DOES NOT belong in /opt

/opt itself is mostly a historical feature or used by anything that can't/won't be packaged(think proprietary software like DB2, Oracle etc...).

Yes I am well aware of what /opt on maemo does but it doesn't mean I agree with it.

As for /usr/local... yes it's there for things an admin would install... but in this case it could be abused.

Also looking at other systems like BSDs /usr/local is where their packages install by default. Most things out of /usr/local are part of the core system. And this would probably fit maemo more as it would be more true.

[titan]

Maemo is NOT a regular desktop system, even though I wish it was more similar...
There is no authority which can enforce the FHS. It's just a guideline.
My point is that using /opt is in accordance with many interpretations of the FHS
(maybe not with yours). However, the current hybrid /usr and /opt mix with optifying
larger files is - I believe we all agree here - is a very ugly hack and also not at all FHS
complaint.
I'd be happy with any solution which clearly separates the core system and
the add-on packages, be it /opt or /usr/local.
Please learn what FHS is and how it fits in here. Seriously. Anything that is packaged for a distribution on a regular desktop GNU/Linux system DOES NOT belong in /opt.
/opt itself is mostly a historical feature or used by anything that can't/won't be packaged(think proprietary software like DB2, Oracle etc...).

[stefanmohl]

Hrmmm... I would like to remind everyone, we are looking for a robust way of partitioning our two built-in flash drives and efficiently running Maemo on them.

How /opt should or should not be used according to standards is not the issue for this thread, please take all such discussions in another topic. Please consider any sort of argument regarding interpretations of FHS (how /opt "should" be used) a troll and ignore without response. The Internet is full of debate on that issue; use one of those forums instead.

[dalonso]

What about using unionfs and just forget all this /opt ****??? Qwerty12 built modules for diablo's kernel: http://talk.maemo.org/showthread.php?t=19104

No need to optify anything, programs would be automatically installed in a root overlay in the mmc. So good, so nice.

The only thing we should take care is to deactivate it when something should really be installed in the real root partition.

Wouldn't this be a good solution?

[@SR]

What about using unionfs and just forget all this /opt ****???
I've already replied about unionfs and similar file systems on previous page of this topic and described the problems it has.

[slaapliedje]

Please learn what FHS is and how it fits in here. Seriously. Anything that is packaged for a distribution on a regular desktop GNU/Linux system DOES NOT belong in /opt

/opt itself is mostly a historical feature or used by anything that can't/won't be packaged(think proprietary software like DB2, Oracle etc...).

Yes I am well aware of what /opt on maemo does but it doesn't mean I agree with it.

As for /usr/local... yes it's there for things an admin would install... but in this case it could be abused.

Also looking at other systems like BSDs /usr/local is where their packages install by default. Most things out of /usr/local are part of the core system. And this would probably fit maemo more as it would be more true.

Ruskie is correct, with the exception that most Linux distributions put everything that is in the package manager in /usr, whereas if it's not managed by the package manager, it should go into /usr/local.

Maemo is NOT a regular desktop system, even though I wish it was more similar...
There is no authority which can enforce the FHS. It's just a guideline.

Titan, while it's technically not a regular desktop system, it basically is a regular Debian based distribution, and putting things in /opt is wrong. While I have seen a lot of distributions deviate from the LSB, almost all of them within the last 10 years have adhered to the FHS. Why? Because the FHS kicks ***, and is USEFUL, and will be useful with my suggestion.

Which is this;

The 256mb should hold / and /sbin, /bin, /root and that is it. That should be ALL that needs to be mounted initially for a Linux machine to boot (I could be missing one, but I'm not sure.)

/usr/bin and /usr/sbin do NOT need to be on the same partition as the one that boots, it can be mounted later. You can even mount it over a network and have all your maemo apps on your server (of course that would be useless when you were out on the road!) /var should be on that separate partition as well.

Most of the space currently that is sucking up the space with the optified packages is more than likely in /var, which is where your apt-get puts files (specifically /var/lib/dpkg/info/. This is where all the deb scripts go after a package is installed). This alone would help a huge amount. The other place that a lot of files go is /usr/share (at least on a normal Linux system). This keeps things clean.

It's one thing to base your distribution off of Debian for apt-get goodness, it's something else to use it and completely ignore the debian packaging guidelines (like Ubuntu Devs tend to do now and again)

It's crazy that they only gave us 256mb for our root file system, Honestly how much more of a cost of production would it have been if they had made that 512mb?

slaapliedje

[stefanmohl]

OK, slaapliedje, you are the first to get a kudos--; as a result of the first half of your posting. I am completely ignoring if what you said there was right or wrong, it doesn't belong in this thread so go put that kind of stuff in another thread (you can make as many as you like, you know).

Enough said about that. Regarding the second half of your posting:

If I understand your approach correctly, you are suggesting we put the bare necessities for booting in the fast flash and everything else in the slow flash. I can see how this would minimize the amount of data in the fast flash, and it would let part of the boot process run from the fast flash (making the boot fast). What strikes me as problematic with this is: Shouldn't we really try to have as much in the fast flash as possible, rather than as little as possible?

Edit:
On further thought, slaapliedje, you are right. We should place only those system files which really need speed in the fast flash. That leaves space for adapting other packages for using the fast flash too, and not only the system (i.e. some sort of speedification, if we assume we install on slow flash by default).

[shadowjk]

256M was the biggest available in volume at design time..

Anyway, I suspect the difference in performance is mostly because ubifs can manage the nand intelligently. The mmc type chips are designed for single-tasking camera/media use, they hide the flash/nand behind a dumb controller which is fully optimized to be cheap and simple for large file use. It doesn't really handle the random updates of 4k blocks here and there very well.. Its native block size is something like 256k, so a random 4k write gets translated into a 256k read-modify-write cycle. It doesn't take much I/O before the emmc is saturated and the reads don't go through and the app running from /opt has a small pause/lag...

I notice this often with gpodder downloading something while I'm browsing the web with disk cache enabled. The 3 of gpodder, browser, swap hitting the emmc at the same time makes the average wait (as measured by iotop) for a request to complete easily exceed 500ms. That's noticeable latency.


On my sheevaplug I compared ext3 and nilfs2 on a OCZ Rally2 dual channel usb flash drive. It has over 20megabytes/s sequential write speed, compared to the promised 6 megabytes/sec of class 6 memory cards. The benchmark consisted of copying my 2 gigabyte squid webcache directory. ext3: 500kbyte/s nilfs2: 5000kbyte/s. A factor of 10 difference! nilfs2 is by accident and its nature faster on flash.. I hope to test LogFS some day too, it has actually been designed with flash in mind..

[bitziz]

@SR, can you explain to me how to disable the watchdog on the device? you were saying you had considerable performance increase after disabling it?

[egoshin]

The mmc type chips are designed for single-tasking camera/media use, they hide the flash/nand behind a dumb controller which is fully optimized to be cheap and simple for large file use. It doesn't really handle the random updates of 4k blocks here and there very well.. Its native block size is something like 256k, so a random 4k write gets translated into a 256k read-modify-write cycle.

I think you right, I have the same impression.

A factor of 10 difference! nilfs2 is by accident and its nature faster on flash.

nilfs2 may have a problem then device can be almost full. Garbage collection would be not simple for battery powered device. It should be delayed until N900 is on charger.

[stefanmohl]

@sr: I did see your previous post regarding aufs and unionfs. Great work in coming up with those results, even if the results themselves were kind of depressing! One thing I was wondering about though: If I got what you were doing right, you were using a ram disk backed by a flash drive; essentially your ram-disk was a write-through cache backed by the flash.

For the root partition problem, the situation is slightly different - both backing and front store are persistent. In this case, the fast flash stores the clean Nokia distro and the big flash stores all user additions. The big flash is superimposed over the fast flash so that reads see the big flash first, and writes only hit the big flash. This is more of a read-through rather than write-through scenario, if you get what I mean. I also assume that writes to files that already exist in the fast flash (i.e. modifications) go directly to the fast flash, rather than stick in the big flash like normal writes.

Do you think unionfs is so hopeless that there is no point in trying, or do you think you could have a look at how unionfs works in those conditions? Your basic idea is so pure and elegant that I really hope there is a way of saving it!

Does anyone here know anyone working on unionfs? Are there any Nokians here? Could Nokia consider putting in some work on unionfs?

[stefanmohl]

The brainstorm has been upgraded to "Under consideration". Can a moderator change the forum thread title to reflect this?

[ruskie]

Well I use unionfs-fuse myself for some alternate use and thus far was rather happy. I am considering using it on the N900 as well.

Of course something like nilfs2 would be nice on the flash as well but I think a more recent kernel would be a good idea first.

[titan]

regarding performance: on a HDD nilfs2 performance was not so great
http://www.phoronix.com/scan.php?page=article&item=ext4_btrfs_nilfs2&num=1

My benchmarks http://talk.maemo.org/showthread.php?p=504332#post504332
showed that real fs performance on NAND is not that much better than eMMC:
in the best case (compressed zero data) seq. write was 8 vs. 24 MB/s, seq read
19 vs. 30MB/s but with much higher CPU load for UBIFS.
I expect the differences to become much smaller with typical data.

If eMMC is not much slower, the firmware could copy the whole rootfs to a partition on eMMC at the first boot and from then on forget about NAND and boot from eMMC.
We need some benchmarks for booting from eMMC vs. NAND. Perhaps some of the differences are caused by the watchdog?

@slaapliedje: most boot stuff is in /usr and not in /bin or /sbin
see https://arch.nord.thebc.ch/wiki/index.php/N900_Installing_and_using_Bootchart

[ruskie]

@slaapliedje: most boot stuff is in /usr and not in /bin or /sbin
see https://arch.nord.thebc.ch/wiki/index.php/N900_Installing_and_using_Bootchart

Yeah sadly. It doesn't follow any of the normal desktop Linux policies in that regard. I wonder if this can actually be fixed by nokie with an update sometime. It would certainly improve the ability to hack it around without breaking it(had it happen a few times).

Anything that is needed to boot the system to a point of mounting /usr should be directly in / not in /usr.

Maybe have a USR_OK event in upstart that would tell the rest of the scripts that USR is ok. It could be a dummy as a start but could be replaced by one that would mount it.

[ruskie]

On a side note. Comment on the bootchart... that's so painfully slow.

[titan]

for those of you who want to play with unionfs: I have uploaded a new kernel to extras-devel non-free
package "kernel-flasher-maemo" (see http://talk.maemo.org/showthread.php?t=43420)

[javispedro]

regarding performance: on a HDD nilfs2 performance was not so great

Now consider that a whole lot of that eMMC throughput will be consumed by swapping in/out and you see why some people report that it is "insanely slow".

[javispedro]

Also, please consider recent improvements in the optification "art". Since it's already creating the symlink forest at package postinst configure time, patching dpkg to "autooptify" packages at runtime doesn't sound that far fetched.

IMHO, it's not an awful solution.

[soeiro]

Like I said, this isn't a workable solution. Then each reflash would require overwriting user data.

Not. An. Option.

What about changing the "flash procedure"? Let everything be in the big partition. The flasher tool could not really <i>flash</i> the whole device. But just replace everything but /home. User files would be preserved...

I think that having this absurdity of moving everything to /opt and still being at the edge of filling rootfs is actually "Not. An. Option". :)

[titan]

What about changing the "flash procedure"? Let everything be in the big partition. The flasher tool could not really <i>flash</i> the whole device. But just replace everything but /home. User files would be preserved...

The flashing is a blind write-only operation. There is no USB mass storage export for the flasher and therefore no mounting or analysis of the current eMMC state.
(correct me, if I'm wrong - the source is not public)

[@SR]

@SR, can you explain to me how to disable the watchdog on the device? you were saying you had considerable performance increase after disabling it?
No there was no actual performance increase. With original firmware you will not notice anything. The problem with watchdog was when I was mixing up file systems. When I disabled it performance was back to the level of original firmware.
If you still want to try disabling it you should use flasher with a command:
flasher-3.5 --set-rd-flags=no-omap-wd --set-rd-flags=no-ext-wd --enable-rd-mode

@sr: I did see your previous post regarding aufs and unionfs. Great work in coming up with those results, even if the results themselves were kind of depressing! One thing I was wondering about though: If I got what you were doing right, you were using a ram disk backed by a flash drive; essentially your ram-disk was a write-through cache backed by the flash.

For the root partition problem, the situation is slightly different - both backing and front store are persistent. In this case, the fast flash stores the clean Nokia distro and the big flash stores all user additions. The big flash is superimposed over the fast flash so that reads see the big flash first, and writes only hit the big flash. This is more of a read-through rather than write-through scenario, if you get what I mean. I also assume that writes to files that already exist in the fast flash (i.e. modifications) go directly to the fast flash, rather than stick in the big flash like normal writes.

Do you think unionfs is so hopeless that there is no point in trying, or do you think you could have a look at how unionfs works in those conditions? Your basic idea is so pure and elegant that I really hope there is a way of saving it!

Does anyone here know anyone working on unionfs? Are there any Nokians here? Could Nokia consider putting in some work on unionfs?
No i was not using ram disk. I was mounting internal flash (the one with 256Mb) and ext3 partition on top of it (the 2Gb one from internal mmc or from external card, results were the same, swap on the card does not really slow down the system especially if swappiness is set to the lower setting, right now my system boots from mmc and everything works fast). While creating union both of them were mounted with writable rights.
The default behavior for the union is to write all modifications to the file to the most top writable partition that already has the file and to create new files in the most top writable partition that already has the closest folder structure. So generally that would be internal flash as it already has all files. ext3 partition is empty at the first boot.
Aufs has settings to override this behavior. For example it can be set to create all new files on the partition that has most free space. With this settings generally everything is written to the ext3 partition (and the existing files from internal flash are copied up to ext3 partition when they are modified).
I did not find how to change the default priorities in unionfs so most of the files end up in internal flash. I am not saying that it is hopeless. If i could override it so that all write and create operations were on the ext3 partition and only rename operations were in internal flash (as rename is the hardest operation for copy up and anyway it does not require space).

[titan]

a reverse "optification" could be another solution:

mount eMMC on /emmc
bind mount top directories in / to /emmc/rootfs/*, e.g., mount /usr to /emmc/rootfs/usr
chroot to /emmc (later done during boot process)
create symlinks to /rootfs/*, e.g., symlink /rootfs/usr/bin/* to /usr/bin

this is basically unionfs for free and works fine:

mkdir -p /home/nroot/rootfs
cd /home/nroot/rootfs
mkdir bin boot cdrom dev etc floppy initrd lib media mnt proc root sbin srv sys syspart tmp usr var
for f in *; do mount -o bind /$f /home/nroot/rootfs/$f; ln -s rootfs/$f ../$f; done
mkdir -p home/user opt
mount -o bind /home/opt opt
mount -o bind /home/user home/user
chroot .

now, instead of symlinking usr we should exclude some directories
for which only the contents should be symlinked as packages might install new files in those
e.g. usr/{bin,lib,share} etc.

[shadowjk]

regarding performance: on a HDD nilfs2 performance was not so great
http://www.phoronix.com/scan.php?page=article&item=ext4_btrfs_nilfs2&num=1


Harddrives have very different performance than flash devices. Benchmarks on harddrives aren't comparable with benchmarks on mmc type flash. Benchmarks on the proper SSDs aren't comparable either.


My benchmarks http://talk.maemo.org/showthread.php?p=504332#post504332
showed that real fs performance on NAND is not that much better than eMMC:
in the best case (compressed zero data) seq. write was 8 vs. 24 MB/s, seq read
19 vs. 30MB/s but with much higher CPU load for UBIFS.


The table is a bit hard to read without fixed-width formatting, but isn't there over 5 times performance difference in the nand's favor? 5573 against the best, 970 fatpart?

Sequential read/write is a meaningless measure for anything except single-tasking video use...

The issues with mmc is kinda the same as with first generation SSDs, but worse. While they have sequential read speeds of hundreds of megabytes/s, and the OS boot "test" shows ok results, in real use they made the system "stutter". anandtech.com has an excellent series of articles on it...

[406NotAcceptable]

What about changing the "flash procedure"? Let everything be in the big partition. The flasher tool could not really <i>flash</i> the whole device. But just replace everything but /home. User files would be preserved...

I think that having this absurdity of moving everything to /opt and still being at the edge of filling rootfs is actually "Not. An. Option". :)

Exactly.

The flasher could zip and move the data to the PC, flash the drive, and then put the data back. Alternatively a SD card could do the same job.

With regards to the speed of SD cards noted earlier, 6 mbps is the minimum for class 6. Standard usage is around 12-15 mbps

[egoshin]

About eMMC performance -

Did you test it with random R/W? I suspect the reason behind a separate 256MB root flash vs 32GB eMMC is actually in performance of single block writes. If eMMC actual block is 256KB then it would have a big lag after a single block write and it may have a toll on writing of close blocks (not adjacent but close enough).

In other words - modification of small files in the same directory of ext3.

[titan]

my benchmarks are certainly not representative - if you know some better fs benchmark program let me know. sequential access can at least tell you something about the raw performance of the device.
However, I was surprised that UBIFS/NAND was not dramatically faster than ext3/eMMC
even in best-case scenario for the compressed UBIFS.
Also the CPU usage for UBIFS was 80-100% vs. 10-15% for ext3.
I consider that an important criterium for a mobile device, too.

@shadowjk: of course, HDDs are different, but that it performs so badly even on HDDs is IMHO not a good sign. again, we need more benchmarks...
the seek performance was 1100 vs 3700 on a fresh device but CPU usage was 14% vs 98%
http://talk.maemo.org/showthread.php?p=504332#post504332
not surprising if seeks in a compressed zero-filled file.

[egoshin]

IMHO, the difference in performance 3 times clearly justifies 256MB flash usage.

But I guess that "uncompressed" root can give a better difference, just because decompression takes toll on performance.

However, it requires more strict strategy of file/block allocation on 256MB because of space problem.

[stefanmohl]

I have rewritten my old, rather messy description of Solution #1 completely, and added an idea at the end of how to bootstrap all this after a reflash:

Solution #1: Place root in the large flash and link speed critical files from the fast flash

Posted on 2010-02-09 01:46 UTC by Stefan Möhl.

Basic idea:

o Have the root directory in a large partition in the big flash instead of in the small fast flash.
o Only files and directories that specifically need speed are placed in the fast flash with symlinks into the large root.

This is basically the same trick as optification, just the other way around. Instead of the fast flash as root, we have the big flash as root. Instead of moving large files and linking, we move speed critical files and link. The main advantage is that the default mode of operations is clean: Just install under the normal root hierarchy without any tricks like moving or linking. Specific files and directories from root may be moved and linked to improve performance if desired.

Optification means that we first install and leave the root files in the fast flash, and then move all packages we install in the future to /opt. With this solution we move just a few specific root files and directories (just those that really need it) to the fast flash, and leave all installed packages in normal locations in the big flash. So by default, we move only a specific and limited number of files, instead of moving all future files that will be installed.

Since packages are now normal, installed in their usual locations, we can now start installing all sorts of packages from the Linux community, as long as the binaries are compiled for the Arm processor. All of Debian, for example, has Arm support by default. We will now also have space to install the standard development environments, so if a package has no support for Arm, we can just install the source package and compile directly on our phones for the Arm processor.



Bootstrapping:

Bootstrapping the configuration after reflash: Since a partition in the big flash can not be reflashed without loosing all the data in the user partition (i.e. camera pics, music, etc), reflashes all have to go to the fast flash. Since we want the root to be in the big flash, we will have to move it there from the fast flash somehow. I suggest we do it the same way as we get root access when installing "rootsh": We "install" a large root partition through the app manager.

The "bigroot" package should be like the "rootsh" package: An officially endorsed and supported power user feature. When installing the package, all files are moved from the fast flash to the 2GB partition in the big flash, except the files that should stay in the fast flash for performance reasons. The bootloader is then modified to boot from the new root location, and the device is rebooted.

The "install bigroot" approach has a few advantages. First of all, it permits a rock solid state after a reflash, since it works just like things do now. Users can be warned and choose for themselves if they want to activate bigroot or not, so the pressure of perfect operation is much lower. Many packages won't be installable without installing bigroot, but packages that cater to casual users can still be optified like before. Bigroot is compatible with optification. Also, if your device bricks and you have to reflash, the old bigroot partition isn't cleared until you re-install bigroot. That allows you to rescue some files and configurations once your system is back up and running, and possibly figure out what went wrong.

The installer trick can also be used for applications by adding "speedification" packages. A "speedification" package for an application will move speed-critical files for that application to the fast flash, leaving links behind in the big flash. Similar to optification (though in the reverse direction), but manually activated or deactivated by the user. This way users get detailed control over which applications get to use the fast flash, and can optimize for their own usage patterns.

[stefanmohl]

The default behavior for the union is to write all modifications to the file to the most top writable partition that already has the file and to create new files in the most top writable partition that already has the closest folder structure.

Couldn't you create copies of the directories that you want to shadow in the ext3 partition first, before you apply unionfs? If you copy the whole directory structure (without the files) from the fast flash to the ext3 partition first, wouldn't all files stick in the ext3 partition then?

[shadowjk]

If the emmc is smart, writes within a 256k block would be fast... Problem is filesystems like ext3 sprinkle data far apart, and have layouts optimized to avoid fragmentation.

Basically filesystems designed for harddrives do all the heavy work when writing, they place various datastructures all over the harddrive in order to avoid fragmentation and so that when it comes time to read or find stuff, the data it needs is close together and not fragmented.
The idea is to take the performance hit when writing, since you'll probably read the data more than once anyway.

This on harddrives where sequential writes and reads are fast, and both random writing and random reading is slow.
On emmc, sequential writes and reads are fast. No difference there. The difference is in random writes and reads. Random writes are REALLY slow. Painfully slow. Random reads, however, are pretty fast.
Thus, the complicated and thought out optimizations that make regular filesystems fast on harddrives end up hitting the weakest point of emmc storage, hitting it hard.

The nilfs2 filesystem is almost like a gigantic journal. Data is written out mostly sequentially. Data is rarely replaced, new data is instead written, with a newer version number. This was the goal of nilfs2, to be able to jump back in the journal/log and restore the filesystem to a previous state. As it happens this ends up in nilfs2 being very slow on harddrives, because the data is all over the place when it's time to read or find data. On emmc these random reads aren't all that bad, and nilfs2 generates alot less random writes. Avoiding the biggest weakness of emmc makes it more than 10 times faster in my test.

LogFS is actually designed with flash based storage in mind, so it will be interesting to see how it performs.

[javispedro]

There's something that (as usual) bothers me with this brainstorm item. What are the benefits of most solutions?

Some of the benefits mentioned:
- Ability to install Debian packages
- Ability to build Debian or Maemo packages

... are IMO more easily done with a Debian chroot/Maemo cloned chroot, which has more benefits:
- ability to have a build environment that is not exactly the same version as your phone environment -- SDK release 0 in chroot for maximum compatibility package building vs PR1.1 in the phone
- ability to trash busybox and install bash and coreutils
- ability to install _unmodified_ ANY Debian release binary packages in Debian chroots
- plus ability to still boot your usual runtime system if the chroot is trashed for whatever reason
... vs IMO a minor tradeoff (more memory usage as you need to load both the chroot and Maemo libc + other libraries).


In my opinion the only one that has a clear benefit is solution #2 (since it means you'd be able to reflash the base firmware without reloading the installed apps!), but then it has quite an important tradeoff: even more Debian upstream compatibility is lost. Yes, changing a package's prefix is not 1-2-3: from all the packages I've ported to Maemo, 50% had some kind of hardcoded path.
I don't mind having to do this since I can report those to upstream, and getting those fixed is always nice, so solution #2 gets my vote (even if it clearly contradicts the brainstorm subject since it actually decreases debian compatibility).


And again I mention the recent improvements in the optifier. The idea is that a developer doesn't even have to know that the autobuilder is optifying.

Also, none of the options save for #3 guarantee that end users won't get their rootfs filled somehow. Then again I'm biased somehow since I've never managed to fill mine.

[stefanmohl]

@javispedro

There certainly is a good case to be made for keeping optification the way it is. If you really feel that way, make it your solution in the brainstorm!

I, personally, feel that even though chroot is a possible alternative, it would be much better if we could just install Debian packages by default. Further, if we could install standard Debian packages by default, that would in no way prevent us from still installing a chroot too, if we like to, so all your benefits of having a chroot will still be available. - Oh, and by the way, I want to trash Busybox in favour of coreutils even in the standard Maemo distro, we just need the space to do it first.

I have seen that this has been discussed a few times, but I did try and couldn't find a Brainstorm or Bugzilla about it. I am new to the Maemo community, and not quite sure how Nokia decides what is officially supported by Maemo. However, it seems to me that the Brainstorm and Bugzilla tools are the main ways of consolidating community disposition. There might be other, less formal routes that come straight from discussions in the forum, but if so I am not aware of them yet.

If Brainstorm/Bugzilla are the main routes, the fact that the issue has been discussed so much that people are getting tired of the subject must mean that it is high time to bring it to the vote, right?

[egoshin]

Solution #1: Place root in the large flash and link speed critical files from the fast flash

The problem with this solution is that many critical files can't be symlinked. It is especially with configuration files, usual way to handle it - create a new version and replace the old one.

Symlinking of directories can help but again it can be a problem with some software and nobody knows which one exactly.

Additional (but not critical) issue - read speed of small files (usually - shell scripts in /etc or localization files in /usr/share/locale or Hildon icons or X11 fonts) initially would be degradated until it sits in kernel cache. Because that files are used pretty often in "search" mode (file-by-file) it would visibly increase an overall access time.

[javispedro]

Oh, and by the way, I want to trash Busybox in favour of coreutils even in the standard Maemo distro, we just need the space to do it first.
But, at least on Diablo, some of the bootscripts didn't work without busybox. And even on Fremantle you are limited to really old versions of Debian tooling (think Sarge/Etch), with no debconf nor any other stuff. Seriously, if you want to try to "make Maemo more Debian" you're going to have to fight much greater issues than optification (which is relatively so simple it can even be done automatically!).

This still doesn't mean optification needs to be keep as is, though, since the rootfs free space is getting lower with every firmware update, and thus the "forest of symlinks" "hard limit" is getting tighter every day (which is why I didn't add it as solution, and instead voted for #2).

[stefanmohl]

The problem with this solution is that many critical files can't be symlinked.

Sure, but that problem is equally there for optification. Since we will have to split across two filesystems in some way - either optification or something else - we will have to tie things together with symlinks or possibly mount-binds however we do it. All I am suggesting is that we do it the other way around, so that we do the linking once, at the start, instead of doing it all the time.

Seriously, if you want to try to "make Maemo more Debian" you're going to have to fight much greater issues than optification

Well, we need to start somewhere, right? And optification is a showstopper for almost all Debian packages (at least if you want to install more than just a few). Let's do the others when that time comes, and focus on optification for now. Even if "all" we get is a big rootfs, I would consider that a substantial improvement. :-)

[javispedro]

And optification is a showstopper for almost all Debian packages
I hope you know about maemo-optify-deb. Save for bugs/TODOs (which TBH are not-as-low-as-I'd-like; see jebba's auto-optification of the entire rebuilt* debian etch archive), it works on most binary packages.

* Cause if you're not rebuilding packages at all you're up for some trouble sooner or later.

[egoshin]

Sure, but that problem is equally there for optification

Look at this - http://www.linux-mag.com/id/7378/2/

"Summary" section:

One area that FS-Cache could prove to be of future use is for caching local file systems. Currently, file systems rely on the kernel for caching data and scheduling for writing/reading to/from the storage. This caching is not directly under your control. But if a local file system can be modified to use FS-Cache then you could use a small but very fast SSD or even a Ramdisk for caching of data.

It gets off symlink problem as is as bootstrap problem.

[titan]

the difference between NAND and eMMC is much smaller than the one between eMMC and hard disks.

One area that FS-Cache could prove to be of future use is for caching local file systems. Currently, file systems rely on the kernel for caching data and scheduling for writing/reading to/from the storage. This caching is not directly under your control. But if a local file system can be modified to use FS-Cache then you could use a small but very fast SSD or even a Ramdisk for caching of data.

[stefanmohl]

I hope you know about maemo-optify-deb. Save for bugs/TODOs (which TBH are not-as-low-as-I'd-like; see jebba's auto-optification of the entire rebuilt* debian etch archive), it works on most binary packages.

* Cause if you're not rebuilding packages at all you're up for some trouble sooner or later.

Sure, but after that, it isn't a Debian package any longer, right? After that, it is an optified Maemo package. And, I think even you will agree that it is better if we can install packages straight as they are, rather than having to run even a very nicely automated process first. At the very least it would mean that package maintainers have one thing less they need to learn and understand before porting a package to Maemo.

Obviously, I don't expect all packages from one distribution to work out-of-the-box on another distribution, but if we can get rid of the optification requirement, there will actually be quite a lot of Debian packages that work as they are. With optification, as we all know, that is not the case.

[javispedro]

Obviously, I don't expect all packages from one distribution to work out-of-the-box on another distribution, but if we can get rid of the optification requirement, there will actually be quite a lot of Debian packages that work as they are.
Again, I think that's a bad thing to do, and the chances for something to break are _high_. See the common question of using Ubuntu packages in Debian.

[stefanmohl]

Look at this - http://www.linux-mag.com/id/7378/2/

"Summary" section:

One area that FS-Cache could prove to be of future use is for caching local file systems. Currently, file systems rely on the kernel for caching data and scheduling for writing/reading to/from the storage. This caching is not directly under your control. But if a local file system can be modified to use FS-Cache then you could use a small but very fast SSD or even a Ramdisk for caching of data.

It gets off symlink problem as is as bootstrap problem.

Yes, I agree, a cache system would get rid of the symlink problem. I am merely arguing against optification. My suggested solution is meant to be as simple as possible, it is actually just a variation of how optification works now. Instead of optifying everything, we just "negative-optify" root once and then we can install everything else as normal.

Your caching suggestion is much more elegant and advanced, and if it can be made to work with stability, it would be better. I gave it my vote as soon as I saw it :-)

If you haven't done so already, you should look at the posts by @SR too. He hasn't made a formal Brainstorm Solution yet that I am aware of, but his ideas are very similar to yours. The main difference is that your solution uses an automated cache policy, whereas his solution assumes manual decision of what goes in the fast or big flash.

[stefanmohl]

Again, I think that's a bad thing to do, and the chances for something to break are _high_. See the common question of using Ubuntu packages in Debian.

Yes, as you yourself quote, "I don't expect all packages from one distribution to work out-of-the-box on another distribution". But even if most packages break, Debian is so much larger than Maemo that we will have a huge increase of working software if only a small proportion of Debian would work. Please forgive me if I misunderstood, but to me it seems that your are arguing that just because some packages break, we might as well break them all.

[SubCore]

But even if most packages break, Debian is so much larger than Maemo that we will have a huge increase of working software if only a small proportion of Debian would work.

do you have a background in debian-style packaging?
i don't think you do, because if you did, you should know that "optification" really is the smallest part of it. it's the part you consider AFTER you've addressed the big issues.

and to illustrate, i'll give you a simple example:

say you wanted to install bc (http://packages.debian.org/stable/math/bc) from the debian stable repository.
although it's a very simple commandline app, the very first dependency of that package will kill you. it requires libc6 > 2.7.1, but the version on the N900 with 51-1 is 2.5.1
if the updater were to install the higher libc6 package, all hell would break loose and you'd have a bricked device. simple as that.

and there are tons of examples like this.
in the easy debian chroot, you can easily provide a proper version (libc6 has v 2.9-4 with the image i have installed currently), but NOT in the native Maemo environment.

it's just not worth the effort, a chroot approach is the cleanest and most reliable way to support multiple distributions. and it works great! (kudos again, qole :) )

[titan]

i don't think you do, because if you did, you should know that "optification" really is the smallest part of it. it's the part you consider AFTER you've addressed the big issues.

correct. downgrading to the outdated Maemo5 components is the biggest problem.
I thought this brainstorm is about avoiding a full rootfs and not about apparently simpler
installation of packages from other distributions?
Getting rid of optification could make porting a little bit easier but it could also mislead naive users to think that they could install normal Debian packages.

[stefanmohl]

I thought this brainstorm is about avoiding a full rootfs and not about apparently simpler
installation of packages from other distributions?


You are right Titan, we are side-tracking. I think we all actually agree: It is never easy to move packages between distributions, and though getting rid of optification makes some difference, that is a far cry from making Maemo into Debian.

Sorry about loosing focus, I'll drop this subject from here on.

[stefanmohl]

So, is this it? Do most people in this forum want the small rootfs and optification? Or have we emptied all possible suggestions? Does the brainstorm contain all solutions we can think of?

I am still seeing new threads of this kind:
http://talk.maemo.org/showthread.php?p=529725#post529725

Or this:
http://talk.maemo.org/showthread.php?p=530088#post530088

Both started today

[titan]

Only Nokia can make the small but necessary changes to implement solution #2 in a SSU.
I have a patch for debhelper7 which makes it easier for developers to set an arbitrary prefix
for their package installation and which could lead to a cleaner /optification.

I'm also working on a implementation of solution #5, which could also interesting for people
who want to run different firmware versions or install the SDK, while keeping most files on rootfs.

Regarding NILFS2, we have some disappointing benchmark results and facts at
http://talk.maemo.org/showthread.php?t=43420

So, is this it? Do most people in this forum want the small rootfs and optification? Or have we emptied all possible suggestions? Does the brainstorm contain all solutions we can think of?

[titan]

A first implementation of solution #5 is available as part of the Moebian project
http://talk.maemo.org/showthread.php?p=559122#post559122
it basically moves / to /home or any other partition and keeps the /usr files on NAND.

[wongdg]

If you read the brochure on Samsung OneNand, the 256MB model in fact is the lowest end model of the family. The siblings include 512Mb, 1Gb, 2Gb, 4Gb, and the largest 8Gb.

Has anyone contemplating swapping out the 256Mb for the 8Gb. I believe they should be pin compatible, just some fine art of soldering needed. Can such a trick be coordinated? and how much? Any possibility to get Nokia to do it for a bunch of us eager user to go full linux.

I would most prefer to have a full debian phone with a decent battery life.

Wilson
Supporter of Debian linux

[cb22]

I recall reading earlier in this thread that the NAND is located inside the OMAP3 core...

[pelago]

I recall reading earlier in this thread that the NAND is located inside the OMAP3 core...

That is correct. It won't be possible to change without changing the processor, which is practically impossible.

EDIT: Turns out that I wasn't correct here.

[Matan]

I recall reading earlier in this thread that the NAND is located inside the OMAP3 core...

No, that is wrong. The nand is soldered directly on the OMAP, a method called PoP (package on package), but they are still different packages.

[wmarone]

I recall reading earlier in this thread that the NAND is located inside the OMAP3 core...

That is correct. It won't be possible to change without changing the processor, which is practically impossible.

Actually, being a Samsung part it is located in the same package as the 256MB of SDRAM, which is soldered on top of the processor. Theoretically you COULD replace it, but it would require access to some rather expensive hardware, and it is rather hard to buy such chips individually.

Also, wongdg, you may need to check your numbers as they typically sell flash memory in MegaBITS and not MegaBYTES, so divide by 8 to get the maximum capacity; Samsung's 8Gb (1GByte) didn't exit sampling and enter production until well after the N900's design cycle ended.

The chip in the N900 is a 2GigaBIT device, the one in the DROID and N1 are 4GigaBIT (512MByte,) and nothing I know of uses the 8GigaBIT (1GByte) chips.

[mrrg]

Hello Ive been reading through some posts here..and this solution might be stupid call it solution D

So rootfs is to small 256MB and moving it to the /home partition in large flash will cause user data to disappear at reflash.

So what about repartitioning the large flash reducing the size of the 27GB partition creating a new root partition on the large flash lets say 1GB.

Now reflashing this partition should not overwrite any user data ?

Perhaps /dev/mmcblk0p1 could be shrunk and then remove the empty primary partition /dev/mmcblk0p4 and add it again just after /dev/mmcblk0p1 and make this the new rootfs.

Dunno how to do the last thing though.


Ohh I just realized solution D2

You could scrap the swap partition 768MB and use this as a new root partition.

Swap can then be added as a swap file (instead of a partition) inside the /home directory.

[Bundyo]

If you read through this thread you would have known that the very slow internal flash is no option. ;)

[dbrodie]

Well what about getting bcache running on this device? It still is very alpha from what i've seen and we have the problem of different kernel versions (maybe on a meego kernel?), but that should solve the problem, right?

[stefanmohl]

I think that the solutions from this thread boil down to four basic types:

1) Put most of the system in the big flash and manually selected files/directories in the fast flash (Brainstorm solutions #1 and #5)
2) Keep optification, but clean it up so that packages end up completely in /opt, without touching rootfs (currently, optification places some files in rootfs and some in /opt) (Brainstorm solution #2)
3) Put the whole system in the big flash and use the fast flash as an automatic cache (Brainstorm solution #3)
4) Don't run Maemo, run Debian (or something else) in a chroot

Of course, a fifth option is that no solution is needed:

5) Optification is not a problem, the systems work well and we have enough space in the rootfs as it is.

Some personal opinions of what I prefer after the discussions in this (and other) threads:

Out of (1) and (3), I believe mostly in (1), since I don't believe that automatic cache schedules will be precise enough to make me happy. However, to make (1) work, we need to figure out precisely what files/directories really need to be in fast flash. As an illustrating anecdote: If you move /lib/dsp to the big flash, the phone will still work fine, but CPU usage will go through the roof. Titan has done some nice work linked from Brainstorm Solution #5 (https://maemo.org/community/brainstorm/view/remove_256mb_limitation_of_the_rootfs_partition_in _the_n900/)

(2) looks nice and clean, but I suspect that there will always be apps that need the performance of the fast flash, so reserving the fast flash only for the system is a bit too much of a limitation

I don't like (4) very much; I want to run Maemo, not Debian. If some Debian apps can easily be made to work under Maemo, that is a bonus. I don't mind if they are a few releases old (like back in Etch). Heck, I am still running XP on my Windows box at work, and that is a lot older! The main point is, I don't want to be running two different systems at the same time.

In my opinion, (5) is plain wrong, as seen from the large number of users that continue to have problems with limited rootfs space. This fact alone makes (5) wrong and is my main reason for starting this brainstorm.

Please go to the Brainstorm (https://maemo.org/community/brainstorm/view/remove_256mb_limitation_of_the_rootfs_partition_in _the_n900/) and vote for your preference!

[mrrg]

I guess I was thinking that it should be like Brainstorm solutions #1 and #5
but this way you would get a partition free for it...

Also some people say the difference in speed between the flashes do not differ so much... or I am just plain crazy.. haaahh!!

[egoshin]

I did my own little research with flash access times. I did it via multiple usage of command

time dd if=src of=dest bs=64k count=3000

where 'src' and 'dest' were combinations of /dev/mtd5, /dev/mmcblk0p3, /dev/mmcblk1p2 (I set swap here) and /dev/null or /dev/zero.

The results are (time in seconds, measured with stock PR1.2 kernel):

MTD5 (flash-on-top-of-CPU, root FS):
READ -
wall time = 9.93,
sys time = 9.79
battery current=1400

/dev/mmcblk0p3 (regular N900 swap space, part of 32GB internal flash):
READ -
wall time = 9.87
sys time = 2.57
battery current = 1223, repeated=1870
battery current=1252
WRITE -
wall time = 11.27
sys time = 2.28

/dev/mmcblk1p2 (a partition on micro SD, class 6):
READ -
wall time = 14.94
sys time = 3.44
battery current=977, repeated=1620
battery current=1100
WRITE -
wall time = 19.23
sys time = 3.10

----------------------------------------------

So, the interpretation - the current usage of MTD5 as root file space is useless from many point of view - the Read performance is the same as internal 32GB flash BUT (!) - CPU usage is overtop!

Actually, I strongly suspect that MTD5 access performance is limited by CPU performance (due to compression in UBIFS?) - wall time == system time.

The move of root FS from MTD5 into 32GB may benefit not only in space but CPU usage and battery life. We need only switch off the disk block caching for file content - it seems that if kernel finds disk block in cache it copies block from that cache and that consumes MORE energy than DMA read from flash chip: repeated reads take more battery current than first one.

After root FS moved to 32GB flash the MTD5 can be used without compression for first order swap, it would greatly accelerate overall N900 performance (today swapout-swapin of huge behemoths like modest/gst-video-renderer may take up to 10secs during call answer). Or use it as file cache (my proposal #3).

EDIT: Battery current updated with more accurate measurement (N900 set offline). No significant fluctuation anymore.

[shadowjk]

What unit is your battery current figure in? How did you measure it?

The big difference for nand vs emmc isn't in sequential read/write.. Whereas the emmc can do something like 4-8Megabyte/s write per second sequentially, the built-in flash translation layer can not at all deal with random write (such as swap, /opt and MyDocs), and the speed for 4K random writes degrades to a few hundred kilobytes.

The OneNand has no FTL, so we can manage it ourselves, and we have a 600MHz cpu and 256 megs of ram, plenty of room to do something intelligent. As a result, the onenand with ubifs is a few magnitudes faster at serving IOs from a multitasking demand-paging operating system.

Reads are same magnitude in speed, as long as there are no writes interleaved.

[egoshin]

What unit is your battery current figure in? How did you measure it?

sleep 15
time dd if=$1 of=/dev=null bs=64k count=3000
time dd if=$1 of=/dev=null bs=64k count=3000
time dd if=$1 of=/dev=null bs=64k count=3000
time dd if=$1 of=/dev=null bs=64k count=3000
time dd if=$1 of=/dev=null bs=64k count=3000
time dd if=$1 of=/dev=null bs=64k count=3000
time dd if=$1 of=/dev=null bs=64k count=3000
time dd if=$1 of=/dev=null bs=64k count=3000
cat /sys/class/power_supply/bq27200-0/current_now

(bq27x00_power.ko module is from Matan's collection)


The big difference for nand vs emmc isn't in sequential read/write.. Whereas the emmc can do something like 4-8Megabyte/s write per second sequentially, the built-in flash translation layer can not at all deal with random write (such as swap, /opt and MyDocs), and the speed for 4K random writes degrades to a few hundred kilobytes.

Swap is not an issue at all - it is read and written in big blocks and it is located NOW in 32GB flash (non NAND!). Also I tested files on /home/user Ext3 - the file read speed is about the same - 19MByte/sec. And reading uncachable /usr/lib/locale/locale-archive from UBIFS gives 18MByte/sec. I don't see a difference besides CPU resource and battery usage.

Short files from /etc and gconfig can land in kernel cache pretty soon. And we may combine file caching in MTD2 for that - it would get the advantage of fast short access if UBIFS implementation works fine for short files.

Besides of that it is possible to use not ext3 but some more flash-friendly FS - LogFS is released in May, for exam.

The OneNand has no FTL, so we can manage it ourselves, and we have a 600MHz cpu and 256 megs of ram, plenty of room to do something intelligent. As a result, the onenand with ubifs is a few magnitudes faster at serving IOs from a multitasking demand-paging operating system.

May be. But I don't see it now - my battery depletes too fast and I know the reason now. Personally, if there is a choice I prefer to have some slow disk access but much less CPU expenses, in my long UNIX experience with old and slow disk equipment it handles file access very well because of kernel buffer cache, especially in multitasking. But I don't think we would have a slow access.

Reads are same magnitude in speed, as long as there are no writes interleaved.

I have a home server which is configured to have root FS read only besides some critical files for X11 which are symlinked to /var and the whole /var is located in Generic Compact Flash. It is done to prevent disk spinning and works very well, at least faster then N900 :)

So, don't worry about writes... well, gconf should be killed or replaced because it writes on any bump. But we can symlink that stuff into OneNAND.

[shadowjk]

Just try gpodder on your N900 with a moderately big datbase (say 100-200 podcasts). When you start a download, the 8 megabits coming from 3g to MyDocs, the 4k random write/read swap activity by the 8mbit io pressure making linux swapout to increase cache, and the random read/write to ext3 to update databases results in half-minute long stalls..

My home server also runs on flash. With ext3, the squid proxy server stalls for half a minute at a time when the flash slows down. With nilfs2, which by accident is quite performance optimized for flash, there are no stalls and doing a backup of the squid cache dir is also about 8 times faster than with ext3.

[Matan]

What unit is your battery current figure in? How did you measure it?


We don't know the units, as they depend on the resistance of a specific resistor on the board. But the values are linear, so they are useful for such comparisons.

The big difference for nand vs emmc isn't in sequential read/write.. Whereas the emmc can do something like 4-8Megabyte/s write per second sequentially, the built-in flash translation layer can not at all deal with random write (such as swap, /opt and MyDocs), and the speed for 4K random writes degrades to a few hundred kilobytes.

The OneNand has no FTL, so we can manage it ourselves, and we have a 600MHz cpu and 256 megs of ram, plenty of room to do something intelligent. As a result, the onenand with ubifs is a few magnitudes faster at serving IOs from a multitasking demand-paging operating system.


But the swap is actually on the eMMC, so I don't really get your point. In normal activity there is very little write access to the root filesystem.

Do you have some numbers for "a few magnitudes faster ..."? It sounds too incredible to be true.


So, the interpretation - the current usage of MTD5 as root file space is useless from many point of view - the Read performance is the same as internal 32GB flash BUT (!) - CPU usage is overtop!

Actually, I strongly suspect that MTD5 access performance is limited by CPU performance (due to compression in UBIFS?) - wall time == system time.


Using root file system on eMMC is slower, even though the hardware is better, as those benchmark show, due to concurrency with swap access also on eMMC.

The best options I can think of are:

Use a microsd card with root filesystem on eMMC and swap on uSD (or vice versa)
Use root on eMMC and swap on NAND.


Both options should give better performance than the current setup and give a larger (configurable) root file system size.

The downside to first method is probably increased power usage due to uSD card.

The downside of the second is limiting of swap to 250MB rather than the current 768MB.

[egoshin]

The best options I can think of are:

Use a microsd card with root filesystem on eMMC and swap on uSD (or vice versa)
Use root on eMMC and swap on NAND.


The downside to first method is probably increased power usage due to uSD card.

My measurements show the DECREASED power usage with uSD (see previous message). It is one of latest uSD Class 6 cards.

The downside of the second is limiting of swap to 250MB rather than the current 768MB.

I tested the two swaps for single system, it works - just add another swap partition and kernel puts it as "secondary" priority swap. It works fine with automatic extension to the 2nd layer swap.

But I am afraid the usage of NAND has a disadvantage in battery performance. I observed high CPU usage for reading just raw 64KB data blocks from /dev/mtd5 and it seems that NAND performance is actually limited by CPU. Plus higher battery usage.

It may be because of block device emulation (not used in UBIFS) but this layer should be used for swap operations too.

So, it may have a sense to symlink or file-cache small files to NAND + UBIFS. For small file access it may be a good trade-off (Battery vs performance).

[shadowjk]

Caching takes care of all our concerns regarding small file access and so on, and also swap. However, this is of course only possible if we have enough RAM to cache things in ;-)

The few magnitudes claim comes from something called write amplification. The native flash block size is some 256kilobytes. Writing a 4k sector (as the OS often does) causes the emmc/sd to erase a new 256kb block from a pool of spares, (if there isn't one already erased available), read in 256kb, modify 4k of it, write it to the new block, put the old block into the spare pool.

Thus, a 4k write request from the OS was turned into a 256k read, 256k erase and 256k write. 256/4 is a factor 64 already :)

On the emmc/sd, 256k physical blocks are linearly mapped to 256k logical blocks.

With nand we can do it smarter, while the erase size is huge 256k, an erased block doesn't need to be completely written in one go. We can write 4k and fill in the rest later. We have the cpu to map blocks of 4k nonlinearly onto the 4k flash, so that even if the filesystem wants to update a table at the start of disk, a file entry in the middle, and some data elsewhere, we turn it into 3 writes into the same block.
This is roughly what modern SSDs also do, and they have ARM cpus and 64 or more megs of ram to do it, which explains why there are no sd cards doing clever stuff.

Now, normal filesystems are optimized for rotational media. There all kinds of non-sequential access becomes slow. So, they operate on the idea that data is read more often than written, and thus spend alot of effort on placing the data optimally, so that related data is close together. This is exactly the opposite thing you want to do on a flash based device.

nilfs2 and logfs, however, are very good for performance on emmc/flash :)
(and ubifs, but that can only be used on nand)

So yes, if we can use LogFS, and if we can have swap on its own device, or someone writes a new loop block device specifically to optimize swap on flash, having non-media system files on emmc would be doable..

Without logfs though, I suspect we get more of all sortsof "stuttering" people cmplain so much about even now :)

[geneven]

So I have a 32 gb I think class 10 card. Any chance I could use it for this? How, exactly?

[stefanmohl]

I have been wondering about that myself, could anyone confirm if 32GB cards work without problems? The phone is officially only rated for 16GB cards AFAIK.

Edit: found a thread that verifies that 32GB microSD cards work:
http://talk.maemo.org/showthread.php?t=57508&highlight=32GB+microSD

[The_Solutor]

I faced a similar problem running a regular linux installation on top of a RS-MMC (damn slow, and too little for a regular linux distro)

I tried almost every solution available (unionfs, aufs, symlinked directories and so on) and finally I tried BTRFS with compression enabled and SSD mount opition.

The result is simply amazing, the system is incredibly fast, the bootime is almost a quarter than the one over a regular ext3 FS, and there is no space wasted as happens with the readonly fs overlaid by writable fs, trough aufs or unionfs.

So I vote for completely ignore the 256MB NAND and use a partition on 32 GB eMMC formatted with compressed BTRFS for the whole FS.

Eventually the 256 MB nand can be used for caching purposes, as suggested before.

The flashing problem can be eventually solved flashing not directly the emmc but the 256MB nand flash as happens now.

A script on the first boot can format the emmc partition and then copy everything on it (adjusting the configuration files when needed), so no worry for the user data on the vfat section.

[egoshin]

I am able now to run Maemo 5 PR1.2 from internal eMMC with working cell phone after some modifications. So, UBIFS is not used after boot and root is located in eMMC.

No root FS limitations, anything works... well, one small problem still exists but there are workarounds.

First impression performance: something different. Sometime it can be a little slow but in many cases it has definitely a faster reaction. Of course, I set swap into external MMC.

Looking now into battery performance. Without GSM it is of course very good, but now I have it with working GSM/3G/WiFI, so - still collecting measurements.

[stefanmohl]

Your battery results are going to be very interesting. I had a strange experience with moving /lib/dsp to the eMMC. Granted, it might have been a fluke, but when I did so, the CPU got stuck at 500MHz and never went below. When I moved the directory back, it fell back to normal.

[egoshin]

OK, couple of day passed and my N900 works fine from eMMC (swap on uSD).

Now I am ready to say that there is no notable difference in power consumption. Actually, it looks like it is a little better (around 20%) but it is in error margin - I rebooted multiple times and it takes battery resource.

Speed. Sometime it is worse but most of it during reboot until I attach uSD swap instead of /dev/mmcblk0p3. However, in day-to day clicks it looks better during changing menus etc.

No problem with phone call receiption - I means, no delays, no sound distortion etc.

No problem with e-mail - it looks like it works a little better: I have a big e-mail account (thousands mails) and it handle it's retrieval now without any problem. In vanilla boot from UBIFS it can sometime hangs (lack of memory) or do it long time.

No any problem with video playback. At least a couple of my video files play smooth.

Of course, no problem with browser or RSS feed.

I even tested google latitude - it works fine too.

It is possible to note an existence of some intermediate windows during starting some applications (like e-mail) - it blinks a fraction of second and disappears before I can read it's content during e-mail application start. But overall reaction is the same.

Two problem still exist with MCE - it doesn't work correctly with power button and doesn't finish "screen slide unlock" (MCE is a buggy application anywhere). But side slider works fine. And there is some another problems with power-button menu like some woodoe needs to be done to see it and switch OFF position doesn't work. All of them are easy to workaround and for now it is not a stop-gap.


But biggest advantage - 2GB root FS, of course - no problem with installing new applications...

[egoshin]

Yet another day passed and I am able to compile and build the whole kernel in my N900 - full power development cycle.

Wall time - 3h 1m 21s.
eMMC space - 943MB.

I did a couple of phone calls in a middle without compilation interruption :) Skype was working too - anything as usual. Delayed reaction on button press was expected and I observed it but it was reasonable.

N900 was warm but not very. It took around 3/4 of battery resource.

[AlMehdi]

I wish i had the guts and knowledge like you Egoshin. Good work! ;)

[egoshin]

I prepared tool scripts and published it here - http://talk.maemo.org/showthread.php?t=59374

After applying this scripts you can have a 2GB (or 32GB - depending from your /home partition size) root file system.

[stefanmohl]

This is really great Egoshin! You should package the scripts and put them in extras-devel as soon as some people have tried them out. Apart from easy availability and official endorsement (as it moves into extras-testing, etc), it will also let un-optified packages simply be made dependent on your package and people could still install them. Oh, and perhaps find a name that is a bit more descriptive than M32GB... ;-)

[appnss]

First of all, sorry if I am posting about something that have already been covered. I have had my N900 for less than a month so I may have missed some older threads/posts even as much as I have tried to read everything related before posting.

At the first moment I saw the rootfs limitation on N900 I thought about the simple yet elegant way it is solved in OpenWRT:

A root filesystem (squashfs) + an overlay with mini_fo (jffs). This gives a readonly basic install plus an additional space for modifications.

I have been holding from trying before I read all I can about this subject in the forum, considering that if it hasn't been done yet must be for a good reason.

Anyway, before going any deeper into this subject, I would like to understand what is the reason because this option was ruled out.

I have only been able to to find a reference in a several months old post from @SR:


I've been experimenting with aufs, unionfs and mini_fo for a while to mount ext3 partition on top of internal memory. All of them have problems.
One common problem was that both of the file system partitions needed to be writable. It is because of the complex algorithm when renaming non empty folder from read only file system (aufs page has a good explanation on that). For example application managed did not work because of this problem. So mini_fo did not work at all.


What is the reason that both fs need to be writable? The idea would be to have "original" root fs mounted as read only and a writeable overlay using mini_fo the same way is done in OpenWRT.

Reads would still be faster as mini_fo doesn't have a big impact performance (needs to be confirmed in this case though) and writes would be directly into the overlay, which wouldn't be significantly slower than writing to the emmc.

About renames... well, that would be obviously much slower but... I don't see a reason for almost any rename of the original binaries much less a non-empty original dir.

Also, simply not mounting, or clearing the overlay would leave a perfectly clean fresh device. And mounting different overlays allows for some easy multiple system.

P.S.: Perhaps Egoshian way has better performance, but anyways I still I would like to understand why this simple solution was ruled out.

[DrWilken]

I don't quite get it.

Why would You want to use the 2GB "/home" partition as Your / partition ?

Instead of using the 256MB NAND + 2GB /home You end up with "only" 2GB in total (+ possible trouble upgrading to next PR)...

On PR1.2 I haven't had any problems with / being filled up. I do, however, believe that /home (containing /home/opt) is a bit restricted in size (2GB). It can easily be filled up by installing lots of games etc.

I'd rather just shrink the FAT32 filesystem by let's say 3GB and add them to /home.

This is what my disk usage looks like:


N900:~# df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 228M 156M 68M 70% /
ubi0:rootfs 228M 156M 68M 70% /
tmpfs 1,0M 76K 948K 8% /tmp
tmpfs 256K 88K 168K 35% /var/run
none 10M 72K 10M 1% /dev
tmpfs 64M 4,0K 64M 1% /dev/shm
/dev/mmcblk0p2 2,0G 1,6G 308M 84% /home
/opt/pymaemo/usr/lib/python2.5
2,0G 1,6G 308M 84% /usr/lib/python2.5
/opt/pymaemo/usr/share/pyshared
2,0G 1,6G 308M 84% /usr/share/pyshared
/opt/pymaemo/usr/lib/pyshared
2,0G 1,6G 308M 84% /usr/lib/pyshared
/opt/pymaemo/usr/share/python-support
2,0G 1,6G 308M 84% /usr/share/python-support
/opt/pymaemo/usr/lib/python-support
2,0G 1,6G 308M 84% /usr/lib/python-support
/dev/mmcblk0p1 28G 3,0G 25G 12% /home/user/MyDocs
N900:~# du -sh /*
980K /bin
0 /boot
12K /bootmenu.sh
0 /cdrom
72K /dev
6,3M /etc
0 /floppy
4,6G /home
0 /initrd
24M /lib
0 /media
0 /mnt
1,3G /opt
0 /proc
544K /root
2,0M /sbin
0 /srv
0 /sys
0 /syspart
76K /tmp
288M /usr
35M /var


So IF I were to repartition I'd rather see something like this:
/ 256MB ubifs on NAND
swap 768MB/1GB on eMMC
/home 1GB on eMMC *
/opt 4GB on eMMC *
(/usr 512MB on eMMC *)

* Or You could just make /home 5GB (though I'm not that big a fan of the /opt symlinking - it's just ugly). I'd also like to get rid of the /opt/pymaemo symlinks. Maybe we could create a /usr filesystem too (~512MB).

And the rest for the FAT partition.

EDIT -> Guess that *solution* is discussed here (http://talk.maemo.org/showthread.php?t=37869).

[javispedro]

What is the reason that both fs need to be writable?

Because a Nokia-blessed solution would have needed to accommodate for over the air updates, which require a writable rootfs.

Of course, for your own use you can completely ignore updates.

[appnss]

Because a Nokia-blessed solution would have needed to accommodate for over the air updates, which require a writable rootfs.

Of course, for your own use you can completely ignore updates.

If that's the ONLY reason, IMHO this option was wrongly ruled out.

For all purposes, the rootfs (not the original, but the original + overlay combination) is perfectly writable. Only for a phisical reflash some steps would be needed to do it correctly, but again, in the current configuration a phisical reflash would also leave /opt somewhat out of synch.

In fact, a process to prepare for a reflash / resynch can be made before the update using the mini_fo approach. I had to do it many times with OpenWRT and never had a problem.

I have read about a watchdog proccess that detects delays in writing during boot and hangs, which could be of some concern, but that also seems to have been solved by Egoshin.

I would leave things at they currently are on nokia side, making now a change is nonsense and would lead to some troubles, but I think the mini_fo modification is perfectly compatible and would add the advantages of:

1- Avoid hang on boot.I have read in some older posts that completely filling rootfs prevents the system from booting. I hope that there's already a workaround for this in PR1.2 or some other third party util (bootmenu?). The workaround would be simple anyways (a 10MB file that gets removed on boot and reacreated after suceess), but I am not sure if my current configuration is protected against that and even after the effort being done in optifiying some apps may fail and cause some havoc.

Ie: Fring for N900 creates a HUGE log on /. I only used it a couple of times for testing and the other day I say a 25MB log file in /, which I have already symlinked to another place as a workaround.


Nokia-N900-51-1:/# ls -al /fring.log
lrwxrwxrwx 1 root root 14 Aug 2 03:29 /fring.log -> /max/fring.log
Nokia-N900-51-1:/# ls -al /max/fring.log
-rw-r--r-- 1 root root 22531688 Jul 14 15:40 /max/fring.log
Nokia-N900-51-1:/#


Luckily, being a log file, its real occupation in ubifs is much less or otherwise it would have nearly filled it.

Also, I have many binaries and libraries directly moved from easydebian chroot to main os which are not EXPECTED to need to write anything to / but one never can be SURE. (Yeah, It may be wrong to do that, but for me it gets the job done and no problems YET).

2- For developing, it is the SIMPLEST way to have different customizations on boot and making sure you have a perfectly fresh system is just a matter of mounting an empty overlay on top of original rootfs.

In my tests, I have confirmed that using a loop interface even with an image file stored in /home/user/MyDocs adds almost NO noticeable overhead. So I don't expect mini_fo do it either. Also I have seen no noticeable differences between ext3 and FAT fs. The bottleneck is always emmc speed... unless you start playing with a truecrypt image file (10 times slower in writing, 2 or 3 in reading) which rules it out for anything except selective storage.

Well, if there are no other reasons against this method, I think it is time for testing... I see unionfs kernel module is included with titan kernel, but I don't find mini_fo... is there a package somewhere for the missing modules?

I would appreciate any other suggestion or something I should take into account before trying.

Thanks!

[appnss]

In the meantime I am trying to decide which would be the best fs for the overlay.

Jffs2 seems a good choice in this case, and it also seems to offer some control for compression settings (perhaps no compression is best as N900 doesn't have much space limitations, unless it noticeable reduces write times) and also for setting when garbage collection may be done. I like that it allows for atomic operations and that it is flash friendly.

Garbage collection might be posponed until on charger AND idle, providing the overlay is adequately oversized.

Have anyone done any benchmarks of jffs2 or any other flash friendly fs in N900?

My only intention is to have an overlay for /, with no changes to the rest of the system. The idea is just to have an oversized rootfs just in case **** happens, and also for having multiple overlays for different purposes. All the rest (/opt etc) being unchanged.

For /opt lack of space I am just using a combination of additional loop mounted images and symlinking which works ok for me... I don't see any issue there.

[pelago]

appnss, I wonder if your ideas are worth chatting with the MeeGo team, e.g. stskeeps or similar mentioned at http://wiki.meego.com/ARM/N900

[javispedro]

You can only use jjfs2 over the mtd layer, and note that UBI is "kinda" its direct successor.

[appnss]

@pelago

Probably the only idea that is worth trying is the mini_fo overlay. Which I would have already tried if I had found mini_fo.ko for titan's kernel.

I am downloading the SDK and will try to make a compatible module... Should be easy, but it's been a very long time since last time I had to recompile a kernel myself. I hope not to break anything :P

@javispedro

Then I guess ext3 would be the most reasonable choice, at least for now.

[appnss]

An update about the mini_fo solution I proposed:

After enabling native compilation I discovered that mini_fo isn't updated anymore. The only update/patches available are those from the OpenWRT project which hevily relies on it.

mini_fo as it was doesn't compile against kernel 2.6.28 so the only way (that I know of) would be to backport mini_fo from OpenWRT and I haven't had the time to do so.

Anyways, I cracked my lcd screen in the meantime I had time to think about it. Now I think the better approach is to just leave maemo as it is, perhaps increasing the size of /opt partition or adding additional image files to increase it and move/symlink stuff over there.

And have another chroot'ed environment, a maemo one (in contrast to easydebian, etc), in which to have a second system, including a native compiler with which to compile every needed programs without having to care about optifiying. It works perfectly and the underlying original maemo keeps unharmed.

For me, it's the way to go, and it covers any need I can think of.