Re: [Waiting] Remove 256MB limitation of the rootfs partition in the N900
 

I think you right, I have the same impression.

nilfs2 may have a problem then device can be almost full. Garbage collection would be not simple for battery powered device. It should be delayed until N900 is on charger.

Re: [Waiting] Remove 256MB limitation of the rootfs partition in the N900
 

@sr: I did see your previous post regarding aufs and unionfs. Great work in coming up with those results, even if the results themselves were kind of depressing! One thing I was wondering about though: If I got what you were doing right, you were using a ram disk backed by a flash drive; essentially your ram-disk was a write-through cache backed by the flash.

For the root partition problem, the situation is slightly different - both backing and front store are persistent. In this case, the fast flash stores the clean Nokia distro and the big flash stores all user additions. The big flash is superimposed over the fast flash so that reads see the big flash first, and writes only hit the big flash. This is more of a read-through rather than write-through scenario, if you get what I mean. I also assume that writes to files that already exist in the fast flash (i.e. modifications) go directly to the fast flash, rather than stick in the big flash like normal writes.

Do you think unionfs is so hopeless that there is no point in trying, or do you think you could have a look at how unionfs works in those conditions? Your basic idea is so pure and elegant that I really hope there is a way of saving it!

Does anyone here know anyone working on unionfs? Are there any Nokians here? Could Nokia consider putting in some work on unionfs?

Re: [Waiting] Remove 256MB limitation of the rootfs partition in the N900
 

The brainstorm has been upgraded to "Under consideration". Can a moderator change the forum thread title to reflect this?

Re: [Waiting] Remove 256MB limitation of the rootfs partition in the N900
 

Well I use unionfs-fuse myself for some alternate use and thus far was rather happy. I am considering using it on the N900 as well.

Of course something like nilfs2 would be nice on the flash as well but I think a more recent kernel would be a good idea first.

Re: [Waiting] Remove 256MB limitation of the rootfs partition in the N900
 

regarding performance: on a HDD nilfs2 performance was not so great
http://www.phoronix.com/scan.php?pag...s_nilfs2&num=1

My benchmarks http://talk.maemo.org/showthread.php...332#post504332
showed that real fs performance on NAND is not that much better than eMMC:
in the best case (compressed zero data) seq. write was 8 vs. 24 MB/s, seq read
19 vs. 30MB/s but with much higher CPU load for UBIFS.
I expect the differences to become much smaller with typical data.

If eMMC is not much slower, the firmware could copy the whole rootfs to a partition on eMMC at the first boot and from then on forget about NAND and boot from eMMC.
We need some benchmarks for booting from eMMC vs. NAND. Perhaps some of the differences are caused by the watchdog?

@slaapliedje: most boot stuff is in /usr and not in /bin or /sbin
see https://arch.nord.thebc.ch/wiki/inde...sing_Bootchart

Re: [Waiting] Remove 256MB limitation of the rootfs partition in the N900
 

Yeah sadly. It doesn't follow any of the normal desktop Linux policies in that regard. I wonder if this can actually be fixed by nokie with an update sometime. It would certainly improve the ability to hack it around without breaking it(had it happen a few times).

Anything that is needed to boot the system to a point of mounting /usr should be directly in / not in /usr.

Maybe have a USR_OK event in upstart that would tell the rest of the scripts that USR is ok. It could be a dummy as a start but could be replaced by one that would mount it.

Re: [Waiting] Remove 256MB limitation of the rootfs partition in the N900
 

On a side note. Comment on the bootchart... that's so painfully slow.

Re: [Waiting] Remove 256MB limitation of the rootfs partition in the N900
 

for those of you who want to play with unionfs: I have uploaded a new kernel to extras-devel non-free
package "kernel-flasher-maemo" (see http://talk.maemo.org/showthread.php?t=43420)

Re: [Waiting] Remove 256MB limitation of the rootfs partition in the N900
 

Now consider that a whole lot of that eMMC throughput will be consumed by swapping in/out and you see why some people report that it is "insanely slow".

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

Also, please consider recent improvements in the optification "art". Since it's already creating the symlink forest at package postinst configure time, patching dpkg to "autooptify" packages at runtime doesn't sound that far fetched.

IMHO, it's not an awful solution.

Re: [Waiting] Remove 256MB limitation of the rootfs partition in the N900
 

What about changing the "flash procedure"? Let everything be in the big partition. The flasher tool could not really <i>flash</i> the whole device. But just replace everything but /home. User files would be preserved...

I think that having this absurdity of moving everything to /opt and still being at the edge of filling rootfs is actually "Not. An. Option". :)

Re: [Waiting] Remove 256MB limitation of the rootfs partition in the N900
 

The flashing is a blind write-only operation. There is no USB mass storage export for the flasher and therefore no mounting or analysis of the current eMMC state.
(correct me, if I'm wrong - the source is not public)

Re: [Waiting] Remove 256MB limitation of the rootfs partition in the N900
 

No there was no actual performance increase. With original firmware you will not notice anything. The problem with watchdog was when I was mixing up file systems. When I disabled it performance was back to the level of original firmware.
If you still want to try disabling it you should use flasher with a command:
flasher-3.5 --set-rd-flags=no-omap-wd --set-rd-flags=no-ext-wd --enable-rd-mode

No i was not using ram disk. I was mounting internal flash (the one with 256Mb) and ext3 partition on top of it (the 2Gb one from internal mmc or from external card, results were the same, swap on the card does not really slow down the system especially if swappiness is set to the lower setting, right now my system boots from mmc and everything works fast). While creating union both of them were mounted with writable rights.
The default behavior for the union is to write all modifications to the file to the most top writable partition that already has the file and to create new files in the most top writable partition that already has the closest folder structure. So generally that would be internal flash as it already has all files. ext3 partition is empty at the first boot.
Aufs has settings to override this behavior. For example it can be set to create all new files on the partition that has most free space. With this settings generally everything is written to the ext3 partition (and the existing files from internal flash are copied up to ext3 partition when they are modified).
I did not find how to change the default priorities in unionfs so most of the files end up in internal flash. I am not saying that it is hopeless. If i could override it so that all write and create operations were on the ext3 partition and only rename operations were in internal flash (as rename is the hardest operation for copy up and anyway it does not require space).

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

a reverse "optification" could be another solution:

mount eMMC on /emmc
bind mount top directories in / to /emmc/rootfs/*, e.g., mount /usr to /emmc/rootfs/usr
chroot to /emmc (later done during boot process)
create symlinks to /rootfs/*, e.g., symlink /rootfs/usr/bin/* to /usr/bin

this is basically unionfs for free and works fine:

now, instead of symlinking usr we should exclude some directories
for which only the contents should be symlinked as packages might install new files in those
e.g. usr/{bin,lib,share} etc.

Re: [Waiting] Remove 256MB limitation of the rootfs partition in the N900
 

Harddrives have very different performance than flash devices. Benchmarks on harddrives aren't comparable with benchmarks on mmc type flash. Benchmarks on the proper SSDs aren't comparable either.

The table is a bit hard to read without fixed-width formatting, but isn't there over 5 times performance difference in the nand's favor? 5573 against the best, 970 fatpart?

Sequential read/write is a meaningless measure for anything except single-tasking video use...

The issues with mmc is kinda the same as with first generation SSDs, but worse. While they have sequential read speeds of hundreds of megabytes/s, and the OS boot "test" shows ok results, in real use they made the system "stutter". anandtech.com has an excellent series of articles on it...

Re: [Waiting] Remove 256MB limitation of the rootfs partition in the N900
 

Exactly.

The flasher could zip and move the data to the PC, flash the drive, and then put the data back. Alternatively a SD card could do the same job.

With regards to the speed of SD cards noted earlier, 6 mbps is the minimum for class 6. Standard usage is around 12-15 mbps

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

About eMMC performance -

Did you test it with random R/W? I suspect the reason behind a separate 256MB root flash vs 32GB eMMC is actually in performance of single block writes. If eMMC actual block is 256KB then it would have a big lag after a single block write and it may have a toll on writing of close blocks (not adjacent but close enough).

In other words - modification of small files in the same directory of ext3.

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

my benchmarks are certainly not representative - if you know some better fs benchmark program let me know. sequential access can at least tell you something about the raw performance of the device.
However, I was surprised that UBIFS/NAND was not dramatically faster than ext3/eMMC
even in best-case scenario for the compressed UBIFS.
Also the CPU usage for UBIFS was 80-100% vs. 10-15% for ext3.
I consider that an important criterium for a mobile device, too.

@shadowjk: of course, HDDs are different, but that it performs so badly even on HDDs is IMHO not a good sign. again, we need more benchmarks...
the seek performance was 1100 vs 3700 on a fresh device but CPU usage was 14% vs 98%
http://talk.maemo.org/showthread.php...332#post504332
not surprising if seeks in a compressed zero-filled file.

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

IMHO, the difference in performance 3 times clearly justifies 256MB flash usage.

But I guess that "uncompressed" root can give a better difference, just because decompression takes toll on performance.

However, it requires more strict strategy of file/block allocation on 256MB because of space problem.

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

I have rewritten my old, rather messy description of Solution #1 completely, and added an idea at the end of how to bootstrap all this after a reflash:

Solution #1: Place root in the large flash and link speed critical files from the fast flash

Posted on 2010-02-09 01:46 UTC by Stefan Möhl.

Basic idea:

o Have the root directory in a large partition in the big flash instead of in the small fast flash.
o Only files and directories that specifically need speed are placed in the fast flash with symlinks into the large root.

This is basically the same trick as optification, just the other way around. Instead of the fast flash as root, we have the big flash as root. Instead of moving large files and linking, we move speed critical files and link. The main advantage is that the default mode of operations is clean: Just install under the normal root hierarchy without any tricks like moving or linking. Specific files and directories from root may be moved and linked to improve performance if desired.

Optification means that we first install and leave the root files in the fast flash, and then move all packages we install in the future to /opt. With this solution we move just a few specific root files and directories (just those that really need it) to the fast flash, and leave all installed packages in normal locations in the big flash. So by default, we move only a specific and limited number of files, instead of moving all future files that will be installed.

Since packages are now normal, installed in their usual locations, we can now start installing all sorts of packages from the Linux community, as long as the binaries are compiled for the Arm processor. All of Debian, for example, has Arm support by default. We will now also have space to install the standard development environments, so if a package has no support for Arm, we can just install the source package and compile directly on our phones for the Arm processor.



Bootstrapping:

Bootstrapping the configuration after reflash: Since a partition in the big flash can not be reflashed without loosing all the data in the user partition (i.e. camera pics, music, etc), reflashes all have to go to the fast flash. Since we want the root to be in the big flash, we will have to move it there from the fast flash somehow. I suggest we do it the same way as we get root access when installing "rootsh": We "install" a large root partition through the app manager.

The "bigroot" package should be like the "rootsh" package: An officially endorsed and supported power user feature. When installing the package, all files are moved from the fast flash to the 2GB partition in the big flash, except the files that should stay in the fast flash for performance reasons. The bootloader is then modified to boot from the new root location, and the device is rebooted.

The "install bigroot" approach has a few advantages. First of all, it permits a rock solid state after a reflash, since it works just like things do now. Users can be warned and choose for themselves if they want to activate bigroot or not, so the pressure of perfect operation is much lower. Many packages won't be installable without installing bigroot, but packages that cater to casual users can still be optified like before. Bigroot is compatible with optification. Also, if your device bricks and you have to reflash, the old bigroot partition isn't cleared until you re-install bigroot. That allows you to rescue some files and configurations once your system is back up and running, and possibly figure out what went wrong.

The installer trick can also be used for applications by adding "speedification" packages. A "speedification" package for an application will move speed-critical files for that application to the fast flash, leaving links behind in the big flash. Similar to optification (though in the reverse direction), but manually activated or deactivated by the user. This way users get detailed control over which applications get to use the fast flash, and can optimize for their own usage patterns.

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

Couldn't you create copies of the directories that you want to shadow in the ext3 partition first, before you apply unionfs? If you copy the whole directory structure (without the files) from the fast flash to the ext3 partition first, wouldn't all files stick in the ext3 partition then?

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

If the emmc is smart, writes within a 256k block would be fast... Problem is filesystems like ext3 sprinkle data far apart, and have layouts optimized to avoid fragmentation.

Basically filesystems designed for harddrives do all the heavy work when writing, they place various datastructures all over the harddrive in order to avoid fragmentation and so that when it comes time to read or find stuff, the data it needs is close together and not fragmented.
The idea is to take the performance hit when writing, since you'll probably read the data more than once anyway.

This on harddrives where sequential writes and reads are fast, and both random writing and random reading is slow.
On emmc, sequential writes and reads are fast. No difference there. The difference is in random writes and reads. Random writes are REALLY slow. Painfully slow. Random reads, however, are pretty fast.
Thus, the complicated and thought out optimizations that make regular filesystems fast on harddrives end up hitting the weakest point of emmc storage, hitting it hard.

The nilfs2 filesystem is almost like a gigantic journal. Data is written out mostly sequentially. Data is rarely replaced, new data is instead written, with a newer version number. This was the goal of nilfs2, to be able to jump back in the journal/log and restore the filesystem to a previous state. As it happens this ends up in nilfs2 being very slow on harddrives, because the data is all over the place when it's time to read or find data. On emmc these random reads aren't all that bad, and nilfs2 generates alot less random writes. Avoiding the biggest weakness of emmc makes it more than 10 times faster in my test.

LogFS is actually designed with flash based storage in mind, so it will be interesting to see how it performs.

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

There's something that (as usual) bothers me with this brainstorm item. What are the benefits of most solutions?

Some of the benefits mentioned:
- Ability to install Debian packages
- Ability to build Debian or Maemo packages

... are IMO more easily done with a Debian chroot/Maemo cloned chroot, which has more benefits:
- ability to have a build environment that is not exactly the same version as your phone environment -- SDK release 0 in chroot for maximum compatibility package building vs PR1.1 in the phone
- ability to trash busybox and install bash and coreutils
- ability to install _unmodified_ ANY Debian release binary packages in Debian chroots
- plus ability to still boot your usual runtime system if the chroot is trashed for whatever reason
... vs IMO a minor tradeoff (more memory usage as you need to load both the chroot and Maemo libc + other libraries).


In my opinion the only one that has a clear benefit is solution #2 (since it means you'd be able to reflash the base firmware without reloading the installed apps!), but then it has quite an important tradeoff: even more Debian upstream compatibility is lost. Yes, changing a package's prefix is not 1-2-3: from all the packages I've ported to Maemo, 50% had some kind of hardcoded path.
I don't mind having to do this since I can report those to upstream, and getting those fixed is always nice, so solution #2 gets my vote (even if it clearly contradicts the brainstorm subject since it actually decreases debian compatibility).


And again I mention the recent improvements in the optifier. The idea is that a developer doesn't even have to know that the autobuilder is optifying.

Also, none of the options save for #3 guarantee that end users won't get their rootfs filled somehow. Then again I'm biased somehow since I've never managed to fill mine.

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

@javispedro

There certainly is a good case to be made for keeping optification the way it is. If you really feel that way, make it your solution in the brainstorm!

I, personally, feel that even though chroot is a possible alternative, it would be much better if we could just install Debian packages by default. Further, if we could install standard Debian packages by default, that would in no way prevent us from still installing a chroot too, if we like to, so all your benefits of having a chroot will still be available. - Oh, and by the way, I want to trash Busybox in favour of coreutils even in the standard Maemo distro, we just need the space to do it first.

I have seen that this has been discussed a few times, but I did try and couldn't find a Brainstorm or Bugzilla about it. I am new to the Maemo community, and not quite sure how Nokia decides what is officially supported by Maemo. However, it seems to me that the Brainstorm and Bugzilla tools are the main ways of consolidating community disposition. There might be other, less formal routes that come straight from discussions in the forum, but if so I am not aware of them yet.

If Brainstorm/Bugzilla are the main routes, the fact that the issue has been discussed so much that people are getting tired of the subject must mean that it is high time to bring it to the vote, right?

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

The problem with this solution is that many critical files can't be symlinked. It is especially with configuration files, usual way to handle it - create a new version and replace the old one.

Symlinking of directories can help but again it can be a problem with some software and nobody knows which one exactly.

Additional (but not critical) issue - read speed of small files (usually - shell scripts in /etc or localization files in /usr/share/locale or Hildon icons or X11 fonts) initially would be degradated until it sits in kernel cache. Because that files are used pretty often in "search" mode (file-by-file) it would visibly increase an overall access time.

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

But, at least on Diablo, some of the bootscripts didn't work without busybox. And even on Fremantle you are limited to really old versions of Debian tooling (think Sarge/Etch), with no debconf nor any other stuff. Seriously, if you want to try to "make Maemo more Debian" you're going to have to fight much greater issues than optification (which is relatively so simple it can even be done automatically!).

This still doesn't mean optification needs to be keep as is, though, since the rootfs free space is getting lower with every firmware update, and thus the "forest of symlinks" "hard limit" is getting tighter every day (which is why I didn't add it as solution, and instead voted for #2).

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

Sure, but that problem is equally there for optification. Since we will have to split across two filesystems in some way - either optification or something else - we will have to tie things together with symlinks or possibly mount-binds however we do it. All I am suggesting is that we do it the other way around, so that we do the linking once, at the start, instead of doing it all the time.

Well, we need to start somewhere, right? And optification is a showstopper for almost all Debian packages (at least if you want to install more than just a few). Let's do the others when that time comes, and focus on optification for now. Even if "all" we get is a big rootfs, I would consider that a substantial improvement. :-)

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

I hope you know about maemo-optify-deb. Save for bugs/TODOs (which TBH are not-as-low-as-I'd-like; see jebba's auto-optification of the entire rebuilt* debian etch archive), it works on most binary packages.

* Cause if you're not rebuilding packages at all you're up for some trouble sooner or later.

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

Look at this - http://www.linux-mag.com/id/7378/2/

"Summary" section:

One area that FS-Cache could prove to be of future use is for caching local file systems. Currently, file systems rely on the kernel for caching data and scheduling for writing/reading to/from the storage. This caching is not directly under your control. But if a local file system can be modified to use FS-Cache then you could use a small but very fast SSD or even a Ramdisk for caching of data.

It gets off symlink problem as is as bootstrap problem.

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

the difference between NAND and eMMC is much smaller than the one between eMMC and hard disks.

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

Sure, but after that, it isn't a Debian package any longer, right? After that, it is an optified Maemo package. And, I think even you will agree that it is better if we can install packages straight as they are, rather than having to run even a very nicely automated process first. At the very least it would mean that package maintainers have one thing less they need to learn and understand before porting a package to Maemo.

Obviously, I don't expect all packages from one distribution to work out-of-the-box on another distribution, but if we can get rid of the optification requirement, there will actually be quite a lot of Debian packages that work as they are. With optification, as we all know, that is not the case.

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

Again, I think that's a bad thing to do, and the chances for something to break are _high_. See the common question of using Ubuntu packages in Debian.

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

Yes, I agree, a cache system would get rid of the symlink problem. I am merely arguing against optification. My suggested solution is meant to be as simple as possible, it is actually just a variation of how optification works now. Instead of optifying everything, we just "negative-optify" root once and then we can install everything else as normal.

Your caching suggestion is much more elegant and advanced, and if it can be made to work with stability, it would be better. I gave it my vote as soon as I saw it :-)

If you haven't done so already, you should look at the posts by @SR too. He hasn't made a formal Brainstorm Solution yet that I am aware of, but his ideas are very similar to yours. The main difference is that your solution uses an automated cache policy, whereas his solution assumes manual decision of what goes in the fast or big flash.

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

Yes, as you yourself quote, "I don't expect all packages from one distribution to work out-of-the-box on another distribution". But even if most packages break, Debian is so much larger than Maemo that we will have a huge increase of working software if only a small proportion of Debian would work. Please forgive me if I misunderstood, but to me it seems that your are arguing that just because some packages break, we might as well break them all.

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

do you have a background in debian-style packaging?
i don't think you do, because if you did, you should know that "optification" really is the smallest part of it. it's the part you consider AFTER you've addressed the big issues.

and to illustrate, i'll give you a simple example:

say you wanted to install bc from the debian stable repository.
although it's a very simple commandline app, the very first dependency of that package will kill you. it requires libc6 > 2.7.1, but the version on the N900 with 51-1 is 2.5.1
if the updater were to install the higher libc6 package, all hell would break loose and you'd have a bricked device. simple as that.

and there are tons of examples like this.
in the easy debian chroot, you can easily provide a proper version (libc6 has v 2.9-4 with the image i have installed currently), but NOT in the native Maemo environment.

it's just not worth the effort, a chroot approach is the cleanest and most reliable way to support multiple distributions. and it works great! (kudos again, qole :) )

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

correct. downgrading to the outdated Maemo5 components is the biggest problem.
I thought this brainstorm is about avoiding a full rootfs and not about apparently simpler
installation of packages from other distributions?
Getting rid of optification could make porting a little bit easier but it could also mislead naive users to think that they could install normal Debian packages.

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

You are right Titan, we are side-tracking. I think we all actually agree: It is never easy to move packages between distributions, and though getting rid of optification makes some difference, that is a far cry from making Maemo into Debian.

Sorry about loosing focus, I'll drop this subject from here on.

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

So, is this it? Do most people in this forum want the small rootfs and optification? Or have we emptied all possible suggestions? Does the brainstorm contain all solutions we can think of?

I am still seeing new threads of this kind:
http://talk.maemo.org/showthread.php...725#post529725

Or this:
http://talk.maemo.org/showthread.php...088#post530088

Both started today

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

Only Nokia can make the small but necessary changes to implement solution #2 in a SSU.
I have a patch for debhelper7 which makes it easier for developers to set an arbitrary prefix
for their package installation and which could lead to a cleaner /optification.

I'm also working on a implementation of solution #5, which could also interesting for people
who want to run different firmware versions or install the SDK, while keeping most files on rootfs.

Regarding NILFS2, we have some disappointing benchmark results and facts at
http://talk.maemo.org/showthread.php?t=43420

Re: [Under consideration] Remove 256MB limitation of the rootfs partition in the N900
 

A first implementation of solution #5 is available as part of the Moebian project
http://talk.maemo.org/showthread.php...122#post559122
it basically moves / to /home or any other partition and keeps the /usr files on NAND.