[Under consideration] More efficient and flexible use of internal flash
 

https://maemo.org/community/brainsto...internal_flash

about 27GB of the N900 internal flash memory are reserved for a VFAT (FAT32)
partition which is mounted on /home/user/MyDocs and exported as USB mass storage.
The VFAT filesystem is an ancient, slow, inefficient (cluster size, bad for many small files) non-POSIX compatible (no symlinks, permissions etc) filesystem and only used for maximum compatibility when exporting it via USB mass storage.

It would be nice if it would be replaced by a ext3 or any another modern POSIX filesystem and merged with the /home partition so that all advantages of those file systems can be used. This would remove the 2GB limit for the home partition and make a more standard Linux home directory layout possible (eliminating MyDocs, new directories Pictures, Documents etc. in the home directory).

For USB mass storage mode file system images with any file system (e.g. ext3, NTFS, HFS+ depending on the users desktop OS) and arbitrary size could be stored on the partition and exported using loop devices.

Re: More efficient and flexible use of internal flash
 

This is an awesome idea. Just some questions:

1) Do you think we could have some sort of script to backup and restore the whole setup before and after applying a firmware update? (I know that until Nokia actually changes N900 default filesystem layout we will have to use some sort of workaround)

2) What would be the best file system for a flash-based disk? I'm a little bit worried about XYZ wearing the disk too much because of too much updates to some sectors. Anyway, FAT32 updates the FAT too much, anyway.

3) I also agree with your last points about having a more standard Linux directory structure. It would make it a lot easier to use a tool like rsycn or Unison to synchronized your desktop files with the N900.

4) Do you think we could also throw into this mix some sort of selective Cryptography (for example: keep a subfolder encripted)?

5) Finally. I'm assuming you have already managed to get your idea working on your device. What are your thoughts about it? Anything unusual happened?

Thanks,
Luis

Re: More efficient and flexible use of internal flash
 

See http://talk.maemo.org/showthread.php?t=35122

There's a few different repartitioning solutions there.

Re: More efficient and flexible use of internal flash
 

Hi Luis,

IIRC a firmware reflash did only overwrite the separate root flash memory but did not change the MMC flash at all. So there is no need to perform a backup only for the reflash (you should always do backups, nonetheless). If you want to backup the paritition table you could simply use sfdisk or save the first sector of the flash.
apparently UBIFS (the fs for the root fs) is not possible.
Nokia decided that ext3 with writeback cache is a good idea for /home and I guess it was a informed choice?
that's actually a good point! you could use your N900 as a backup device for your desktop home and vice versa.
everything which is possible on Linux should be possible on the N900 as well.
for instance, you could have a encrypted loop device using a file on the ext3 partition
and bind the directories into your home.
no problems so far. My next plan is to run some benchmarks on the root, the ext3, the vfat partition and vfat image to check how much overhead the loop device causes.

I wish Nokia would get rid of the MyDocs partition so that applications don't have to deal with a non-POSIX fs but can use a large POSIX fs.

Re: More efficient and flexible use of internal flash
 

I voted against. So what use is to have a ext3 MyDocs partition? Most people are already storing their documents in FAT32 USB drives already. I don't need symlinks nor fancy features to store my documents, and in fact it is mounted noexec for a reason.
Also, $HOME is already ext3.

Don't think that making it ext3 will instantly remove the 2GiB limit in /opt and $HOME. Do you want them unmounted when connected through USB?

And I can't see what are the benefits of having a 25GiB FAT32 loop file in a 27GiB ext3 partition instead of a 25GiB FAT32 partition and a 2GiB ext3 partition. Before you say "dynamically resizable!" consider that for me the only difference is that with the loop file method I won't have to use e2resizefs (but I'll have to use a tool to resize the FAT32 filesystem either way).

This is not such a simple issue.

Of course, if you don't care about windows or exporting through USB.... but that's not the common use case here. Still, I agree it has to be possible to do this on your own device.

Re: More efficient and flexible use of internal flash
 

Thanks for elaborating your concerns.

For people who
* use a single-user Windows with a FAT fs (no NTFS, no compression or encryption, and only few small files)
* and will never install more than 2GB apps
* and who will never need more than 2GB in their $HOME (e.g. only smal email folder)
my solution is neither helpful nor harmful.

But for tech-savy people (which I believe a high percentage of the Maemo community is) who would like to
* use different modern file systems (ext3, NTFS, reiserfs, zfs, HFS+ etc)
with their extra features (POSIX, permissions, compression, efficient storage for small files, encryption - important for a mobile device, versioning or backup etc)
to store or sync files with their desktop
* need are large $HOME
* use the flash efficiently (see later)
* export different filesystem images depending on the context (e.g. FAT image for compatibility, ext3 at home)
my solution offers them much more flexibility.

Please have a look at my loop-device solution in the brainstorm.
It already works flawlessly on my N900 and it would be great it was default
so that users would not have to go through the hassle of repartitioning their MMC.

Basically you would have a single ext3 partition on the MMC flash mounted as /home.
On this partition you would not only store /opt and $HOME but also virtual disk images
with arbitrary size and filesystem.
The images are sparse files, i.e only allocated sectors in the disk image are actually
allocated on the ext3 fs and free space is not wasted.
This way you could have at virtual 20GB FAT image (with 10GB used) and your
ext3 fs would still have 22GB free (32 minus 10GB).
It is possible to create several images and to select which one is unmounted and exported for USB mass storage mode.
it's not easy to use e2resizefs on /home as you'd have to umount it first.
AFAIK online resizing works only for growth.
Its much easier to resize an optional disk image than a crucial partition which
holds most of your apps and your home.
see above. It already works perfectly with Windows and USB mass storage mode.

Re: More efficient and flexible use of internal flash
 

Please don't vote against it. If you don't care about all the missing features just let it go. It won't affect you. If you would like to have a very large FAT32 partition you could just set it to the available free space, giving you more potential space than with the current partition scheme.

What you seem to be missing are many things that would be interesting if all that default space was not wasted on one of the worst file systems of the last +-20 years. For example, if we could have a default installation with just a decent and appropriate fs:

• We could use rsynch or Unison (or any other tool) to synchronize files with a desktop installation without the risk of overwriting files because FAT is not case sensitive.
• We could preserv all fs attributes
• We could use "ln -s" to symlink things
• We could avoid wasting some much space as FAT does
• We could have modern filesystem niceties, such as journaling (if we wanted), good performance, better indexing and so on
• We could have much better security, because we could use space in the /home/user partition to place an encrypted folder that doesn't get exported as mass storage
• We could even have, if we think appropriate, different VFAT partitions (loop devices ponting to files, actually) that are exposed as mass devices
• We could indeed not have to worry about only having 2GB for the OPT partition
• We could even try (it seems to be possible) to use AUFS to stack the root partition and the 32GB partition together, allowing the system to only write to the 32GB partition, either getting rid of the 256MB limit problem or minimizing it. When you would install some files to the /usr folder it would get transparently written to the other partition, where the limit is much higher.
• I think this list could go on.

Anyway, what is been proposed here is NOT an one or zero solution. You would not lose MyDocs (I, to be sure, would love to get rid of that). You can still use it, it can still grow to 22GB or even more than that. It would still be exported. It could still be mounted noexec. Whatever. The idea is that for those that care, we could have a better partitioning scheme.

Re: More efficient and flexible use of internal flash
 

Here is a quick one: if I had titan's partition scheme I would try to use AUFS, bind mount or other techiniques to transparently make better use of disk space. That done, i would probably have the FULL debian repository at my disposal, without some of the IO problems that the easydebian people are trying to solve. All this in a nicely, secured, scructured manner.

Don't you think this would be nice?

Re: More efficient and flexible use of internal flash
 

Seriously. I don't see how Nokia shipping a larger _ext3_ partition by default will help users who want to use zfs.

No, it doesn't work flawlessly. It's a bit slower. And it's sparse, but not smart. If you full format it/fill it ONCE you'll have a 20GiB FAT image _FOREVER_. You'd have to implement vmware-like shrinking capability, etc. Good luck selling that to the average user.

I though your idea was to create one big ext3 partition only.

Re: More efficient and flexible use of internal flash
 

You can very easily do it yourself on the device. I think the ability to still do that and choose whatever filesystem you want is MUCH MORE IMPORTANT than Nokia changing something that works pretty well _right now_.

I can do most of these now, in the 2GiB ext3 partition and the 256 MiB OneNAND. And if the sizes worry you, you can repartition, which works _better_ than the loop file. And you don't get to confuse users when they ask where their 32 GiB are.

Again, what's the difference between a 25GiB MyDocs file in a 27GiB ext3 partition and a 25GiB MyDocs partition with a 2 GiB ext3 partition?
So far, I only see "No need to e2resizefs", which is something I could consider a point if it weren't because a user like you would usually just shrink the MyDocs partition to 2GIB and then enlarge the ext3 partition ONCE and be done with it.

You aren't free to do that now?

Re: More efficient and flexible use of internal flash
 

@javispedro

I agree with you in a very general way: you can make the changes you want yourself. However, as titan has pointed out, there are some standards in generally agreed Linux file system structure that could be improved.

I'm assuming you are only talking about leaving MyDocs as a fixed FAT32 partition that gets exported. I agree that letting the user choose whatever he or she wants is important.

However, if we consider the whole partition scheme, N900 seems to be broken by design: sell a 32GB device which very rapidly fills up a 256MB partition (which is already half full) and turns into a brick.

There must be a reason for having this partition scheme in place, but I'm eager to see something better that could be done to fix it. One idea is to use AUFS or other stackable file system mechanism.

If you follow the optifying threads you will see that even having a tool to help change file locations to /opt is not a complete solution. There are dependencies and lots of other things that make porting a standard Linux package to the N900 a little more difficult.

If we could just grab a standard debian package and install it (or at most recompile it) to N900, without the extra care needed to put files in different places, in no time we would have a wealth of available applications for the N900. With additional time and effort, those applications could be adjusted to the Hildon environment, but at least it would be faster to have them. I know there is easydebian but I'm talking about a speed up in porting applications to the N900 with minimum effort.

Now a question: do you or anybody knows how Nokia is planning to deploy fixes to the base Maemo 5 system? Is it going to be updated deb packages or will we have to reflash the devices?

Re: More efficient and flexible use of internal flash
 

Please note that this is NOT about the 256 MiB OneNAND.

I could start saying that end users won't fill their 256 MiB partition, that optifying packages is very easy, and there's more reasons to why you can't drop Debian .deb packages in the N900, but then this thread would convert into yet another /opt discussion and there's been too many of them already. So please don't, and let the discussion of the ext3 partition keep on.

Re: More efficient and flexible use of internal flash
 

Sorry. I'm not a Nokia developer. When I saw this thread named "More efficient and flexible use of internal flash" i could only assume that this was the place to discuss improvements in that regard.

But since you know of those other threads, you could lead a movement to gather them all and talk to a Nokia representative in order to try to improve things for the next generation device ;-)

Re: More efficient and flexible use of internal flash
 

I will help all users who don't want to use FAT but not harm FAT users.
??? how do you know? have you tried it? what doesn't work? show me the benchmarks.
The sectors are mapped 1:1 and writing is cached on ext3. I don't expect noticable
overhead.
SO FAR we have a 27GiB FAT "forever". Finding unallocated FAT sectors is so simple
that it could be done before every mount.
But the average user (you probably mean a less tech-savy user than the average N900 user) would only have a default, non-sparse 27GiB FAT image, anyway.
correct. how is that related to my previous comment?

Re: More efficient and flexible use of internal flash
 

Well, they're mapped 1:1, but not linearly or else the file would not be allowed to fragment. So yes, there will be overhead, but I concede the point it may not be noticeable for most use cases since the inode and pointer blocks will be cached.

FINALLY some benefit to the sparse file approach. Feel free to start coding that.

Until such a software is made, I see no benefit to having a sparse FAT32 file over the current dual partition approach.

Re: More efficient and flexible use of internal flash
 

I don't understand your motivation for posting ultra-conservative posts in this thread.
This a brainstorm thread where some people discuss the pros and cons of possible
solutions to a problem. Not everyone might have that problem but WE do.
It's not up to us whether Nokia will implement one of our solutions in future devices
or even the N900, but here we can try to find with a good solution and suggest it to Nokia
or try to implement it in the community.

Yes, you can do quite a lot with N900 and there many ways to shot yourself into the foot
and brick your device. One of them is repartitioning, which is NOT trivial.
In fact, you can also do a lot things with crippled Android devices or Iphones if you root them. BUT some people bought the N900 hoping to get a real Linux mobile computer out of the box. We don't want to run dangerous hacks just to get something useable.

I am pretty disappointed by the default partitioning scheme and the fact that the standard
applications hardcode the MyDocs FAT partition ($HOME is not visible to them).
I cannot trick them to use files on my large ext3 partition as not even bind or symlinks seem to work in MyDocs.
So the MyDocs partition does NOT work pretty well right now!

Can you give us ANY evidence why a loop file is bad?
Do you think the 256MB root + 2GB home is not confusing?
For average joe there would be a 27GiB FAT image and 2GB free space on ext3.
so no change for average joe but many benefits for advanced users.

Re: More efficient and flexible use of internal flash
 

Please read my ultra-conservative proposed solution then :D

This has nothing to do with the partition layout. $HOME has never been visible to the standard applications, even when MyDocs was in the same partition as $HOME.
If you're really interested in this, you can "move" where applications expect MyDocs. At least, you could in Diablo with the $MYDOCSDIR environment variable.

So why don't you create the larger ext3 partition into the loop file instead? Unless your plan is to grow it maniacally you "shouldn't" even notice it's running under such a bad performing filesystem.

Not for the average Joe. Or at least -- that was the intention.

I consider myself an advanced user and I see no benefit other than the "still not done" dynamic resizing.

Re: More efficient and flexible use of internal flash
 

I already agreed that for the default configuration a fully allocated FAT image
would be the simplest and a comparable solution to the current approach.
Advanced users, however, may play with sparse files or whatever they like.
For you, there would be neither an immediate benefit (maybe later?) but also no disadvantages.
If you can find some drawbacks of the approach, I would be happy to discuss
them. but so far you only defend status quo because YOU see no benefits.

because the FAT partition is exported for USB mass storage

yes, that's a separate (reported) bug, but if MyDocs was a ext3 directory I could simply symlink its subdirs to my $HOME subdirs...

IIRC it failed when I tried last time but, thanks, I'll give it another try.

Re: More efficient and flexible use of internal flash
 

Not only I don't see benefits (not even for power users, since those will repartition/remove the FAT32 partition entirely either way) but I see added complexity -- since any modification to the default system scripts would need to handle units with the older layout.
In fact, ANY modification to the currently relatively understood simple system will probably make it harder for people to create their custom layouts, unless it's specifically directed to simplifying it - like for example removing the autogenerated fstab nonsense.

hildonfm is opensource; you can edit it and name the folders per your liking.

Re: More efficient and flexible use of internal flash
 

(This deserves its own solution entry : ) I've come to like the idea of a user-friendly extras GUI app where you can resize the FAT32 ("user documents & music storage") and ext3 ("application storage") partitions. What do you think?

Re: More efficient and flexible use of internal flash
 

repartitioning is NOT a piece of cake!

my current modifications already work with both old and new layout.
I plan to release them in an updated ke-recv package.

I don't consider the current system simple but a mess.
What do you think about future Maemo devices. Should Nokia stick to the current approach (forever)?

image viewer, media player and camera app are not open source :(

Re: More efficient and flexible use of internal flash
 

Despite IMHO not being a solution (unless you can click on "replace FAT32 partition by image") every user-friendly app is a good idea :)
How would you shrink and grow mounted ext3 home partition?

Re: More efficient and flexible use of internal flash
 

While camera has MyDocs path & camera safe folder path hardcoded (this is a bug, feel free to file it, but iirc it was filed already), neither media player nor photo viewer do. The file open dialogs are in hildonfm, and for the rest they use Tracker either way.

Re: More efficient and flexible use of internal flash
 

ok, thanks. I'll try changing MYDOCSPATH again, but IIRC camera then always
fails... (yes, I had filed the bug)

Re: More efficient and flexible use of internal flash
 

Titan,
I really like your idea. It seems it would solve all the problems I am having in a simple way. Could you write a detailed how-to and/or script? The description in the other thread is confusing. For example, how best can we get files from the current, perhaps full, FAT partition onto the new ext3 partition? If a lot of people do this and it works well, perhaps Nokia will adopt it.

Re: More efficient and flexible use of internal flash
 

ext3 supports online resizing of filesystems. So this wouldn't actually be that difficult for someone that is familiar with the partitioning/resizing filesystems.

I've resized several ext3 partitions while they were running. But not for some time now.

This may only work with growing the filesystem, not shrinking, and it may require a kernel patch that the n900 may or may not have.

So I guess really.... it *should* be possible.

Re: More efficient and flexible use of internal flash
 

What about adding solution #1 as a command line option for the flasher tool? maybe something like "--partition=ext3".

Re: More efficient and flexible use of internal flash
 

1UP! Especially if one could eliminate the "image container" for usb exposure completely and simply expose the home partition directly. I run linux everywhere, I don't need a container for compatibility!

Re: More efficient and flexible use of internal flash
 

i don't need it too but camera don't working if MyDocs isn't FAT32.

Re: More efficient and flexible use of internal flash
 

I'm working on an updated version of the ke-recv package which should be able to
* deal with both the standard layout and the single-partition layout
* support export of virtual images via loop device
* perform partition resizing during boot before mounting /home
the another project will be a "repart-flash" package which will contain scripts
to grow the ext3 and shrink the FAT partitions online (or to get rid of FAT)
and to setup virtual fs images.

ext3 only supports online growing, not resizing in general.
so that's only a one-way solution. but see above..

Re: More efficient and flexible use of internal flash
 

if flasher was open-source it would be nice to able to specify the partition layout.
I'm not sure whether it already generates the partition table or whether it contained in the flash image.

Unfortunately you cannot export a mounted partition. use NFS for that.

Re: More efficient and flexible use of internal flash
 

I am pretty sure the eMMC image containts the partition layout -- after all, it has changed a few times (the ext3 partition size has increased).

Re: More efficient and flexible use of internal flash
 

Single partition layout (one big home) may not be ideal when flashing new firmware. Since /home/opt is extension of root file system (most installed 3rd party packages has most bits there) and reflash clears package database I wouldn't be surprised if some initial setup after firmware reflash would do mke2fs on whole home partition to clean leftovers of previous system. Having separate data partition for /home/user/MyDocs might be handy then.

BTW, currently it is not doing it but to me it looks more like bug than intention.

Re: More efficient and flexible use of internal flash
 

I disagree. reflash should never touch the eMMC unless the user explicitly asks for it.
/home continains your configuration and other stuff you can't store on a VFAT fs.
/home/opt is simply overwritten when you restore your backup.

Re: More efficient and flexible use of internal flash
 

I agree: reflash should never touch the eMMC unless the user explicitly asks for it.

Re: More efficient and flexible use of internal flash
 

Hence the word command line option. If not specified, it wouldn't be changed.

I guess with the flasher being closed source, my idea would have to be submitted as a feature request to Nokia.

A less elegant solution would be to wrap the script inside another one, applying any necessary changes after the flash has been completed. Not trivial, but possible.

Re: More efficient and flexible use of internal flash
 

This is true for data (i.e. the FAT partition) but false for /opt (controlled by debian packaging) and IMO also false for user settings. More details explained here

Re: More efficient and flexible use of internal flash
 

I added:
export MYDOCSDIR="/home/user"
to /etc/profile. Reboot.
It doesn't work. FileManager can see only /home/user/MyDocs and videoplayer too.

Re: More efficient and flexible use of internal flash
 

I had tried the same with changing the definition in /etc/osso-af-init/af-defines.sh
to no avail several weeks ago. But IIRC I only checked the camera app.

Re: More efficient and flexible use of internal flash
 

This is an excellent discussion. In my view, the main issue is not whether users need this repartitioning but whether applications which people develop need it. There may only be small percentage of users who want a large, posix-compatible, never-unmounted, filesystem but there may be some applications those user's want to use which have those requirements (primarily games, probably).

Can we get some agreement on the requirements of the new architecture, before deciding the best way to achieve them?

I agree with Fanoush that we need the system to be able to have part of the MMC which it owns and which can be over-written during upgrades. This could be mounted as /home (or, maybe, something like /home/system).

For those of us who need to be able to boot different software releases, for testing, we would want to have different /home's available, which we arrange to be mounted correctly for the image we are booting (a task for the bootmenu).

There should be a user area (/home/user) which should be common, preserved and not touched by upgrades/reflashes. This area should always be mounted (not unmounted during USB access) and should be a Posix filesystem.

There should also be an area which is exported as a VFAT filesystem over USB for easy transfer of files to/from the device from/to Windows systems. This could be /home/user/MyDocs (although it might be better to have it more explicitly labelled as something like /home/user/My Exported Folder).

The hardest part of the requirement is to work out how the space should be divided up between these three areas. Ideally, of course, they should by dynamic. If anyone can come up with a solution which does that I would be very happy.

Graham