rootfs full? Here's a temporary solution until more packages are optified
 

I was a little irritated this afternoon with a full rootfs, so I've knocked up a quick solution: http://pastebin.com/f1355fe1f

This is a simple script that calculates which packages are taking up the largest number of blocks in /usr on the filesystem and can relocate portions of them to /opt. It does this in a safe manner - every operation is atomic and it will verify the files are identical before the final switch. It can also undo the relocation, if needed.

The first run will take ages - leave the device on power for fifteen minutes or so while it churns through all your packages and works out which ones use up most space in /usr. Subsequent runs will use a cache, unless you start it with the --clean option.

A few caveats:
* The script does *NOT* make sure /opt is in a sensible location. Make sure it actually is referring to a different partition!
* There's no blacklisting of packages. It *WILL* be possible to damage your system with this script if you relocate something that is needed before the partition holding /opt is mounted. Only relocate packages you believe to be safe! Anything you know you've installed yourself is a good bet, or anything which is entirely GUI based. The n900's nokia-maps app is a prime candidate. 'busybox' is *NOT*, even though it will probably be detected as the largest package.
* The script has been tested and works on my Ubuntu box and my n900. There is no reason it won't work for the n8x0 on Maemo 4 or Mer.
* Technically, there's a race condition when switching files, so I suggest everyone disables multi-user logins on their tablets and makes sure there's no malicious software installed. Or in other words: It's not really a problem in this case.
* The script will only relocate normal files in /usr. Nothing from elsewhere, no directories, links, special files, etc.

Use:

Get root on your tablet.
Make sure bash is installed.
Download the script.

(anything other than 'arc' or 'undo' will skip that package)

Re: rootfs full? Here's a temporary solution until more packages are optified
 

A few notes:

Maemo 4 does not have an ext3 partition on the built-in card so it's a no-go unless you repartition (at which point you are better off cloning the whole root).

Also, I have not analyzed your script, but be VERY careful about moving /usr/lib stuff around - that can cause serious breakage.

Re: rootfs full? Here's a temporary solution until more packages are optified
 

A loopback filesystem would do the job there, of course. They're not at all difficult to work with.

Yeah, that's why I stuck a big caveat in. It's also why the script only moves individual files: that way, you isolate the relocations to just the packages you want.

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Thanks! So... which were the applications contributing to the load of your rootfs? Please share them here.

See also http://wiki.maemo.org/Opt_Problem

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Well, according to the script itself, here's what I've moved:

libqt4-core 9,988k
hildon-theme-alpha 7,668k
nokia-maps-ui 6,876k
adobe-flashplayer 6,700k
nokia-maps-core 5,316k
nokia-maps-maplets 2,608k

I'll probably move more stuff as and when I need it - for instance, microb can be moved, and that will free up another 20 meg or so counting all its packages. Really, it should be safe to relocate anything wholly gui-based - so gtk, libpurple and so on are candidates.

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Don't forget the eMMC is slower and also the device you swap to. From what I understand that's why originally Nokians did not optify qt and libs in general.

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Yeah. But I'm happy with that as a temporary solution. It's improtant to remember that the right answer is to get packages optified properly.

Re: rootfs full? Here's a temporary solution until more packages are optified
 

ccooke, this is an (interesting) list of pre-installed packages but what I was asking was the apps you have downloaded getting your rootfs full.

You won't believe it but I'm asking this question to anybody here that gets in this situation and still nobody has given names of apps to follow and fix. It would be appreciated. :)

Re: rootfs full? Here's a temporary solution until more packages are optified
 

@qgil: Working on the list. Will update as it goes along. Repos enabled: extras-testing and extras-devel.
Package: libwebkit-1.0-1
Files: 3
Size in /usr: 23104 blocks
Type 'arc' to archive: arc
(lots of microb/busybox)
Package: python2.5
Files: 3
Size in /usr: 10456 blocks
Type 'arc' to archive: arc

Package: libas-storage-0
Files: 1
Size in /usr: 5292 blocks
Type 'arc' to archive:

Package: libpurple0
Files: 2
Size in /usr: 4792 blocks
Type 'arc' to archive:

Package: canola2
Files: 2
Size in /usr: 1924 blocks
Type 'arc' to archive: arc

Package: fbreader
Files: 2
Size in /usr: 1640 blocks
Type 'arc' to archive: arc

Package: openssh-client
Files: 2
Size in /usr: 1104 blocks
Type 'arc' to archive:

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Going from memory, fennec/xulrunner were particularly bad, but they're long gone now. From the stuff that's still installed, the worst are (omitting smaller files for brevity) mplayer:

pc-connectivity-manager:

and bluemaemo:
BTW, this command:
can be used to show the top 50 space hogs.

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Won't this just show the largest packages. But won't show if any of them are optified. For instance if you install libicu42 it is over 20 megs of libraries; but since the library is optified the amount of space in usr/lib is just the symlinks.

Nathan.

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Yes, it's just a quick way of looking for good candidates for further checking with something like this.

While on the subject, can we have exceptions for the openssh packages (and/or the dropbear equivalents)? They could come in handy for recovering from a broken /home partition so I would prefer to have them in the root filesystem (if that's also broken then it's probably time to reflash anyway).

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Thanks a lot! Please keep these names of fat apps coming. Knowing about specific apps makes it easy to rationalize potential sensible exceptions. For instance, it is clear the difference between a user after openssh or a user after Fennec.

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Once a few qt based apps have started up, won't the
qt libs be cached in RAM and performance just as fast
as if the libs were stored in nand?

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Should this thread be moved from General to N900 then?

Re: rootfs full? Here's a temporary solution until more packages are optified
 

I must have missed the memo.

a) From a developer perspective how are packages "optified"?

For example do you mean - instead of
$ configure --prefix=/usr --host arm
use
$ configure --prefix=/opt --host arm
???

b) From a [power] user perspective is there a more permanent way (a dpkg flag or such) to get packages to install into an alternative location rather than using this script to copy the contents of /usr to /opt (which I assume to be the desired behaviour)

c) What is the current partition structure of the N900 (and highlighting of the problem area would be appreciated)

Re: rootfs full? Here's a temporary solution until more packages are optified
 

The easist method is to use maemo-optify as the step before the dh_mdgenerate in the rules.


Wiki Link: http://wiki.maemo.org/Opt_Problem

Nathan

Re: rootfs full? Here's a temporary solution until more packages are optified
 

And what exactly does this mean? Since this is the content of the Fremantle libwebkit package:

http://www.bundyo.com/things/libwebkit_contents.txt

It is optified from the day 1 it was posted there.

Re: rootfs full? Here's a temporary solution until more packages are optified
 

I can confirm that:

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Hmm. I'd guess from that that the script is following the symlinks into /opt. That's easily fixed - I'll post an update this evening, when I'm back from work.

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Did this thread fizzle out into nothing? Did ccooke ever come back from work? :-)

(My understanding of /opt according to Linux Filesystem Hierarchy: nothing to discuss, all apps should have gone into /opt from the beginning, we should never even have started to put things in rootfs... bad habit from older Maemo?)

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Is it possible to use parts of the script at installation of packages? so that debian-packages could be installed on the 2gb-partition and symlinks automatically created on the 256mb-partition during installation? i think symlinks are the only solution for the problem with the small root-partition, without changing logical structure of the system (remove files for booting from /usr and move/symlink /usr to 2gb-partition) or bootloader configuration (to boot from 2gb-partition).

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Have a look at: Repartition the internal drive

Re: rootfs full? Here's a temporary solution until more packages are optified
 

subversion in -devel isn't optified...

Re: rootfs full? Here's a temporary solution until more packages are optified
 

While in the process of moving some packages with the script, I'm wondering what is going to happen when one of them is to be updated by the application manager (or apt-get).

It is going to install its new files in their pristine position, on the root filesystem, overwriting the links, and leaving parts of the old packages on /opt? Shall it get me into trouble? What am I going to do with those remnants on /opt?

Or I better undo optification of packages before I update the packages?

Thanks for your responses.

Re: rootfs full? Here's a temporary solution until more packages are optified
 

on my N900

:command not found
:command not found
'kgscan : line 9:syntax error near unexpected token `{
'kgscan : line 9: `{

Re: rootfs full? Here's a temporary solution until more packages are optified
 

I've been thinking about the same thing. If the updated package still isn't optified it will overwrite the links I believe. If it is optified it won't be a problem AFAIK. That would create links to the new files (in /opt) overwriting the old links AND files.

Re: rootfs full? Here's a temporary solution until more packages are optified
 

First of all shouldn't the name of the script have been pkgscan...? :)

When i downloaded the script by using the download function on pastebin the file had WinBlows newlines (^M) in it, so first I tried a search and replace in vi but that didn't work (%s/^M//g), so I ended up copy/pasting from the site instead.

I've attached the script. unpack with tar -xf pkgscan.sh.tar... The script is called pkgscan.sh because it's a shell script.

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Hi, I've seen too that an update or uninstall only updates the symlinks and leaves all files in /opt/relocated/.

In order to keep things clean I've done an "undo", then apt-get --purge remove of all the applications originally relocated, then apt-get autoremove (since there were some remanents) and re-installed those apps.

IMHO, people not really familiar with gnu/linux should stay away from this script for many reasons already mentioned plus that if one relocates an app already optified the "undo" will un-optify it. That is, the undo function is not a real undo.

Hope it helps.

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Yes that's the problem. I changed from CR/LF newlines to LF only newlines in Scite, from easy debian.

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Thanks , I used pkgscan.sh.tar

It works good.

Re: rootfs full? Here's a temporary solution until more packages are optified
 

If there is an upgrade available for any of the packages i "archived", do i need to "de-optify" (undo) before applying it?

Re: rootfs full? Here's a temporary solution until more packages are optified
 

See post #27 :)

Re: rootfs full? Here's a temporary solution until more packages are optified
 

BTW after next reboot mc and dosbox icons (i "optified" both) were replaced with blue squares. It happens to desktop shortcuts only, icons the launcher are intact.

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Is it save to archive huge libqt4-gui?

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Personally I would *ONLY* archive apps I installed myself (I've archived subversion), but that's just me... ;)

Remember that this is a workaround. I'm sure Nokia/Maemo Community comes up with a solution to the "opt problem" (other than to *manually* optify packages before submitting them to the repositories - I'm thinking automatic opt checks when submitting a package to -devel/-testing) sooner or later... :D

Another thing; Before a firmware upgrade I would also undo whatever I have archived (just to be on the safe side)... You never know what that firmware might bring us... :)

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Hi, I've missed this earlier post.

From what I see the above post describes how it should work but actually it works a little diferently.

If the updated application isn't optified it will overwrite the links and you'll end up with unessacery files leftover in /opt/relocated/.
The same will also happen if the updated application is optified too. It will not overwrite the old files simply because they are in a different folder from what the correct optification uses.

Basically, all the heavy-lifting in the script has been done but it is not complete so you should be carefull and willing to reflash if you decide to use it.

Hope it helps.

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Makes sense... ;) So maybe the script should be edited to "archive" using the correct paths?

Or just undo before any update of a package You KNOW has been archived using the script...

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Maybe not, because it is simpler to find later which stuff have been moved by the script and undo them. The script looks in the /opt/relocated folder to provide the undo functionality.

Yes, at least that's what I did. Nevertheless, as it was mentioned in an earlier post, the undo of the script is not a real undo. Thus, IMHO the script should be used only for applications/libraries that can be re-installed easily from extras* repos.

Re: rootfs full? Here's a temporary solution until more packages are optified
 

Yes, that's one of the side-effects of the script.
One more related is that the relocated applications don't appear any more in the list to add them as a sortcut in the desktop.

Haven't understood yet the reason but it might be related on the bootup sequence. That is, the /opt might be mounted after this info is being cached.