Re: [Announce] Enhanced BusyBox package
 

I have again lost my history. Now there is no garbage in files but I couldn't trigger any command pressing CTRL+R.

I have checked that file .ash_history is empty. There is also a file .ash_history.1918.new that is empty and a ash_history.3055.new with only 100 very old commands.

Re: [Announce] Enhanced BusyBox package
 

Hmm, that's real unfortunate news to hear.

When you close ash, it'll append all of its new history to .ash_history, completely re-load .ash_history (other instances of ash might have written to it, and re-loading trims it to MAX_HISTORY), write the trimmed history out to .ash_history.$PID.new, and rename .ash_history.$PID.new to .ash_history.

The used code for this is pretty solid (look for save_history() in libbb/lineedit.c), and I don't see any direct cause as to why you lost your history :(. The only clue we have is that rename() somehow failed, since you still got .ash_history.XXXX.new files. It's rather odd though that you experienced this twice - the only other report is from Mentalist Traceur, but he mentioned that the device was unstable.

You could add something like the following to your ~/.profile file. This will backup your old history to /home/user/hist_bak/ when you close ash. It won't solve the issue you're experiencing, but at least you won't lose all of your history. I'm sorry I can't be of more help.

Re: [Announce] Enhanced BusyBox package
 

Since this thread got bumped anyway, let me take this opportunity to call for testers :). Busybox-power 1.19.3power4 has been in extras-testing for a pretty long while now, and we most certainly can't leave out those extras users from shiny new versions, can we?

Also, for those that have noticed, busybox-powers' git repo has BusyBox 1.19.4 in it since a while now. It isn't pushed to the repositories because there is no code change between 1.19.4 and 1.19.3 + hotfixes, of which busybox-power 1.19.3power5 consists. Just a FYI ;).

Re: [Announce] Enhanced BusyBox package
 

Hi,
Got a problem with creating user and group, at least, the postinstall script from the icecast2 deb had!
Can't install this app and busybox power report a prob in user and group creation, any ideas?

Re: [Announce] Enhanced BusyBox package
 

Thanks for reporting.

The line that causes this error in the postinst script is:
However, Maemo isn't shipped with adduser at all! Busybox-power does provide an adduser applet, but it takes different arguments than the postinst tries to use.

The solution is rather simple though. Execute:
and icecast2 will install flawlessly :). The postrm script also uses utilities from the adduser package (namely: deluser and groupdel), so you shouldn't uninstall the package right after installing icecast2. It seems like the package maintainer simply forgot to add a dependency on adduser to icecast2.

Re: [Announce] Enhanced BusyBox package
 

OK I will try that, in fact in case it happens again it would be useful to know when really happened. It might be related to any other issue.

Re: [Announce] Enhanced BusyBox package
 

Hey iDont,

just for the reports:
I also faced this a few times. But afair always after a hard shutdown (real sw_res). I found total senseless entries in history or even lines of my e-mails (which really scared me the first time ;))...

But just loosing the history sometimes is just a inconvenience, no more. And the positice effects of busybox enhanced is SO much more!
Thanks again for that package and support.

Re: [Announce] Enhanced BusyBox package
 

Thank you very much IDont, I haven't thought about it myself...
Thanks also for the hard work about busybox-power, it's a huge enhancement for the N900.
Have a nice day.

Re: [Announce] Enhanced BusyBox package
 

Happened again, this time after using preenv from the command line. Could it be related?

Re: [Announce] Enhanced BusyBox package
 

Thanks, I'll check it out when I get back home. Are you able to reproduce the issue, or did it happen at random? Also, how did you close preenv and its parent shell (by executing "exit" twice, simply closing the window, or ..)?

Update: I'm still not able to (re)produce any kind of issue with the history handling. This issue is driving me nuts! I must have looked over the code a hundred times by now. What completely puzzles me, are the reports about the completely unrelated text in .ash_history; I can't seem to relate that to BusyBox in any way :(.

Re: [Announce] Enhanced BusyBox package
 

Now I'm not able to reproduce it. Seems to be totally random. :(

Anyway, considering the low amount of similar reports I think it must be a combination of different pieces of software.

Now back to preenv,. it's strange that it seems not to work any more. I can run WebOS games from WebOS Game Manager application but not from the command line using preenv. After executing preenv, trying to run any executable fails with a can't locate any of the SDL libraries.

I have even reinstalled preenv with no luck, so I wonder if this is not working due to Busybox Power managing something related with environment variables different than standard Busybox.

I have yet to test if uninstalling Busybox Power results in preenv working again. In the meantime, has anyone tried to run preenv from the command line and execute a WebOS game with Busybox Power?

Re: [Announce] Enhanced BusyBox package
 

I had the known history problem so I installed busybox-power and the problem remained, until I manually deleted the history file. Thought I should let you know.

Re: [Announce] Enhanced BusyBox package
 

It seems to suffer from the same issue as TMPDIR.
Add these two lines to /etc/environment:
And starting preenv programs from the commandline will probably work again.

Re: [Announce] Enhanced BusyBox package
 

Enhanced BusyBox shell 1.19.3power4 is suitable for promotion to Extras. Did you receive the message from Autobuilder?

Re: [Announce] Enhanced BusyBox package
 

Confirmed. Ade's right about the issue causing this.
I'm leaning towards reverting the whole SUID-situation to how things were with busybox-power 1.19.3power1 (that is: no SUID bit and no workaround patch at all). Being able to run ping, su, etc. as a non-privileged user seems to be not worth the effort and the risk of breaking applications like preenv (there may be other applications getting broken that we're not aware of).

Yep, just read the message. However, I'm having second thoughts on promoting it to extras. As much as I hate to basically discard the hard-won thumbs-up (three months of voting), breaking an application like preenv outbalances the features of 1.19.3power4 over 1.19.3power1 (which is currently in extras).


I'll very most likely push out a new busybox-power version this weekend with the whole SUID-situation reversed, and promote that version to extras-testing. I know, it's a real shame of the thumbs-up, but I think this is the best thing to do. My apologies to all the testers, I hope y'all understand.

Re: [Announce] Enhanced BusyBox package
 

I did not look into it to the matter thoroughly, but it only effects starting preenv from the command prompt. As almost all users will use the menu for starting preenv apps, I see the impact as very minor.
Next to this issue I have not seen any other related findings.

I am very satisfied with the current implementation, but would also have no objections if you decide to revert the SUID situation :)

Re: [Announce] Enhanced BusyBox package
 

Aren't there any programs/scripts that are going to break after removing suid bit? why it was introduced for the first tiem - for sole purpose of using few command-line tools, or was it requested to allow easier handling of some scripts for developers?

I.E - aren't there any drawbacks sneaking on us, due to reversing SUID?

/Estel

Re: [Announce] Enhanced BusyBox package
 

Thank you for your input. IMO, even though that there might be just a few applications actually affected, the issue is very real. I didn't mind having to include a workaround or two (like the $TMPDIR thing), but the recent report of preenv revealed the underlying issue to be of a much more far-reaching nature than I had originally figured. That has made me want to reverse the situation to how it was before.

You've got a very good point here. However, the SUID bit was indeed only introduced to be able to run several applets as an unprivileged user -- convenience. There weren't any special requests for this feature. Also, there are no packages explicitly depending on busybox-power (except for yamas, which is executed as root anyway), so we should be fine.

Re: [Announce] Enhanced BusyBox package
 

A major update to busybox-power is on its way to extras-devel! Last Sunday, BusyBox 1.20.0 was released. Not soon thereafter, busybox-power got updated against this new release. This update brings 285 upstream commits to our beloved mini Linux boxes.
I've waited a few days for hotfixes to emerge prior to pushing this busybox-power update. The released hotfixes are already integrated in this update.

Please note that major updates have a larger chance to break stuff than minor updates (although nothing should break, of course). As always, let me know if you encounter any issues with the new release.

By the way, we need one more vote to get busybox-power 1.19.4power1 out of extras-testing and into Maemo's extras. If you've been running this version from extras-testing or -devel and are satisfied about it, please consider giving it a thumbs up. Thanks!

edit: whoa, that was fast! Two thumbs up within a few minutes after posting, thanks! The package has been promoted.

Re: [Announce] Enhanced BusyBox package
 

It happened again :(
Ash history corruption...

Just for info:
I updated my system to kp 50 and busybox 1.20.0 hand other devel stuff).

Yesterday night my N900 did shutdown unexpected (batt is old, keeps me up 1 day, but on low voltage just let my N900 "die"; i. e. it does not terminate all processes nicely. just kills and is off, in below 2 seconds time).

This morning I had again corrupted .ash_history (see att).
This time it looks like some conf file, but I also had already parts of my e-mails or other unrelated text. It looks like the text is randomly taken from memory or file system.
So I can confirm the cause (x-term killed "unnicely", but do not any clue about reason).

I had two x-terms open, one as user and one as root, and it happened to both it seems. Once I checked for root history it contained only the last few commands entered this morning. After closing user x-term, I checked user .ash_history and it contained also only the last few commands.

Maybe this helps you, iDont?
Not demanding anything, as this happens rarely and is just an inconvenience.

Re: [Announce] Enhanced BusyBox package
 

For those running cssu-thumb, I've uploaded a thumb-enabled busybox-power build to the garage page. You need to have kernel-cssu installed to be able to safely use thumb-enabled binaries. Also, please note that this is an experimental build, although it appears to work flawlessly :).

Initial thumb ISA support is also available in busybox-power's source tree, if you want to compile it yourself. You'll need a thumb-enabled target in Scratchbox (instructions). Once the target is set up, compiling is as simple as running `sh build.sh thumb` from your local busybox-power git repository.

Furthermore, a new busybox-power version has been pushed to Maemo's repositories last Sunday. It's mostly a bugfix release, incorporating upstream's hotfixes, but it also has a nice new feature for the vi users amongst us: when vi is ran in a terminal emulator (i.e. X Terminal), your terminal's output is restored after closing vi.

Lastly, I must say that the transition from BusyBox' 1.19.x to 1.20.x series went extremely smooth; there have been zero new issues reported since busybox-power 1.20.0power1 has been released!

-

As of now there is just one issue with busybox-power: when ash is killed -for the lack of a better term- "unnicely", there is a small chance ~/.ash_history could get garbled up. I still can't find any problematic code in the source, any help is greatly appreciated. Look for save_history() in libbb/lineedit.c in upstream's latest stable BusyBox release. Unless someone else finds an issue in the source, I'm assuming the issue is not related to BusyBox itself.
Currently, history saving is handled as follows: all new history is appended to ~/.ash_history. After that, the history file gets reloaded via load_history(), which effectively trims it to n lines (n = MAX_HISTORY). Finally, the newly trimmed history is written out to ~/.ash_history.$PID.new, which will be moved to ~/.ash_history upon finishing writing it out.

Re: [Announce] Enhanced BusyBox package
 

Thanks a lot, iDont! I've updated to new busybox from day first, and indeed, I don't have anything to report.

As for busybox-thumb - maybe it's good idea to sent it to freemangordon, for inclusion in thumb repository?

/Estel

Re: [Announce] Enhanced BusyBox package
 

Great to hear that you haven't got anything to say :p.
But seriously though, this is a useful confirmation on the software's stability/maturity.

Thanks for the pointer! I spoke to freemangordon or #maemo yesterday: there is no problem with including busybox-power in community-thumb. Of course, it won't get pulled in by cssu-thumb or anything, the repository will just provide a place for it to sit in.

Now I just need to get around to cleaning some things up (version string, thumb-related dependencies, ..) before I can send the sources to freemangordon for upload. Well, the weekend is just around the corner :).

Update: the package is in community thumb!

Re: [Announce] Enhanced BusyBox package
 

I had a thought a couple of days ago about this - haven't had time to look at the code, but I figured I'd type it 'out loud' here in case anyone wants to look into it: busybox is written in C, correct? And in C, a file is accessed using a variable of type FILE *, if I recall correctly?

Well, given how C is, it's fairly possible that writing too large of a sequence of values to a given address, or an inappropriately assigned pointer somewhere, could result in rare cases in the overriding of the FILE *, no?

If you're referring to the bug I think you're referring to, in my experience, when it has happened, I ended up with lines of XML as my 'shell history' instead of actual commands. Suggesting that at some point the wrong file got copied over into .ash_history. I'll be honest, I know nothing about how 'FILE' is implemented, on our device or otherwise, but I would think that, if a FILE pointer's value got changed, it would either point to a different file, or to a non-existent file, depending on the exect bytes at the newly pointed to location. C being C would assume you knew what you were doing and would interpret the data at the new location as a typical FILE struct (I think it's a struct, idk). If it was a corner case resulting from some rarely occuring overwrite of a FILE pointer, the compiler would probably not catch it?

(If I am wrong, if someone could let me know how it actually works, it would be greatly appreciated.)

Re: [Announce] Enhanced BusyBox package
 

Exact description.
That will be the consequence, and also explains the absolutely random structure of text content of history (sometimes XML or other config or even email content).
But not yet the reason, we (iDont ;)) have to find ...

But if it would be that (having history written from FILE to a file, I do not understand why .... wait just last 100 lines are written from FILe to history file. OK, fully agree.

Re: [Announce] Enhanced BusyBox package
 

The idea sounds good. But honestly the chances of a random FILE* pointer (e.g. one having been used for cat'ing an XML file..) being overwritten so as to end up having the exact value of the FILE* pointer of .ash_history are close to zero.

Well, a FILE structure essentially contains a file handle (so to speak, the kernel's equivalent of C's FILE) as well as some pointers to various read/write buffers, etc.

I'm not 100% sure now but I think busybox keeps the history in RAM until it exists, where the history file is actually written to. Meaning [ assuming that what I say is true ] that at some point, for some reason, the RAM buffer is being overwritten (partially) with other data (XML, whatever).

I'll see if I can have a look at busybox's source code and figure out what's going on, but my severe lack of time may prevent me from actually achieving anything anytime soon.

Re: [Announce] Enhanced BusyBox package
 

For the record, we don't need to use XML files as the example - yesterday I got the bug again on my testing N900 when playing with the twl4030 watchdog, and the new histories I got were nothing at all for the root shell (the one you get to by using 'sudo gainroot', not the one gotten to by using 'root', if that makes a difference), and something which was probably a mozilla config file for the 'user' user shell.

Well, you don't need a random FILE* to get the value of the .ash_history FILE*, which would indeed be vanishingly unlikely. You'd just need the value of the .ash_history FILE* to get the value of any random file on the system - a much more likely possibility.

I wasn't suggesting, as I think you interpreted, that by using the shell to manipulate file X, I somehow managed to cause that file's pointer to point to the .ash_history file. Rather, I was saying that the .ash_history file struct may have gotten overwritten by random 'garbage' data - and the depending on your luck it might either end up with a handle to a non-existing file, perhaps explaining the 'blank' root user shell history I got in my last instance of the bug, or the file handle at the C FILE struct might end up being one pointing to an actual file somewhere - explaining why people have different experiences with what file contents end up overriding the shell history.

This should indeed be the case - after all, it was in this thread that I originally suggested to iDont that he implement a buffering of history in RAM (so that devices with flash memory don't waste flash chip write cycles with every press of the enter key. And I remember being very happy about him saying that this got implemented upstream.

Either way though, you know C more than me, so you probably got more insight into this than me.

Re: [Announce] Enhanced BusyBox package
 

True. But then the other file would be overwritten with the .ash_history contents (which would actually be a major bug). Unless the rewriting would happen after the history is read but before it is written to.

The garbage data has to come from somewhere. Busybox does not run as root (normally -- or it drops root right at the beginning), so I can't imagine how it would be able to get some random data from memory it doesn't own.

So to me a plausible explanation would be that the garbage comes from data that has been read/processed by busybox at some point (hence my example of cat'ing an XML file).

But this is only a wild guess.

As long as I don't look at the code (and I haven't, yet) we're on the same level :)

(and even if/when I look at the code: I find it hard to read (=like) what others have written)

Re: [Announce] Enhanced BusyBox package
 

Some comments (from bottom up):
My basic knowledge of Linux memory handling would say me that it is almost impossible that some other process overwrite bb owned memory.

Possibly is that bb overwrites its own RAM space for history.

But another thing is that also bb on its own screws the FILE pointer itself, pointing to any space in RAM (reading will be allowed out of non-belonging space?). This would also explain the totally random content of history.

Re: [Announce] Enhanced BusyBox package
 

That's what I've been trying to get at: not one file overwriting another file, but Busybox's own code messing up and writing some other values to the location of it's own FILE*. I.e. suppose busybox is running, it's storing new changes to history somewhere in RAM, etc, etc. Maybe it's using a temp file in the /tmp/ part of the file system, for instance (or not, either way).

Anyway, at some point when it's supposed to close, it'll have a FILE* sitting in the code - perhaps one that has been declared but not initialized. Maybe it's uninitialized so it has whatever other bytes were at that location instead of a 'real' FILE* - after all, garbage at the low level of C doesn't have to be put in from anywhere, the contents of uninitialized memory are typically garbage too, no? Or, if, say, I point an int* at a location in RAM containing 4 chars, if the compiler doesn't catch it (because it's not predictable to the compiler for watever reason), the code will just interpret the 'garbage' data at that location as if it were an int. Sure, the data isn't really garbage in a technical sense, but it is in context of reading that spot in RAM if you're interpreting it as an int.

So what if the busybox-is-not-getting-killed-nicely scenario occurs? Shell tries to save history, as described by iDont - so what if the FILE* that's supposed to correspond to the ~/.ash_history.$PID.new file gets overwritten - pointer itself, causing it to point to any other segment of memory which Busybox can access - then when the time comes to copy/move ~/.ash_history.$PID.new over to ~/.ash_history, bam, your FILE* is no longer pointing at ~/.ash_history.$PID.new, but may very well be pointing at one file in the OS or another.

Meh, all of the above is still speculation until I've looked at the code, but I think it's a likely explanation.

Re: [Announce] Enhanced BusyBox package
 

Thanks Mentalist Traceur, reinob, and peterleinchen for the productive discussion. If any of you (or others) find the time to have a look at BusyBox' sources, that'd be much appreciated :).

BusyBox indeed keeps its history in memory and writes it out upon shell closure. The writing out process goes as I briefly described earlier. BusyBox keeps its history in memory in a line_input_t struct (see include/libbb.h), and only assigns relevant file pointers in load_history() and save_history() (see libbb/lineedit.c).

I'll be out of the country tomorrow for a few weeks with only very limited or no internet connection available at all. Therefore, I probably won't be able to follow TMO during that time. Just to let you know. But by all means, don't hesitate to continue discussion ;).

Re: [Announce] Enhanced BusyBox package
 

I've found some spare time :)
http://i50.tinypic.com/2yzj6tl.png

However, it isn't finished just yet. The packaging still needs to be updated, as Harmattan requires some special attention regarding messing with system binaries (with kind regards from aegis). Then everything still has to be tested. Anyway, when it's done you'll be able to install and use it just like busybox-power on Fremantle.

Re: [Announce] Enhanced BusyBox package
 

You can upload the package to MAG's repository for applications that require Inception.

Re: [Announce] Enhanced BusyBox package
 

Thanks for the suggestion. However, busybox-power itself won't need to be incepted. Nokia deployed, just like in Fremantle, a metapackage in Harmattan that depends on specific versions of system packages, including busybox.
That means we have to replace /bin/busybox in busybox-power's postinst to not break any (C)SSU updates while still being able to provide busybox-power updates via community repositories, just like busybox-power did in Fremantle.

Busybox-power for Harmattan will depend on inception/opensh and aegisctl though. It requires either inception (more specifically: ariadne) or opensh to replace /bin/busybox, update aegis' refhashlist, and resign the list.

Aegisctl is required to temporarily allow untrusted binaries to be executed. Aegis loads all reference hashes in the kernel when booting your device. This means that during the period between replacing /bin/busybox and the next reboot, access to /bin/busybox would otherwise be denied for root. That's highly undesirable of course, hence the dependency on aegisctl.

Re: [Announce] Enhanced BusyBox package
 

A general notice: next CSSU release will ship a new version of Maemo's default BusyBox; one with portait mode support. This means that your /bin/busybox will be changed to CSSU's version.

Simply reinstalling busybox-power (either graphically or via the terminal) will get you busybox-power again. I'll make sure to incorporate all patches from CSSU's busybox in busybox-power so you won't miss out on any new functionality or bugfixes.

There is one important detail, however! If you've made manual modifications to Maemo's critical scripts (e.g. /sbin/preinit) that made those dependant on busybox-power during boottime, you will have to revert those prior to updating CSSU. Otherwise you could risk a boot loop.

Lastly, upon removal or reinstallation of busybox-power, busybox-power will detect and warn you if /bin/busybox has been changed. You can safely ignore this warning, as we know what has caused it (CSSU update changing /bin/busybox).

The above message will be included in the first post of this thread.

Edit: I've updated the warning message mentioned above in busybox-power 1.20.2power1. It now states why /bin/busybox could've been modified, and explicitly mentions CSSU as a possible cause. Just to comfort users who may have missed this post :).

Re: [Announce] Enhanced BusyBox package
 

It's a pity, that CSSU doesn't use it as great opportunity to incorporate busybox-power. Not much sense in upgrading busybox with portrait support *only*.

/Estel

Re: [Announce] Enhanced BusyBox package
 

replacing busybox by busybox-power eats rare precious space in rootfs (due to larger binary size of busybox-power), since busybox is needed during early boot and thus has to live in rootfs. I.E. it can't get optified.
Busybox-power however doesn't - by default - bring anything new that might justify indiscriminate waste of rootfs space for it, on all CSSU installs.
It's absolutely reasonable to keep busybox-power optional for those users who think they need it and want to accept the rootfs penalty it comes with.

/j

Re: [Announce] Enhanced BusyBox package
 

We are talking about 500k !
Not really meesing up rootfs space, or?

I, personally, can not imagine "living" with standard busybox and its limitations anymore. But can also live with ""optional"" installation, but would/do see this package as an (default) enhancement, also to CSSU.

Re: [Announce] Enhanced BusyBox package
 

Does the CSSU busybox have the fix for the non-persistent history bug? If not were should it be reported?

Long version: default maemo busybox forgets history arbitrarily. This was not fixed until some recent version of cssu and when I asked I was told to install busybox-power which indeed does not suffer from this bug. I have b-p ever since and I don't know if this bug has been fixed in CSSU.

Re: [Announce] Enhanced BusyBox package
 

False - busybox power brings tons of upstream bugfixes and optimizations. Obviously, without breaking anything

Also, it's funny, that You started to care for few kb in root space - then, you should be major advocate of bringing thumb2 support to CSSU, as it saved *much* root space.

Now, waiting for You to bring "compatibility with scripts relying on bugs" argument, that You have used on IRC...

/Estel