encoding chars problems in Xterm
 

Hello,
I have some problems on Xterminal, I'm gonna try to explain it by giving you several examples.

First, I'm french, so I use specials chars and accents like "é", "è", "ç", "ê", "ë".

By mistake, I type on the terminal (because the "é" letter is next the "m"). So I erase the "é":
It means that the "é" has not really been erased on the command line!


second test: I've erased the "é", I move the cursor left until the beginning of the line. Another problem: the cursor moves one char before the start of the command line! I've tried with two "éé", and then the cursor moves on the "$" (and far left as the number of special chars). Then I move it rightmost I obtain:
with the cursor on the second "l". (the command line has been shifted to the left...)

Another example: try to insert a "é" char at the middle of a string (I place the cursor on the "y").
(and the command line is not shifted). So, I understand that the "é" char is encoded on 2 chars...

Another test: i type a command with the "é" letter, and i move left and right. Each time the line has been shifted to the left.

The special char take 2 spaces on the line, but is printed on one box...

Does anyone has already had this problem? Does a solution exists to fix it?

I hope you understand my problem.

Re: encoding chars problems in Xterm
 

Re: encoding chars problems in Xterm
 

How can they call it fixed if it isn't fixed for the N900 nor anything before it?

Re: encoding chars problems in Xterm
 

FIXED in Harmattan (back when Harmattan was going to be Maemo 6). But see iDont's enhanced busybox which should contain the upstream fix.

Re: encoding chars problems in Xterm
 

thanks to have responded so quickly!

I've installed the last version of "busybox-power" (09/11/2011), when I launch xterminal, it prints "BusyBox v1.19.2 (Debian 1.19.2power1)" (and I can use new useful tools), but for my previous problem nothing has changed, same symptoms...

Re: encoding chars problems in Xterm
 

Upstream BusyBox (as can be found in busybox-power) has much improved Unicode support over the stock BusyBox. However, Unicode support is turned off by default. You can verify this by executing "echo $LANG" in your shell. If the resulting string doesn't contain UTF8, then unicode is indeed turned off.

You need to start a new shell like:
to have Unicode support enabled in the shell ;)

Edit: since you're French, you probably want to replace en_EN with fr_FR in the code above.

Re: encoding chars problems in Xterm
 

Is there a way to change the default?

Re: encoding chars problems in Xterm
 

YESS! works perfectly

Not yet understood how the configuration works... (does this require to recompile busybox??)
Have found this http://busybox.net/FAQ.html#configure , will try tomorrow :)

Re: encoding chars problems in Xterm
 

Is there any reason this is not default? Been getting �'s in one application and this fixes it. Thanks!

Re: encoding chars problems in Xterm
 

You need to change the system's locale. I don't know how the set locale affects Maemo as a whole though, since it does quite a bit differently than your typical Linux distribution configuration wise. Anyway, editing /etc/osso-af-init/locale seems to do the trick. Only the line containing LANG needs to be changed in there. In my case, changing:
to
appears to work after a reboot.

The only known side effect I'm aware of is that Ctrl-Space is seen as a NULL character in the terminal, which doesn't happen when using a non-Unicode locale.

The proper fix is changing the system's locale setting. BusyBox merely respects what you've set there (launching a shell with LANG= basically overrides your default setting).

You can configure BusyBox to disrespect your system locale and force Unicode support though. To achieve this, run "apt-get source busybox-power" in Scratchbox, modify "CONFIG_FEATURE_CHECK_UNICODE_IN_ENV=y" in the two config files in debian/config/ to "CONFIG_FEATURE_CHECK_UNICODE_IN_ENV=n", and recompile busybox-power with "dpkg-buildpackage -r"fakeroot -u" -uc -us -nc"

I would encourage you to try changing your system's locale first however, rather than recompiling BusyBox with the above change. If you do want to recompile BusyBox though, feel free to contact me if you encounter any issues.

I don't know why Maemo hasn't got an UTF8 locale set by default. As to BusyBox, or any package for that matter, I prefer it to adhere the system's settings ;)

Re: encoding chars problems in Xterm
 

Out of curiosity, is there a listing of power features? We all might be missing on something. Good job, hope you keep it up.

Re: encoding chars problems in Xterm
 

There isn't a complete listing of all power features provided by busybox-power; the bulk of it is introduced by over three years of upstream BusyBox development. I can give you a global summary of busybox-power's features over those of the stock BusyBox though.

[shameless self-plug]
There are roughly four areas of improvement:

• Utilities - A lot more utilities are provided (e.g. iostat, wget, ip, httpd, crond, unix2dos, telnet, and ~180 more)
• Features - More features in most provided utilities (e.g. swapon with priority support and colored ls output) and more features in the default shell (e.g. much improved Unicode support, Ctrl-R reverse history search and proper shell history handling)
• Upstream fixes - Over 3 years of fixes (bug fixes, security fixes, ..) from upstream BusyBox. Also see upstream BusyBox' git log.
• Maemo fixes - Providing Maemo-specific bugfixes, e.g. bug #700, #5317, #4174, #4175, #7014

[/shameless self-plug]

Further busybox-power specific discussion (if any) should probably continue over at its own thread, since we're getting kind of off topic now.

Re: encoding chars problems in Xterm
 

Thanks for it iDont :-)

No more problem in terminal but with my application (smssend), have the same problems I used to before... (example: instead of "é", I get "é", and latin-1 chars)

Re: encoding chars problems in Xterm
 

You're welcome :)

I'm not sure if this is related to the terminal itself. Setting a locale enables your applications to know what language/charset you want to use, but this only works if the application respects it and handles the charset properly. While the shell itself might support Unicode perfectly fine, applications launched from it might not.

This post supports that smssend might be at fault, although the author hasn't responded yet at the moment. I'll keep an eye on the smssend thread.