Microb and TLS 1.2
 

More and more servers now require TLS 1.2 which microb doesn't currently support. I am willing to help write code (or adapt existing code) if we can figure out what needs to be done.

"use another browser", "use maemo-leste" and similar options are not acceptable to me, I want to make microb (and gecko engine/nss) that are currently running on my N900 support TLS 1.2, I do not want to change browser or OS.

Re: Microb and TLS 1.2
 

i think we're all with you and would most definitely love it if that succeeded (and Leste evangelist that I am, Leste would also most likely benefit from such improved microb).

Re: Microb and TLS 1.2
 

@jonwil I thought you had once a deep look into this microb engine.

I think what I know you know already.:)

For sure you need to get new libnss3 running.

So have a look into:
libnss3
microb-engine
libmaemosec-certman0
microb-eal
libssl

microb white paper
microb browser architecture

Re: Microb and TLS 1.2
 

I did find https://bugzilla.mozilla.org/show_bug.cgi?id=480514

There's some work in Leste repo on updating maemo-security-certman to use openssl 1.1.0. Not sure if that will help out as well in the long run to keep components and certificates up to date. Some Leste components like that could be backported to Fremantle to aid in testing.

Re: Microb and TLS 1.2
 

I think the best way forward is as follows:
1.Figure out what version of NSS or what NSS tag or what NSS revision matches what's in microb-engine (we know what Gecko revision matches microb-engine but we need to match that back to the NSS repository)
2.Figure out of any of the local patches made to the microb-engine source code apply to NSS and whether those patches need to be handled going forward.
3.Grab the oldest release of NSS that has working TLS 1.2 support (actual official release by the NSS team, not just a repository revision)
4.Examine both sets of code and figure out what has changed that will affect microb-engine (e.g. has the public API changed in ways that matter)
and 5.Once we know what's changed, figure out whether its possible to port microb-engine to the newer NSS and then do it.

As far as I know, everything that talks to the NSS libraries is FOSS so the ABI doesn't matter, just the API.

Re: Microb and TLS 1.2
 

Ok so I have identified that microb-engine is using a Gecko revision taken from somewhere in the mozilla-1.9.2/Firefox 3.6.x tree and is using NSS_3_12_6_RTM.
The TLS 1.2 support didn't land until NSS 3.15.1 and the first version of Firefox to use that is Firefox 24.

Based on what I have observed, we have several options here as a way forward:
1.Start from the mozilla-central commit for NSS 3.12.6 RTM, move forward in time and look for any commits that update NSS (or that are necessary in order to update NSS to newer versions) and try and back-port those changes to microb-engine

2.Take the Gecko base code from Firefox 24.x and figure out if we can make it work as a replacement for microb-engine ([porting various local microb patches to that code, identifying if anything has changed API or ABI wise in the header files that get used by the stuff outside of microb-engine, figure out if any maemo-specific bits have been removed from the codebase etc) then if we can make it work (either as-is or with some porting work on the things that talk to microb-engine) use that to replace microb-engine.

or 3.Try to get Fennec 24.x working on Maemo5 and modify it so it can slot in as the "system" browser but with microb-engine remaining around for maps and rtcom-messaging-api and such (again identifying if there are any bits removed that matter, any maemo-specific changes we need to make/port across/whatever). I dont know when Fennec dropped all the bits needed for Maemo/Linux and became "Firefox for Android" but I have seen a Fremantle port of Fennec 17 so that might be a place to start.

The goal of this work is to allow people who are using Maemo Fremantle (and who are interested in the work) to have a browser that can at least connect to web sites using the latest security protocols and (depending on how things are done) maybe also get some browser improvements in there as well so it can do better on rendering web pages.

Re: Microb and TLS 1.2
 

There IS an option 4: Find all the patches needed for TLS 1.2 support in NSS and mozilla and back-port those to microb-engine somehow.

Oh and for those who say "forget about microb and stuff and use an existing modern replacement", what modern replacement do you suggest people on Maemo Fremantle use?

The newest Fennec build currently available for Maemo is Fennec 17 and that's not new enough to support TLS 1.2 and the newest security stuff so there is no existing Gecko browser that we can use.

All the new-enough webkit based browsers out there would need just as much porting work to run on Maemo as a newer Gecko does so that's not an option. And Leste isn't usable as a daily driver yet (and doesn't have a browser yet anyway) so switching to a new OS is also not an option. (and obviously "get a new phone" isn't the answer either since people using the N900 in 2018 are almost certainly using it because they like it and prefer it to iOS and Android :)

Re: Microb and TLS 1.2
 

Maemo code wasn't completely removed until 26.

https://bugzilla.mozilla.org/show_bug.cgi?id=906072

Re: Microb and TLS 1.2
 

There are other bugs related to Maemo support including
https://bugzilla.mozilla.org/show_bug.cgi?id=653201 (cant tell what version that was removed in)
https://bugzilla.mozilla.org/show_bug.cgi?id=1080529 (although that looks like a website change rather than a code change so we can ignore it)
and https://bugzilla.mozilla.org/show_bug.cgi?id=648156 (which looks like it was removed in Firefox 5.x so long before the build of FF we need to be worried about, this one looks like the important one)

And there may be other bugs in there that I haven't found yet that are relavent to us.

Re: Microb and TLS 1.2
 

After some thoughts I would say option 2 is the best way forward (get FF24 Gecko code running in microb-engine).
To pull it off we need to
A.Identify which of the Maemo local patches need to be forward ported to FF24 and port them
B.Identify what needs to change in the debian packaging to make FF24 work
C.Identify any breaking changes to API or ABI between microb-engine and FF24 and find a solution to those somehow
D.Identify anything important that got removed between microb-engine and FF24 (such as the above linked bugs) and find a solution.
and E.Identify any cases where FF24 needs newer versions of libraries than microb-engine and find a solution to that somehow.

Re: Microb and TLS 1.2
 

Best codebase to go with would probably be the 24.8.1 ESR release at https://hg.mozilla.org/releases/mozilla-esr24/ as it would have had security fixes and such back-ported to it.
It doesn't contain the removals from bug 906072 so we are good there I hope.

Re: Microb and TLS 1.2
 

I know embedlite (QT) built ok up to 29 development when they removed QT4 support. if I recall correctly I did try GTK build then but it didn't build because of it made user of features in newer GTK.

Re: Microb and TLS 1.2
 

If we are going to replace microb-engine (the best option IMO), we need to stick with GTK as the closed UI blobs send GTK objects (or rather xid's of GTK objects) to the browser daemon and vice versa. I see no reason we can't back-port the specific things in GTK 2.18 that FF24 needs back to the Maemo GTK 2.14 tree.

Re: Microb and TLS 1.2
 

Oh and if we dropped support for Flash (the version we have in Maemo is ancient and has who knows how many security flaws and such) we could probably avoid the need to care about bug 653201 and related items (IIRC the Flash plugin is the only thing that used it)

Re: Microb and TLS 1.2
 

Flash gone would be fine. Update would allow fixes for other plugins such as adblock. Versions supporting microb had awful memory leak iirc.

Agree about GTK. Only mentioned embedlite because I know dependency wise, other than GTK, we should be OK.

Re: Microb and TLS 1.2
 

I have compared the microb-engine sources from https://github.com/community-ssu/microb-engine to the matching Mozilla 1.9.2 sources and if we ignore the debian folder (and the patches in there), the only other differences are the change for
https://github.com/community-ssu/mic...7daa7b8e2cfa63
and the change for https://github.com/community-ssu/mic...405a7e12865ea2

Not sure if we need either of these changes in the new microb-engine or not.

In terms of the local patches applied by the build process, the list is in https://github.com/community-ssu/mic...series.microb2

We need to identify which of these changes already exist in FF24, which are no longer needed and which we still need going forward.

The biggest problem is going to be identifying all the bugs on bugzilla.mozilla.org that we may need to do something with (we found a few but finding them all will be difficult I suspect)

Re: Microb and TLS 1.2
 

I am analyzing the patches listed in the previously linked series.microb2 file. That list contains 114 patches. These are the patches I have analyzed so far and what I have found.

http://browser.garage.maemo.org/docs/patches.html is also potentially useful since it describes some of the patches.
Also https://bugzilla.mozilla.org/show_bug.cgi?id=401821 has lots of links to useful bugs in bugzilla.mozilla.org.

Patches that match to a bugzilla.mozilla.org bug: (the ones I have found so far anyway)
002_drawingMissingGlyphs.diff bug 463927 (fix never committed, looks easy to forward-port)
010_BUG376279_security_cross_compile.diff bug 376279 (build system has been changed so much in FF24 its unclear if this is needed or not)
010_BUILDFIX_embedding_tests.diff bug 93213 (related to the now removed gtk embedding stuff, will need to be sorted out as part of bringing that back)
020_nspr_dpkg_cross.diff bug 376256 (build system has been changed so much in FF24 its unclear if this is needed or not)
020_nsWindowCreator2_iface.diff bug 437018 (related to the now removed gtk embedding stuff, will need to be sorted out as part of bringing that back)
023_PC_CONFIG_sqlite_neededby_libsoftokn.diff bug 463928 (unclear if we still need this for Fremantle or not)
030_host_libidl_dpkg_cross.diff bug 401831 (build system has been changed so much in FF24 its unclear if this is needed or not)
130_456015.certview.diff bug 456015 (fix never completed, looks easy to forward port)

More analysis to come later.

Re: Microb and TLS 1.2
 

More patches that match to a bugzilla.mozilla.org bug:
040_netwerk_buffer_count.diff bug 401836 (unclear if we still need this, too many changes to the networking code to tell where this code has moved to)
675_unknown_protocol_handling.diff bug 401848 (fix never committed, looks easy to forward-port)
GConfProxySettings.diff bug 458211 (fix never committed, dont know if we still need it, can't find where the relavent code has moved to)
020_BUG349921_toolkit_branding.diff bug 401851 (build system has been changed so much in FF24 its unclear if this is needed or not)
JSBreakExperimental.diff bug 449109 (fix never committed, looks easy to forward-port)
150_attachment.cgi?id=342275.diff bug 459078 (fix never committed, dont know if we still need it, can't find where the relavent code has moved to)
854ab30d101e.diff bug 513544 (already in FF24)
92422.workaround.diff bug 469667 (fix never committed, looks easy to forward-port)
attachment.cgi?id=408390.diff bug 485252 (already in FF24)
attachment.cgi?id=418542.diff bug 536031 (already in FF24)
attachment.cgi?id=419012.diff bug 536560 (already in FF24)
attachment.cgi?id=419536.diff bug 507811 (already in FF24)
attachment.cgi?id=45208.diff bug 532626 (fix never committed, dont know if we still need it, can't find where the relavent code has moved to)
attachment.cgi?id=45209.diff bug 532627 (already in FF24)
attachment.cgi?id=46259.diff bug 488046 (already in FF24)
BMO530075_connectivity_ui_fix.diff bug 530075 (fix never committed, looks easy to forward-port)
bug137606.diff bug 523941 (fix never committed, doesn't look important since its just string changes)
bug143563.diff bug 532612 (already in FF24)
bug491722.diff bug 491722 (already in FF24)
bug505018.jemalloc.diff bug 505918 (related to the now removed gtk embedding stuff, will need to be sorted out as part of bringing that back)
Bug513630.workaround.diff bug 513630 (related to the now removed gtk embedding stuff, will need to be sorted out as part of bringing that back)
bug513736.wr.diff bug 513736 (already in FF24)
bug533950_wr.diff bug 533950 (fix never committed, dont know if we still need it, can't find where the relavent code has moved to)
bug546074.diff bug 546074 (already in FF24)
bug588724.diff bug 588724 (fix never committed, looks easy to forward-port)
CheckCompatibilitySave.diff bug 532620 (fix never committed, the relavent code is gone now so we dont need this I guess)
conic.diff bug 532072 (already in FF24)
connectivity.diff bug 532078 (fix never committed, looks easy to forward-port)
DBFlush_systemIdle.diff bug 518804 (already in FF24)
disable_refresh_in_bg.diff bug 518805 (fix never committed, looks easy to forward-port)
enable_16bpp_format.diff bug 386440 (unclear from the bug if its in FF24 or not or what we need to do about it)
gnomevfs_ext_without_gnomevfs.diff bug 532621 (already in FF24)
hildon_mime_missing.diff bug 532610 (seems to not be required anymore due to other code changes)
idleservicetimer.diff bug 559784 (seems to not be required anymore due to other code changes)
LowMemoryManageImprovements.diff bug 532623 (seems to not be required anymore due to other code changes)
nb128288.diff bug 532614 (already in FF24)
nolibxul_pkgconfig.diff bug 482104 (fix never committed, looks easy to forward-port)
osso-mem.diff bug 532608 (already in FF24)
UICallback_CC.diff bug 508518 (already in FF24)
watchdog.diff bug 477850 (seems to not be required anymore due to other code changes)

"already in FF24" means the patch was committed to mozilla-central with a milestone that pre-dates FF24 and therefore the fix doesn't need to be applied.

There are a total of 47 patches that I can match to a bugzilla.mozilla.org bug (all of which I have mentioned here).

I suspect all of the remaining 67 patches will need to be manually checked (figure out if they need to be forward ported to FF24, rewritten to do something totally different or just ignored)

Re: Microb and TLS 1.2
 

I have now taken the Firefox 24.8.1 ESR source code, copied over the /debian folder from microb-engine, disabled all the patches (since they obviously wont apply to FF24 cleanly) and tried to build. Doesn't get very far before complaining about a syntax error on line 80 of python/mozbuild/mozbuild/controller/clobber.py
with open(self.src_clobber, 'rt') as fh:

Turns out we need Python 2.7 (all the other dependencies listed on the "linux prerequisites" page that matches Firefox 24 seem to be correct/new enough/whatever, its just Python that is out of date.
But it looks like http://talk.maemo.org/showthread.php?t=91341 has instructions for getting Python 2.7 going so we should be good.

Re: Microb and TLS 1.2
 

And now I found Python 2.7 in extras-devel. That should work :)
And yep, that works, now to find a newer GCC :)

Re: Microb and TLS 1.2
 

Documenting the steps I am taking to set up the build environment that I will be using to compile Firefox 24 source code. I am doing this on the Maemo Fremantle Ubuntu dev VM, if you have a different setup the commands to run may be different:
Items in <> are things you need to type in or do, other lines are commands to be run in the shell.

sudo su
<input password for user maemo>
cd /scratchbox/compilers
wget http://maemo.merlin1991.at/cssu/deve...armv7a.tar.bz2
tar xvf linaro-4.7-2012.07-fremantle-armv7a.tar.bz2
rm linaro-4.7-2012.07-fremantle-armv7a.tar.bz2
exit
/scratchbox/login
sb-conf st FREMANTLE_ARMEL_GCC472 -c linaro-4.7-2012.07-fremantle-armv7a -d qemu:perl:debian-etch:doctools:svn:git -t qemu-arm-sb
sb-conf se FREMANTLE_ARMEL_GCC472
wget http://repository.maemo.org/stable/f...36-2_armel.tgz
sb-conf rs maemo-sdk-rootstrap_5.0_20.2010.36-2_armel.tgz
sb-conf in -edL
rm maemo-sdk-rootstrap_5.0_20.2010.36-2_armel.tgz
dpkg -i /scratchbox/compilers/linaro-4.7-2012.07-fremantle-armv7a/packages/libfakeroot_1.14.4-0+sb1_armel.deb
apt-get update
fakeroot apt-get install maemo-sdk-debug
<add deb http://repository.maemo.org/ fremantle/******************************** nokia-binaries line to /etc/apt/sources.list>
apt-get update
fakeroot apt-get install nokia-binaries nokia-apps
rm -rf /targets/FREMANTLE_ARMEL_GCC472/opt
mkdir /targets/FREMANTLE_ARMEL_GCC472/opt
apt-get install dbus-glib-1-dev
apt-get install autoconf2.13
apt-get install libidl-dev
apt-get install automake1.7
apt-get install python2.5
apt-get install quilt
apt-get install bc
apt-get install sharutils
<add deb http://repository.maemo.org/extras-devel/ fremantle free line to /etc/apt/sources.list>
apt-get update
apt-get install python2.7
apt-get install libogg-dev
apt-get install libvorbis-dev
apt-get install libtheora-dev
<remove deb http://repository.maemo.org/extras-devel/ fremantle free line from /etc/apt/sources.list>
<add deb http://maemo.merlin1991.at/cssu/community-thumb/ fremantle free non-free to /etc/apt/sources.list>
apt-get update
apt-get install gcc-4.7-base
apt-get install libgcc1
apt-get install libstdc++6-dev
apt-get install libstdc++6-dbg
<remove deb http://maemo.merlin1991.at/cssu/community-thumb/ fremantle free non-free from /etc/apt/sources.list>
<add deb http://repository.maemo.org/community-testing/ fremantle free line to /etc/apt/sources.list>
apt-get update
apt-get install libpixman-1-dev
<clone git repository git@github.com:jonwil/microb-engine-ff24.git into your scratchbox home folder>
cd microb-engine-ff24
dpkg-buildpackage -rfakeroot -b

Re: Microb and TLS 1.2
 

Updated the build instructions for installing the libs needed for GCC 4.7.2 and updated the code to switch to using the in-tree version of Cairo 1.10 (the one local patch to Cairo 1.8.8 that matters is already in Cairo 1.10 upstream so we are good there)

Re: Microb and TLS 1.2
 

Also had to add --disable-elf-hack to the mozconfig since it didn't seem to work right on Fremantle.

Re: Microb and TLS 1.2
 

Updated the build instructions again (we dont need to install libjpeg but we do need to install libogg, libvorbis and libtheora)

Re: Microb and TLS 1.2
 

I think this beast might actually build(!)

Re: Microb and TLS 1.2
 

Fixed a piece of bit-rotting in some Maemo specific code.
Fixed some typos in moz.build that were causing the OpenGL code to not compile properly (it was trying to build some GLX code even though Maemo needs EGL not GLX)

Re: Microb and TLS 1.2
 

Now I am stuck because I need GStreamer 0.10.26 and we only have GStreamer 0.10.25.

Re: Microb and TLS 1.2
 

Have disabled gstreamer in the mozconfig for now and will worry about any resulting bustage later once I actually get this thing to build...

Re: Microb and TLS 1.2
 

Have now disabled some other media stuff in the hope that it will no longer be giving me libogg issues.

Re: Microb and TLS 1.2
 

Had to back-port part of the fix from mozilla bug 928547 and remove the --enable-necko-protocols line from the mozconfig (the default settings in FF24 are fine for our needs and the settings in the mozconfig were disabling some protocols that we actually need enabled including websocket and wyciwyg). Also had to give my vmware VM 3.5GB of ram (the maximum vmware will let me give it on my 8GB system) along with 6GB of swap.

But with those things done, hopefully libxul will now finally link and then the build process can continue (and hopefully give me a set of deb files at the end of it all)

Re: Microb and TLS 1.2
 

IT WORKED!.
The beast actually compiled(!)

Re: Microb and TLS 1.2
 

I have now gotten things to run using the bits built from this effort (including the xulrunner binary it built) and the latest version from the http://conkeror.org/ git. It is slow as molasses for some reason I have yet to identify (top showed it taking up to 90% or more CPU at times) even though it should be an optimized build. And there are definitely bugs. But at least I know that Firefox 24.8.1 DOES run on the Nokia N900.

Re: Microb and TLS 1.2
 

Now that I know the thing not only compiles but works (to the point I can browse web pages using TLS 1.2 at least) the next step is to figure out why the debian packaging isn't working right and get some .deb files. After that, I can try and compile various bits that use microb-engine and see what happens.

Re: Microb and TLS 1.2
 

Ported a few patches over from the microb-engine patch set:
002_drawingMissingGlyphs.diff (mozilla bug 463927)
010_MICROB_paths_and_names.diff (this one is what's causing the packaging to fail, now that I ported it the packaging should find what its looking for where it needs it I hope)
050_MICROB_autoset_grehome_display.diff (makes it easier to use xulrunner)
130_456015.certview.diff (mozilla bug 456015)
92422.workaround.diff (mozilla bug 469667)
BMO530075_connectivity_ui_fix.diff (mozilla bug 530075)
BUG153489_increase_max_script_runtime.diff (increase max script runtime)
bug588724.diff (mozilla bug 588724)
connectivity.diff (mozilla bug 532078)
HiddingLibxulSymbols.diff (hides symbols that aren't needed to be exported)
JSBreakExperimental.diff (mozilla bug 449109)
nb162660.diff (changes a function that gets the physical memory size so it only gets the size once and returns the stored size every time)

No clue if any of this stuff actually compiles, I haven't tried it yet but I will do so soon. The next steps are to get this to compile and produce deb files then to look into ABI/API changes, see which users of microb-engine still compile (and what breaks there) and look into getting the gtkmozembed stuff working again (I may need to get that working before I get working deb files)

Oh and for anyone following this thread, DO NOT replace any system binaries with any binaries you may get from building any code specified in this thread. (I learned that the hard way when I had to use RescueOS to put the old contents of /usr/lib/microb-engine back when the stuff built from this new code made the phone go into a boot loop due to browserd being unable to load).

Re: Microb and TLS 1.2
 

Oh and identifying all the users of the microb-engine bits is also high on the todo list so I know which closed source bits might cause ABI/API issues (the biggest problems will likely be nokia-maps, the various addons/plugins and possibly the closed-source browser UI)

Re: Microb and TLS 1.2
 

After some analysis I have identified the following closed-source packages as those that would be affected by any ABI changes (i.e. they link directly to microb-engine libraries or talk to microb-engine in some other way)
nokia-maps-core (nokia maps core plugins and libs)
camel-as-provider-0 (provides Microsoft ActiveSync email support for modest I believe)
libssoautologin (single sign on stuff, not sure what for, likely something Nokia/Ovi)
adobe-flashplayer (Flash player plugin)
tablet-browser-default-plugin (default plugin for microb, not sure what this does exactly)
tablet-browser-mediaplayer-plugin (media player plugin for microb)

Re: Microb and TLS 1.2
 

Getting closer to working packages.
Had to fix a few of the back-ports and remove the back-port for HiddingLibxulSymbols.diff since I couldn't make that patch work (gave me linker errors on libxul)
Just need to figure out what the packaging is looking for but can't find and why it can't find it.

Re: Microb and TLS 1.2
 

xulrunner-dev ?

Re: Microb and TLS 1.2
 

I managed to get elfhack working so that's a little progress.

Although I am very close to saying "screw it" and looking for a fork of webkit or similar that works on our ancient libraries but supports the features needed (TLS 1.2 in particular but also more modern HTML/CSS/JS so it can render web pages that microb can't do).

I dont know of anything suitable though...

Re: Microb and TLS 1.2
 

And what happened then?