[Announce] modRana: a flexible GPS navigation system - Page 185 - maemo.org

MikeHG | # 1841 | 2015-08-14, 11:33 | Report

I tried messing with constants.py, but didn't get anywhere useful.

You can remove the COW attribute from a directory, without having to create a whole new subvolume, which sounded a lot less scary. So I closed modRana, mv'ed /home/nemo/.local/share/harbour-modrana/maps out of the way, recreated the directory, ran 'chattr +C' on it (you should theoretically be able to copy the map tiles back into the new directory, and they should inherit the attribute... but I didn't try that), and finally restarted modRana.

So far no major drama, performance seems *slightly* better, and it seems harder (though not impossible) to cause the complete lock-ups... but it could be placebo, or it could be the fact that I recreated the database afresh. I guess I'll find out after a few days.

(please be very aware that I don't really know what I'm doing ... so if you try this and it borks your phone, it's your fault, not mine

but it stands to reason it should work I think, and I can't really see how it could break things too horribly...)

---

eta- two possibly useful things to note: When the program does lock up, one seemingly reliable way to unlock it is to wait a few minutes, then switch to another main map layer, pan / zoom around a bit, and switch back again. And if you run netstat in a terminal when it's locked up, you get a lot of connections to mapservers in a "CLOSE_WAIT" state, which seems like it could be a clue for a programmer.

Last edited by MikeHG; 2015-08-15 at 14:14.

The Following 2 Users Say Thank You to MikeHG For This Useful Post:
MartinK, Wikiwide

Shadowdog | # 1842 | 2015-08-22, 16:18 | Report

I guess there wont be any further updates for modRana for Meego!?? Is it possible to use offline map data, lets say from here http://download.geofabrik.de/ ?

The Following User Says Thank You to Shadowdog For This Useful Post:
Wikiwide

int_ua | # 1843 | 2015-08-25, 00:20 | Report

CLOSE_WAIT looks familiar. Are any tiles retrieved over HTTPS?

The Following 3 Users Say Thank You to int_ua For This Useful Post:
handaxe, MartinK, Wikiwide

handaxe | # 1844 | 2015-09-04, 23:08 | Report

As promised, I have started to look at map layers that are not returning tiles. I will add to this list, cumulatively, as I plow through (nope, no need - finished!). Using,debug logging to file, I am examining the errors generated, then checking the logged URLs in a browser to see if anything revealing comes up. So far:

1) Yandex - returns tiles, but never for the correct coords - what system is it using?
2) Yahoo - fails, service unavailable (prolly true for all map types)
3) Virtual Earth - fails invalid request URL, URL seems malformed (true for all map types)
4) MapQuest Satell. - Service Unavailable (MapQuest Map OK!)

Debug logfile for the above available, if needed.

Not tested: Freemap.sk, Czech.

In addition: batch download around a route appears as if it is completing (and certainly takes it's time) but subsequent off-line driving quickly shows that many of the tiles are not available, at the zoom levels requested. I know this for certain but will need to run a separate test with debug logging enabled to see wots wot.

The Following 4 Users Say Thank You to handaxe For This Useful Post:
kureyon, MartinK, skanky, Wikiwide

handaxe | # 1845 | 2015-09-05, 10:04 | Report

By the way, I tried v 0.52.10 on a low end Android tablet and map tile display was porked, with every 2nd tile displayed along a row and then a row missing altogether and so on. OSM Cycle but I think it affects all.

"Sent from Ubuntu Touch MX using the Forum Browser app"

The Following 2 Users Say Thank You to handaxe For This Useful Post:
MartinK, Wikiwide

MartinK | # 1846 | 2015-09-05, 11:39 | Report

Originally Posted by handaxe View Post

As promised, I have started to look at map layers that are not returning tiles. I will add to this list, cumulatively, as I plow through (nope, no need - finished!). Using,debug logging to file, I am examining the errors generated, then checking the logged URLs in a browser to see if anything revealing comes up. So far:

Thanks!

Originally Posted by handaxe View Post

1) Yandex - returns tiles, but never for the correct coords - what system is it using?

Yeah - it looks like some sort of northern offset. If we can find how the offset depends on zoom level (or other variables) I should be able to compensate for it when generating tile coordinates.

Originally Posted by handaxe View Post

2) Yahoo - fails, service unavailable (prolly true for all map types)

Weird, I'm pretty sure they have still been working quite recently.

Originally Posted by handaxe View Post

3) Virtual Earth - fails invalid request URL, URL seems malformed (true for all map types)

This should already be fixed in the latest modRana source code, but has not yet been released.

Originally Posted by handaxe View Post

4) MapQuest Satell. - Service Unavailable (MapQuest Map OK!)

Debug logfile for the above available, if needed.

Not tested: Freemap.sk, Czech.

In addition: batch download around a route appears as if it is completing (and certainly takes it's time) but subsequent off-line driving quickly shows that many of the tiles are not available, at the zoom levels requested. I know this for certain but will need to run a separate test with debug logging enabled to see wots wot.

I have unfortunately not tested downloading tiles around a route in ages, so it is quite possible it is indeed broken. :P So I've added a tracking issue for this and will check what's wrong.

Originally Posted by handaxe View Post

By the way, I tried v 0.52.10 on a low end Android tablet and map tile display was porked, with every 2nd tile displayed along a row and then a row missing altogether and so on. OSM Cycle but I think it affects all.

Check that you have map scaling disabled (set to 1x) in Options->Map. It is unfortunately broken at the moment and behaves the way you describe - I really need to finally fix it.

The Following 5 Users Say Thank You to MartinK For This Useful Post:
handaxe, kureyon, mike727, skanky, Wikiwide

MartinK | # 1847 | 2015-09-05, 12:27 | Report

Originally Posted by MikeHG View Post

Two questions:

1) Do normal map tiles 'expire'? It'd be ideal if the app could check whether tiles were up to date if they're older than a certain age and still being used, though if that isn't possible just having an option to flush them after a year or so would be good.

By default tiles never expire, but a timeout property can be set for a layer to set expiration for tiles corresponding to the layer.

Some layers that have frequently changing data include the timeout property by default and users can of course also set in themselv in the configuration file (which is in /home/nemo/.config/harbour-modrana on the Jolla).

Also an important note about how the timeout logic is currently implemented in modRana - it currently only prevents loading of expired tiles from local storage (so that dynamic map layers don't show stale data). No scrubbing of stale files has been implemented yet, even though the needed data for it (timeout value & tile timestamps) is available.

Originally Posted by MikeHG View Post

2) Don't copy-on-write filesystems have terrible performance for databases? I read older versions of BTRFS were notorious, and Sailfish has an oldish one...

According to the official BTRFS wiki it is still the case:

Originally Posted by

Files with a lot of random writes can become heavily fragmented (10000+ extents) causing trashing on HDDs and excessive multi-second spikes of CPU load on systems with an SSD or large amount a RAM.

Originally Posted by MikeHG View Post

I'm wondering whether creating a subvol without COW for the maps, or perhaps just tuning down the write frequency massively in the config file might help performance (not sure I'm brave enough to try the former - and I'm certainly not suggesting anyone else should, unless they know what they're doing...)

Note that even though BTRFS has the nocow mount option, it has serious drawbacks and gotchas:

Originally Posted by

Disable it by mounting with nodatacow. This implies nodatasum as well. COW may still happen if a snapshot is taken. However COW will still be maintained for existing files, because the COW status can be modified only for empty or newly created files.

And that assumes everything works correctly and you don't hit any of the miriad BTRFS bugs while doing this.

My suggestion would be to use a uSD card formated with a sane filesystem (EXT4) and store your tiles there.

You can achieve this either by changing the map storage path in:
/home/nemo/.config/harbour-modrana/user_config.conf
Or you can just symlink the folder modRana uses for map data storage by default to somewhere on the uSD card:
/home/nemo/.local/share/harbour-modrana/maps

Originally Posted by MikeHG View Post

Love the app, but hitting some weird problems on sqlite where it seems to refuse to load tiles it has clearly loaded before. Still, I realise it's early days yet for the port. I'll try to capture some debug output if I can.

Have you tried switching to the "files" storage method in Options->Map? That makes modRana store tiles as files in a folder structure. AFAIK unlike database files BTRFS is quite sane when handling many small files.

Originally Posted by MikeHG View Post

This is a log from when the database seems to 'lock up', apparently refusing to load tiles it has already downloaded. I'm not 100% sure how to trigger it, but generally scrolling around and zooming in and out in an area I haven't yet downloaded works reliably eventually. I've left it in that state for 1/2 hour probably, and it doesn't recover - though interestingly, it does seem to be able to load other tiles sometimes (e.g. if I zoom really far out) - it just seems to choke on certain particular areas of map, but after closing the program and relaunching it, and sometimes even just by zooming out then back in again, those areas of map will draw fine.

I have also hit this q couple of times - but as it seems to happen quite at random I have not yet been able to fully pin it down yet. But a lot of zooming & panning around seems to be a factor.

Originally Posted by MikeHG View Post

http://pastebin.com/dVuPYC4k

Quite a few examples of this:

Code:

Traceback (most recent call last):
  File "modules/gui_modules/gui_qt5/gui_qt5.py", line 302, in _selectImageProviderCB
    return self._imageProviders[providerId].getImage(imageId, requestedSize)
  File "modules/gui_modules/gui_qt5/gui_qt5.py", line 599, in getImage
    log.exception()
TypeError: exception() missing 1 required positional argument: 'msg'
2015-08-13 00:53:58,644 ERROR mod.gui.qt5: tile image provider: loading tile failed
2015-08-13 00:53:58,652 ERROR mod.gui.qt5: mainMap/mapnik/13/4094/2727
2015-08-13 00:53:58,655 ERROR mod.gui.qt5: (-1, -1)
2015-08-13 00:53:58,658 ERROR mod.gui.qt5: image loading failed, imageId: tile/mainMap/mapnik/13/4094/2727
Traceback (most recent call last):
  File "modules/gui_modules/gui_qt5/gui_qt5.py", line 579, in getImage
    download=False)
  File "modules/mod_mapTiles/mod_mapTiles.py", line 211, in getTile
    tileData = self._storeTiles.getTileData(lzxy)
  File "modules/mod_storeTiles.py", line 369, in getTileData
    return self._getTileDataFromSqlite(lzxy)
  File "modules/mod_storeTiles.py", line 385, in _getTileDataFromSqlite
    result = self.getTileFromDb(lookupConn, dbFolderPath, lzxy)
  File "modules/mod_storeTiles.py", line 455, in getTileFromDb
    (z, x, y))
sqlite3.OperationalError: database is locked

Interesting! Thanks a lot for the log - I think I can do something about the loockups now.

Originally Posted by MikeHG View Post

Side note:

Due to a bug, it seems it's currently impossible to send the logs via the Jolla mail client using gmail, unless you rename them (removing the #).

https://together.jolla.com/question/...ent-via-gmail/

Voted!

(And others should too!)

Originally Posted by Shadowdog View Post

I guess there wont be any further updates for modRana for Meego!?? Is it possible to use offline map data, lets say from here http://download.geofabrik.de/ ?

Unfortunately no, at least not from me. Of course if someone from the community volunteers to move the MeeGo/Harmattan port forward I'm willing to help them get started. But I'm unfortunately far to overcommitted to work on it myself.

Last edited by MartinK; 2015-09-05 at 15:50.

The Following 7 Users Say Thank You to MartinK For This Useful Post:
DeadHorseRiding, handaxe, mike727, MikeHG, Shadowdog, skanky, Wikiwide

MikeHG | # 1848 | 2015-09-05, 17:49 | Report

Originally Posted by MartinK View Post

...
My suggestion would be to use a uSD card formated with a sane filesystem (EXT4) and store your tiles there.
...
Or you can just symlink the folder modRana uses for map data storage by default to somewhere on the uSD card:
/home/nemo/.local/share/harbour-modrana/maps
...

This is exactly what I did in the end, having realised just how much space the maps were going to end up taking up

I used 'chattr +r' for long enough to be reasonably confident that it's not going to explode though, so might be worth a go if anyone is still using BTRFS.

The Following 3 Users Say Thank You to MikeHG For This Useful Post:
MartinK, skanky, Wikiwide

marmistrz | # 1849 | 2015-09-07, 12:32 | Report

I'm unable to rebuild mapnik anymore. So I can't really fix the bad path. The build silently fails, see https://garage.maemo.org/builder/fre...log.FAILED.txt

Run standalone, the g++-4.6 command returns 1 without printing anything as well.

The Following 2 Users Say Thank You to marmistrz For This Useful Post:
MartinK, Wikiwide

zod | # 1850 | 2015-09-14, 12:38 | Report

Last month I wanted to cache tiles for several areas and best tool I found was Mobile Atlas Creator (older versions also allow you to download gmaps) .
Perl script attached that converts from RMaps/Big Planet SQLite format used in MOBAC to Modrana's SQLite. Actually its modified Beermad's script.

Attached Files

RMaps2modrana.pl.gz (1.5 KB, 76 views)

The Following 5 Users Say Thank You to zod For This Useful Post:
ggabriel, handaxe, MartinK, skanky, toben