[Announce] modRana: a flexible GPS navigation system - Page 96 - maemo.org

Active Topics

Is there a section to talk about Java ME? Apps, etc. (6)
to General by Kalatti - 4 days, 21 hrs ago
more...

Page 96 of 207

Thread Tools

jafd	2012-03-14 , 05:46
Posts: 29 \| Thanked: 26 times \| Joined on Oct 2009	#951

An idea worth considering: an empty tile takes 103 bytes as a file + storage overhead of a row in SQLite. At high zoom levels, OSM has lots of those. Maybe it's worth to have one more column in the lookup table to indicate an empty tile and thus process it more quickly?

Quote & Reply |

MartinK	2012-03-14 , 15:54
Posts: 1,548 \| Thanked: 7,510 times \| Joined on Apr 2010 @ Czech Republic	#952

Originally Posted by jafd

Yet another conclusion: it's a nice idea to pre-render some large chunks of the world at all zoom levels and distribute the resulting SQLite files. Shame no one has done it yet.

If you won't mind to share some of your country/are specific database files, I think I can arrange hosting on the modRana project website for them.

Users would then just download a corresponding database file and run the import script.

Of course a proper, more automated (building, distribution & import) solution tile package handling should be implemented eventually.

BTW, just got a and email from Beermad, the import script is back on-line !

Originally Posted by jafd

An idea worth considering: an empty tile takes 103 bytes as a file + storage overhead of a row in SQLite. At high zoom levels, OSM has lots of those. Maybe it's worth to have one more column in the lookup table to indicate an empty tile and thus process it more quickly?

Are they all really the same & how can they be reliably identified (eq, when downloading tiles) ? What about ocean tiles ?

So something like this ?

Code:

table tiles (z integer, x integer, y integer, store_filename string, extension varchar(10), unix_epoch_timestamp integer, empty integer, primary key (z, x, y, extension))

1 = empty
0 = nonempty

And possibly other values (like for ocean or other solid-color tiles). But that might be taking it too far & unnecessary complicating it (also something like that should be probably in the store database, not in the lookup one).

EDIT: On the other hand, solid color PNG tiles should be really small, so probably not worth the bother & database handling complications.

__________________
modRana: a flexible GPS navigation system
Mieru: a flexible manga and comic book reader
Universal Components - a solution for native looking yet component set independent QML appliactions (QtQuick Controls 2 & Silica supported as backends)

Last edited by MartinK; 2012-03-14 at 16:03.

Quote & Reply |

The Following User Says Thank You to MartinK For This Useful Post:
Joseph9560

jafd	2012-03-14 , 17:35
Posts: 29 \| Thanked: 26 times \| Joined on Oct 2009	#953

Yep, empty PNGs are just 103 bytes in size. But you can take advantage of preloading one at the very start, and rendering it wherever an empty tile is needed, thus conserving some time (and battery) otherwise spent on decoding. In terms of database disk size, getting rid of them would yield next to nothing.

As for sharing the pre-renders, I suspect it would take a few months to do them on commodity hardware (three days for Poland, zoom levels all the way to 15 and 16-18 for select cities); also, to achieve acceptable results, the database for the whole planet is needed (for some reason I've got proper administrative borders only around one of two countries).

Also the stylesheet I ended up with, default one from OSM-bright (which played well with imposm data) is bad, too low contrast to be viewed on a phone. Font size is also next to pathetic.

Attached Images

Last edited by jafd; 2012-03-14 at 17:54.

Quote & Reply |

The Following User Says Thank You to jafd For This Useful Post:
don_falcone

abubakar	2012-03-14 , 18:18
Posts: 242 \| Thanked: 97 times \| Joined on Sep 2009	#954

I installed modrana few weeks back and recorded 3 tracks. Then I used the "show on map" feature to plot them on the maps, all 3 of them. I dont know when after that this problem started that modrana is not downloading any tiles anywhere, I have tried zooming in/out a lot, change map location, restarted the device several times, the tiles just dont load anymore and all I see is the "Loading" in each tile. What can I do to provide the debugging info so this can be debugged. I was thinking I should uninstall and install, but I have not done that yet.

__________________
so i guess the the lesson learned is: "if you want a thing done well, do it yourself"

Quote & Reply |

The Following User Says Thank You to abubakar For This Useful Post:
MartinK

MartinK	2012-03-14 , 22:07
Posts: 1,548 \| Thanked: 7,510 times \| Joined on Apr 2010 @ Czech Republic	#955

Originally Posted by jafd

Yep, empty PNGs are just 103 bytes in size. But you can take advantage of preloading one at the very start, and rendering it wherever an empty tile is needed, thus conserving some time (and battery) otherwise spent on decoding. In terms of database disk size, getting rid of them would yield next to nothing.

This could be implemented as an optional extension, as you would also need to know the empty-space color for the given map layer for this to work without artifacts. It would activate only provided that you know the empty space color and the database has this additional info.

It would indeed probably speed things up both in the current GTK interface - you would just draw a rectangle and fill it with solid color instead of loading an image to an image surface and then painting it. It would also make sure only real tiles are cached (the amount of cached tiles is currently limited by their count).

In the QML interface it would help too - just drawing a colored Rectangle instead of an Image element. But the performance is already very good, so it might not be that needed in this case.

To detect that a (downloaded) tile is empty, simple == comparison might be enough & pretty fast, as a different size is automatically False and comparing 103 Byte sized files should be very fast. Or just get the header, check size and don't download it in the first place, just mark as empty.

Originally Posted by jafd

As for sharing the pre-renders, I suspect it would take a few months to do them on commodity hardware (three days for Poland, zoom levels all the way to 15 and 16-18 for select cities); also, to achieve acceptable results, the database for the whole planet is needed (for some reason I've got proper administrative borders only around one of two countries).

Well, thats quite brutal. But I'd say doing just some selected european countries and/or big cities might be doable.

Proper on-device realtime rendering would be ideal though - I have to finally look how the Kothic renderer actually works. I met some people on the SotM in Viena suggesting that Mapnik is horrendously inefficient so it might indeed be possible.

Originally Posted by jafd

Also the stylesheet I ended up with, default one from OSM-bright (which played well with imposm data) is bad, too low contrast to be viewed on a phone. Font size is also next to pathetic.

What about the stylesheet used for the detail layer ?

At least this tools does custom rendering from OSM snapshots and uses them default stylesheet, so it should be possible:
http://osm.kyblsoft.cz/historie/
(the text in the header says that you need to select on the plus in the upper left corner to select the snapshot)

Quote & Reply |

MartinK	2012-03-14 , 22:16
Posts: 1,548 \| Thanked: 7,510 times \| Joined on Apr 2010 @ Czech Republic	#956

Originally Posted by abubakar

I installed modrana few weeks back and recorded 3 tracks. Then I used the "show on map" feature to plot them on the maps, all 3 of them. I dont know when after that this problem started that modrana is not downloading any tiles anywhere, I have tried zooming in/out a lot, change map location, restarted the device several times, the tiles just dont load anymore and all I see is the "Loading" in each tile. What can I do to provide the debugging info so this can be debugged. I was thinking I should uninstall and install, but I have not done that yet.

There are several thinks to try:

check general Internet connectivity - are sites in Microb loading, etc.
try another map layer, in Menu->Options->Map->Map Layers
turn off map overlay in Menu->Options->Map->Map overlay
if nothing of the above helps, try to delete/rename the file options.bin in /home/user/.modrana/

If even all of the above does not help, start modrana with:

Code:

modrana

And post/pastbin the output you get.

Quote & Reply |

The Following User Says Thank You to MartinK For This Useful Post:
abubakar

jafd	2012-03-14 , 23:50
Posts: 29 \| Thanked: 26 times \| Joined on Oct 2009	#957

Originally Posted by MartinK

Well, thats quite brutal. But I'd say doing just some selected european countries and/or big cities might be doable.

The countries are not a problem. But usually you don't want blank data right after the state border, that's the point.

I met some people on the SotM in Viena suggesting that Mapnik is horrendously inefficient so it might indeed be possible.

The way it works currently makes me think those people are right. It issues a bloody heavy SQL query to the GIS for every 256 by 256 tile. I suspect that rendering a 2048x2048 tile in one go and then cut it into 64 256x256 tiles (think multiprocessing, again) would be far more efficient, but trying this requires quite a bit of time I don't have at the moment.

I also have this idea of using a clustered setup. I also don't want my Twisted knowledge to rust. The company I'm working at announced an R&D lab for noncommercial, spinoff and hobby projects, so maybe someday I would talk them into building the most efficient OSM rendering farm in the world.

Seeing demise of t@h, OTOH, I'm a bit skeptical about this one.

What about the stylesheet used for the detail layer ?

I don't know a thing about layers. All tiles are single layer, with all details already burnt-in. The point is that in OSM default stylesheet has better contrast overall, and maps rendered with it have proper city/village/whatever boundaries rendered, which my maps don't.

At least this tools does custom rendering from OSM snapshots and uses them default stylesheet, so it should be possible:
http://osm.kyblsoft.cz/historie/
(the text in the header says that you need to select on the plus in the upper left corner to select the snapshot)

Let me reiterate a bit.

OSM works generally by building a geospatial database out of its input files, and schemas can be designed in different ways. The xml styles for maps are tuned to a particular schema, containing all the queries. The osm.xml shipped with Mapnik is tuned to their own database schema as produced by osm2pgsql tool, which either is dog slow or requires gobs of memory (8 GiB is way too small for Poland, let alone Europe).

imposm, on the other hand, handles data much more efficiently on your average desktop/laptop hardware, my humble entry-level Thinkpad would crunch all of Europe in around 4 hours which is fast. The problem is, tables, tags and so on as produced by this tool, are totally different from what osm2pgsql provides.

So, in fact, I had to google "imposm osm.xml style compatible" and so on, getting irrelevant things at the top, until I've found OSM-bright and Carto. Then, the stylesheet produced makes what you see on the screenshot.

The OSM XML is so big that I don't have the faintest idea what to tweak so it would work.

Or maybe I would leave osm2pgsql running for a weekend with the latest Europe data, set up a cron job to incorporate differences as they flow in (osm2pgsql data allow this, imposm don't, AFAIK), and then render particular countries (so they have consistent data at the borders and so that anyone would be able to merge tilesets as he wishes).

Quote & Reply |

Joseph9560	2012-03-15 , 05:39
Posts: 440 \| Thanked: 160 times \| Joined on Aug 2010 @ Las Vegas, NV	#958

Originally Posted by MartinK

Yep, Google geocoder is currently used for address search. But I'd like to add other geocoders in the near future - Nominatim, Geonames, etc. Weird that it works with Google Earth - either it uses its own different private API or there are some encoding issues with the ampersand in the string ?

Correction: The address search worked with presets online -> Custom query but not from address online.

I can see that Rana (on which modrana is based) actually uses vector data rather than tile data, any specific reason for modrana to use tile data while starting the port?

Last edited by Joseph9560; 2012-03-15 at 06:17.

Quote & Reply |

jafd	2012-03-15 , 07:28
Posts: 29 \| Thanked: 26 times \| Joined on Oct 2009	#959

Originally Posted by Joseph9560

Correction: The address search worked with presets online -> Custom query but not from address online.

I can see that Rana (on which modrana is based) actually uses vector data rather than tile data, any specific reason for modrana to use tile data while starting the port?

One reason I'm thinking about is that there are quite complex rules on which details at which sizes are to be rendered, based on a zoom level. Your average SVG files don't provide those rules, and even if they did, rendering would become so complex that your phone would die of overheating / battery drain very fast.

Another way is to use OSM data (maybe in PBF format) to do the same, but I'm wondering how much study one has to invest into it, and how efficient would it be in the end. Wouldn't it be that in addition to quite big OSM datafiles there would need to be an even bigger index to efficiently search for needed data fragments?

Tiles are simple, on the other hand, and straightforward. You don't need to muck around with projections all the time, resorting instead to simple Cartesian arithmetic.

Maybe when the phones grow another gigabyte of RAM and a few more processing cores, these reasonings will be outdated.

Quote & Reply |

jafd	2012-03-15 , 08:23
Posts: 29 \| Thanked: 26 times \| Joined on Oct 2009	#960

By some clever use of normalization and a few lines patch for ModRana I have been able to slim dataset for the whole of Poland with zoom level up to 15 and select cities up to 18, from 4792M to 4589M. That's the whole 203 megabytes less, with no performance impact.

I also wonder how much it would be squeezed further by using png16 instead of png256. It would, of course, require a new low-color stylesheet too... And some tweaks to renderer