maemo.org - Talk

maemo.org - Talk (https://talk.maemo.org/index.php)
-   Applications (https://talk.maemo.org/forumdisplay.php?f=41)
-   -   [SFOS] [Announce] Native offline maps: OSM Scout Server (https://talk.maemo.org/showthread.php?t=97823)

rinigus 2016-10-14 17:49

Native offline maps: OSM Scout Server
 
I would like to announce an offline maps solution: OSM Scout Server.

The server can be used as a drop-in replacement for online map services providing map tiles, search, and routing. As a result, together with the map client, a full offline solution is available for map search and navigation (car, bicycle, and walking).

The server supports:
License: GPL
Source code: https://github.com/rinigus/osmscout-server
User's guide: https://rinigus.github.io/osmscout-server
Packaging: OpenRepos https://openrepos.net/content/rinigus/osm-scout-server

Starting from 1.4.3, distribution via Jolla Store stopped until systemd linking is allowed (https://github.com/sailfishos/sdk-ha...tor/issues/102)

Translations: https://www.transifex.com/rinigus/osm-scout-server

At present, the server can be used to provide:
  • map tiles for other applications;
  • search for locations and free text search;
  • search for POIs next to a reference area;
  • calculate the route between two or more locations.

Starting from version 0.7, the server uses maps distributed via online distribution network. It is expected that the users would download, update, and, when needed, remove maps via server's GUI. This distribution model allows users to specify which components are required (libosmscout, geocoder-nlp, for example) and download only the required components.

To use the server, you have to start it and configure the client to access it. Recent versions of Poor Maps and modRana cover the full functionality of the server without any additional steps needed to be done by the user. In addition, an example configurations for Poor Maps and modRana are provided under "thirdparty" folder (see Github source tree).

The server can be used as Sailfish application or running in console on regular Linux. By default, it is configured to accept connections from localhost only, but you can change the configuration and serve it on LAN or even provide a globally accessible service, if you wish.

On Sailfish device, route calculations could take some time. As a reference, route calculation for shorter distances takes few seconds on a device (I use it on Nexus 4). For example, Stockholm -> Malmö (600 km) is about 15 seconds. Route calculation from Stockholm to Aarhus (950 km, via bridges) would take about 45 seconds. These are times that are visible for end user when using libosmscout backend (2016) - from starting route calculation request in Poor Maps till Poor Maps displayed the result.

For detailed up-to-date description and map import instructions, please see https://github.com/rinigus/osmscout-...ster/README.md

The server relies on the same library as an offline map viewing application OSM Scout by Lukáš Karas (@Karry). As the result, you could use the same maps when using the server or OSM Scout. However, at the current stage of development, the server is using the latest version of the upstream library without the compatibility with the older map formats, as done in OSM Scout. This may change in future, but at present the focus is on other aspects of the development.

Starting from 0.6.0 version, the server provides an additional geocoder that uses libpostal for normalization and parsing of the search requests.

When a new backend is added, such as with the release of 0.9.0, please ensure that you have subscription information updated. For that, in Map Manager, click on "Check for updates", decline updates, and unsubscribe/subscribe each territory that you have subscribed. After that, proceed with the downloads. This is not needed if you don't want to use the added backend(s).

Acknowledgments

The server is just a small part in large amount of work that have been done on mapping solutions by many other developers.

The underlying library was originally developed by Tim Teulings (@Framstag) who is the principal developer right now as well. Lukáš Karas (@Karry) has been working actively on the development of this library and Sailfish application OSM Scout. The server relies on their and other libosmscout developers code to provide the service.

On the other side, the server allows us to use excellent map applications developed over time for Sailfish and Linux. Already in the early development stage, I was helped by Poor Maps and modRana developers Osmo Salomaa (@otsaloma) and Martin Kolman (@M4rtinK), respectively. Thank you! While developing server, I could not cover all possible client applications and limited myself to these two. I am sure that if the server would become popular, the other plugins would be developed as well.

Changelog

Please see https://github.com/rinigus/osmscout-server/releases for changelog

otsaloma 2016-10-14 20:15

Re: [Announce] Native offline maps: OSM Scout Server
 
I expect to ship the provider files in Poor Maps, so that if you have OSM Scout Server installed, it'll be listed in Poor Maps along rest of the providers. It just needs a bit of testing, so that release is maybe a week or so away.

Quote:

Originally Posted by rinigus (Post 1516735)
I would like to ask the community developers to help with the aspects of this project. In particular, a vital part of using any offline maps solution is getting the maps on a device. At this stage, we would need an Import utility that would be compiled for different platforms (preferably statically linked, if possible) that users could download and use to convert OSM maps into libosmscout format. Maybe there is someone wishing to help with that?

Wouldn't it be easiest to just put the generated database files on a server for users to download? Bandwidth shouldn't be that expensive these days. It would mean writing conversion and upload scripts (preferrably in a separate repository) and having your app list what's on the server and download what the user chooses. Karry already has some kind of a download server.

Or are you looking to be able to import arbitrary smaller regions? Are full countries/states too large or slow?

rinigus 2016-10-14 20:56

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by otsaloma (Post 1516746)
I expect to ship the provider files in Poor Maps, so that if you have OSM Scout Server installed, it'll be listed in Poor Maps along rest of the providers. It just needs a bit of testing, so that release is maybe a week or so away.



Wouldn't it be easiest to just put the generated database files on a server for users to download? Bandwidth shouldn't be that expensive these days. It would mean writing conversion and upload scripts (preferrably in a separate repository) and having your app list what's on the server and download what the user chooses. Karry already has some kind of a download server.

Or are you looking to be able to import arbitrary smaller regions? Are full countries/states too large or slow?

Karry is working on it, indeed. He wrote some scripts that he uses to generate maps that are available on his server. Good news is that these maps are compatible with the server as well. From his server you could see the sizes of the files. Its a very important work that he does on it and I am looking forward to see where it would lead.

Unfortunately, I think that he used a bit older format of the map database, so right now you would still have to make the database yourself. Later, it would be possible to build the server using Karry's version of the library, so this problem would go away.

The second problem with pre-build maps is the cross-country routing. At present, libosmscout supports routing only within a map database. Whether the database covers 2, 3, or 1 countries does not matter. So, if you want to get from Sweden to Denmark, you would need a map covering the both countries. Since I'd like to get routing working as well, I see the need for letting user to decide the map region.

Rendering bigger maps are slower, sure. Fortunately, caching by Poor Maps and other applications would offset it a lot. There maybe parts in the library that would need to be optimized. But with the increased number of users, it should be easier to tackle the corner cases and fix them.

Your idea on getting lists of maps and downloading them is probably the way it is the easiest for the users and, as far as I know, Karry is planning to implement it for his application. I haven't thought about it too much yet and looked for bandwidth providers. My hope was that having a simple Import tool would at least postpone the need for a full-blown maps service. However, if someone wants to help Karry with the scripts and setting up the infrastructure, I am sure he will be very happy for all the help he can get!

rinigus

otsaloma 2016-10-15 23:37

Re: [Announce] Native offline maps: OSM Scout Server
 
I had a chance to try your server. My observations:
  • libosmscout is no masterpiece of cartographic rendering -- upsidedown labels, all kinds of things overlapping -- but I guess that was to be expected.
  • There's a huge difference in rendering time between using a Mapzen metro extract for the Helsinki region and a Geofabrik extract for the whole Finland. So, your desire for users to be able to have databases of arbitrary regions instead of country extracts makes sense from a speed standpoint as well. With a Mapzen metro extract, the rendering time is surprisingly fast -- it's very usable!
  • The large tile size seems fine. I doubt we need the smaller ones.
  • The search seems very particular about the query format. e.g. "Erottaja" works, but "Erottaja, Helsinki" doesn't, "Tukholmankatu" works, but "Tukholmankatu 3" doesn't.
  • The "admin_region" field in search results and thus the description field in Poor Maps is often empty. I can't conclude from the results when or why.
  • The Nearby search seems to work fine, expect for the missing region data, which is a known problem. The raw OSM data is really bad for this though -- it's incomplete, outdated and very particular about how exactly POIs are tagged.

rinigus 2016-10-16 08:15

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by otsaloma (Post 1516793)
I had a chance to try your server. My observations:
  • libosmscout is no masterpiece of cartographic rendering -- upsidedown labels, all kinds of things overlapping -- but I guess that was to be expected.
  • There's a huge difference in rendering time between using a Mapzen metro extract for the Helsinki region and a Geofabrik extract for the whole Finland. So, your desire for users to be able to have databases of arbitrary regions instead of country extracts makes sense from a speed standpoint as well. With a Mapzen metro extract, the rendering time is surprisingly fast -- it's very usable!
  • The large tile size seems fine. I doubt we need the smaller ones.
  • The search seems very particular about the query format. e.g. "Erottaja" works, but "Erottaja, Helsinki" doesn't, "Tukholmankatu" works, but "Tukholmankatu 3" doesn't.
  • The "admin_region" field in search results and thus the description field in Poor Maps is often empty. I can't conclude from the results when or why.
  • The Nearby search seems to work fine, expect for the missing region data, which is a known problem. The raw OSM data is really bad for this though -- it's incomplete, outdated and very particular about how exactly POIs are tagged.

  • libosmscout as a masterpiece: To my understanding, its constant optimization of speed vs quality. Label rendering is considered as an expensive part in tile rendering and there might be issues with it. If you have upside-down labels then note that this is unexpected and we should raise an issue in libosmscout developers list or github. An example tile would help (if you could provide server URL that would be of help too).
  • difference in rendering times: Sometimes, there are corner cases that could make rendering slower than it should. For example, in Estonia, the rendering was majorly slowed down due to the handling of ferry lines with the ferry between Rostock and Helsinki increasing rendering time by order of magnitudes. This case was solved by @Karry. I don't know if you are hitting something similar. How long do you have to wait on the phone for a 1024x1024 tile to be rendered? There are also some messages printed in stdout by the library that are not displayed in Events of the server. I have converted Finland map and can check out on the phone if you would send me the zoom level and rough coordinates. Then we can see if there is something peculiar with Finland. But true, rendering is far from instant. That's were caching is helping a lot.
  • Tile size: yes, large are probably the way to go.
  • Search of Erottaja and admin_region: see below
  • POI: that's maybe can be improved if I'll manage to find a way to get admin_regions for that. But yes, OSM data limitations are inherited by the server as well.

The search is implemented in two parts: search for location and search for any free text in the map. While the first is case insensitive, the free text search is case-sensitive.

The search results can be distinguished easily right now by presence (location) or absence (free text) of the admin_region field in the reply.

For location search, you have to add frequently an area where you search. When you search for Erottaja, you'll get the first response from the location search with

"admin_region": "Erottaja, Iitti, Päijät-Häme, Etelä-Suomi, Etelä-Suomi, Suomi"

The replies from the free text search find all objects that have the search string in object title. So, if you were searching for a bus stop in Helsinki then you get it as a part of free text search that can find an object with the name Erottaja, but its not recorded as "Erottaja, Helsinki" in that database. I don't know why the bus stop is not available in location search, maybe we should ask in libosmscout list.

As for Tukholmankatu, again all can be explained and the search can be guided. When you just use Tukholmankatu, location search is not finding anything since you haven't specified the area where you search. So, you get replies from the free text search. In free text, you have an object with this name which is a road. All buildings in that road are not a part of the database, unless they enter it as POI. To get an address, just add an area. So, the searches

Tukholmankatu 3, Helsinki

or even Tukholmankatu 3, Suomi would probably give what you want.

I am not sure how to improve this - whether to educate us as users or try to fix it in the library or server.

kaari 2016-10-16 12:22

Re: [Announce] Native offline maps: OSM Scout Server
 
i have problem installing server. libosmscout depends on libprotobuf which i have newer version. could it be possible to update to newer version, or i just back down. tried to make symlink, didnt work.

rinigus 2016-10-16 12:58

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by kaari (Post 1516809)
i have problem installing server. libosmscout depends on libprotobuf which i have newer version. could it be possible to update to newer version, or i just back down. tried to make symlink, didnt work.

You know what, wait a little bit. I have a suspicion that protobuf is not needed. As far as I know, its needed for import and I cannot see it in ldd of the server binary, which is a good sign. I'll check it out and will report back on it.

otsaloma 2016-10-16 14:40

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by rinigus (Post 1516804)
If you have upside-down labels then note that this is unexpected and we should raise an issue in libosmscout developers list or github.

I doubt it's unexpected, since I see it all over the place: regular roads, forks, labels wrapping around a U-turn, etc. Couple examples below, center of the screen.

https://postimg.org/gallery/d4dc0wbm/

Quote:

Originally Posted by rinigus (Post 1516804)
difference in rendering times:

I'd say those Mapzen metro extract tiles render in about half the time of tiles from whole Finland data. And those tiles from whole Finland data are about what I'd expect, so it was not to report a problem, but a pleasant surprise when using a Mapzen metro extract. I don't have much use for offline maps myself, so I'll leave any possible debugging and timing for others.

Quote:

Originally Posted by rinigus (Post 1516804)
I am not sure how to improve this [search] - whether to educate us as users or try to fix it in the library or server.

I understand that it's a very difficult problem to parse and match search queries when people give them in many different formats and sometimes you might have partial matches, e.g. street found, but not the the full address, etc. But I also think that search should work with the same format queries as with online providers. Especially in an app like Poor Maps, which supports several providers and has a common history. No-one is going to read a long manual -- if normal search queries don't work, users will just conlude your software is broken.

Since it's a big problem, I wonder if there's a suitable external library you could use instead of trying to solve the problem on your own? For example, Mapzen has recently released Libpostal -- "a fast, multilingual, international street address parser trained on OpenStreetMap data".

https://mapzen.com/blog/inside-libpostal/

rinigus 2016-10-16 15:44

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by kaari (Post 1516809)
i have problem installing server. libosmscout depends on libprotobuf which i have newer version. could it be possible to update to newer version, or i just back down. tried to make symlink, didnt work.

@kaari, please try again. I have changed RPM packaging script and dropped import library from the package. As a result, protobuf dependency is not needed. The new package for the library is uploaded to OpenRepos.

Please report back if it breaks anything. For those, who want to clean their devices:

* uninstall protobuf: pkcon remove protobuf . This should prompt for removal of the server and the libosmscout-qt.

* install the server again.

Everything is working on my device, but please report back and tell if anything is OK after this change. Sorry for this extra dependency, I seem to have left it by mistake.

rinigus 2016-10-16 21:14

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by otsaloma (Post 1516818)
I doubt it's unexpected, since I see it all over the place: regular roads, forks, labels wrapping around a U-turn, etc. Couple examples below, center of the screen.

https://postimg.org/gallery/d4dc0wbm/

In general, it looks similar to what I see. However, I haven't spotted upside-down labels as you did yet.


Quote:

Originally Posted by otsaloma (Post 1516818)
I understand that it's a very difficult problem to parse and match search queries when people give them in many different formats and sometimes you might have partial matches, e.g. street found, but not the the full address, etc. But I also think that search should work with the same format queries as with online providers. Especially in an app like Poor Maps, which supports several providers and has a common history. No-one is going to read a long manual -- if normal search queries don't work, users will just conlude your software is broken.

Since it's a big problem, I wonder if there's a suitable external library you could use instead of trying to solve the problem on your own? For example, Mapzen has recently released Libpostal -- "a fast, multilingual, international street address parser trained on OpenStreetMap data".

https://mapzen.com/blog/inside-libpostal/

libpostal seems to be an interesting option. It looks that this is a preprocessor for the database query (see https://github.com/openvenues/libpos...not-a-geocoder ). Its an interesting option and certainly has to be tested. However, they do state that use on a mobile is "conceivable" and that its not designed with that in mind. I'll add libpostal to the todo (issues in github) and would take a look on it in future.

In case of Erottaja, Helsinki, libpostal would not help, I think. I'll have to check what is included into the search database as location and what is not. Hopefully, we can fix it.

kaari 2016-10-17 08:22

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by rinigus (Post 1516826)
Everything is working on my device, but please report back and tell if anything is OK after this change. Sorry for this extra dependency, I seem to have left it by mistake.

Thanks, working fine now.

rinigus 2016-10-25 16:28

Re: Version 0.2.0 - route calculations
 
I have just published a new release of the server. This version introduces the support for route calculations. Thus, together with a client, you could use it for offline navigation by car, bicycle, or while walking. The changelog will be added to the first post when it will be updated.

As a reference, route calculation for shorter distances takes few seconds on a device (I use it on Nexus 4). For example, Stockholm -> Malmö (600 km) is about 15 seconds. Route calculation from Stockholm to Aarhus (950 km) would take about 45 seconds.

Since it can take significant amount of time, the client program has to be a bit more patient with timeouts. See https://github.com/otsaloma/poor-maps/issues/23 for a reference and how to patch Poor Maps to work around it.

There are also changes that allow you to tweak label rendering (thanks to @Karry!). These are under advanced settings. Use caution while experimenting with them.

rinigus 2016-11-02 08:24

Re: [Announce] Native offline maps: OSM Scout Server
 
With the latest release of Poor Maps, there is no need anymore to add plugins from the thirdparty folder into your .local/shared/harbour-poor-maps folder. To avoid confusion, its probably better to use official Poor Maps plugins to interact with the server.

After exposing routing functionality of the libosmscout library, there are more issues with the stability of the server. For example, @otsaloma has submitted the cases where either the server crashed https://github.com/rinigus/osmscout-server/issues/31 or was not able to find the route https://github.com/rinigus/osmscout-server/issues/32 .

It seems to me that there were not that many users of routing functionality of the library so far and we are one of the first ones. This is, in general, not a problem as long as we help to fix the bugs and make the library better. The principal developer of the library - Tim Teulings (@Framstag) - is very active and is resolving the issues rather quickly. For example, during a week he was able to: [1] fix the issue related to suggesting illegal turns and the routes were only public transport can go; [2] propose a fix to an issue leading to segfault of the server in a case reported by @otsaloma. These fixes would be released with the next version of the server.

So, I would like to ask you to file the issues that you have. Please try to get an URL that leads to a problem. I will try to react in timely manner and either fix them myself or report upstream in a way that would allow Tim to reproduce and fix it.

In addition, please keep the library installed on your PCs. Some fixes, such as the ones related to taking illegal turns, would require regeneration of the maps. So, you would have to get a newer version of the library (git pull), compile and install it again, and then reimport the maps. One of the tasks that I am planning to resolve is related to simplifying this cycle by providing an Import tool, but such cycle is needed until its done.

We are not yet in the stage where everything is working as it should. But, with all of our help, the library and the server could be improved significantly.

The current issues and my todo list are shown under https://github.com/rinigus/osmscout-server/issues

KylliOrvokki 2016-11-02 13:49

Re: [Announce] Native offline maps: OSM Scout Server
 
1 Attachment(s)
I am getting following error when trying to install from Warehouse.
Any ideas/help what I am doing wrong?

Jolla-C and 2.0.4.14

rinigus 2016-11-02 14:08

Re: [Announce] Native offline maps: OSM Scout Server
 
You are the second user who has this problem - I have received earlier report via OpenRepos.

I have just tested it on my Nexus 4 and I can install the server without a problem. This was tested on an SFOS image that I built earlier today for 2.0.4.13 (for ported devices we have a scratchbox that corresponds to .13 version). I have libgomp-4.8.3-1.2.5 installed as a dependency. [I don't own any official device and I cannot test it on jolla].

I suspect that your repos need to be refreshed (maybe Jolla refreshed gomp recently?). Try to run from command line:

pkcon refresh
pkcon search name libgomp

and see if the library is available.

XOleg 2016-11-02 14:20

Re: [Announce] Native offline maps: OSM Scout Server
 
I installed without problems from Warehouse. It's not easy to import maps for this moment :-)

JollaC(exAquaFish), v.2.0.4.14.

nodevel 2016-11-02 16:33

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by KylliOrvokki (Post 1517903)
I am getting following error when trying to install from Warehouse.
Any ideas/help what I am doing wrong?

Jolla-C and 2.0.4.14

I had the same problem. I believe you need to refresh all repos first ('pkcon ref' should do) - at least I think that solved it for me.

KylliOrvokki 2016-11-02 20:14

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by rinigus (Post 1517904)

pkcon refresh
pkcon search name libgomp

Thanks!

Refresh helped and after that i could see libgomp and able to install OSM Scout Server from Warehouse!

otsaloma 2016-11-02 20:31

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by rinigus (Post 1517891)
So, I would like to ask you to file the issues that you have. Please try to get an URL that leads to a problem. I will try to react in timely manner and either fix them myself or report upstream in a way that would allow Tim to reproduce and fix it.

Just to add to this, if you're using Poor Maps, the way to get the URL, is to run Poor Maps via the terminal with command

Code:

sailfish-qml harbour-poor-maps
Among the output will be something like

Code:

Requesting http://localhost:8553/v1/route?type=car&p[0][search]=Helsinki&p[1][lng]=24.811749&p[1][lat]=60.218380
and similar for search and nearby venues. That's the URL to look for. You can also save the output to file by using command

Code:

sailfish-qml harbour-poor-maps > poor-maps.log 2>&1

rinigus 2016-11-12 19:52

Version 0.3.0
 
I have just published a new release: 0.3.0.

0.3.0 release is built against libosmscout release https://github.com/rinigus/libosmsco...0.git.20161110 . I am going to tag libosmscout versions, so you could always build corresponding Import tool.

For end users, the main change is in fixing multiple bugs related to routing in the libosmscout library by its authors: segfault at certain conditions reported by @otsaloma (thank you!), taking the wrong turns, and few others.

There are several important changes "under the hood". I am using now GNU Libmicrohttpd to provide HTTP backend. As a result, the clients can use up to 100 connections served using as many threads as there are CPUs on your device. The limit of connections is hard-coded, but if there is a huge interest in specifying it as an option, I will add it. This change should simplify the client's side of communication since it would not have to worry about exceeding the limit.

The server is now built using static linking to libosmscout, marisa, and libmicrohttpd. Its mainly done for two reasons: libosmscout API is changing as it is developed leading to incompatibilities between the versions; there is no need for other RPMs as you install the server. After an upgrade, I suggest to remove libosmscout-qt and libmarisa from your devices. I am planning to drop these packages from OpenRepos a bit later when users would move to 0.3.0 or later release.

NB! When you start using a new version, please update the maps! Few bugs were fixed in libosmscout library by fixing an importer. So, to get these fixes, you would have to update the maps with the latest version.

I have added a session log (see Settings to enable it). You could use it when reporting bugs to get the corresponding URL.

If your favorite client does not support routing yet, but supports GPX, you could now calculate the route and save the result in GPX format (see URL parameters in server GitHub project page).

Feathers McGraw 2016-11-12 20:36

Re: [Announce] Native offline maps: OSM Scout Server
 
Guys, I just wanted to say thank you for working on this. Looking forward to testing it when there are prebuilt maps or an import tool!

ferlanero 2016-11-13 11:16

Re: [Announce] Native offline maps: OSM Scout Server
 
Hi guys!

Thank you very much for this piece of gold for our Sailfish OS!

I'm trying to making OSM Scout Server working on a Jolla 1, but after installing correctly OSM Scout Server, import and convert maps correctly with libosmscout and copying it to

Code:

/home/nemo/Domcuments/Maps
After opening OSM Scout Server I only can see this error on main screen:

https://3.bp.blogspot.com/-YZAX6D-lp...161113_001.png

I have to say that with same imported maps, and in same folder, OSM Scout aplicattion from Warehouse https://openrepos.net/content/karry/osm-scout everything is working as spectated:

https://1.bp.blogspot.com/-NChJQX8HX...161113_002.png

What is what I'm doing wrong?

For importing maps from http://www.geofabrik.de/ I'm doing:

./build.sh nordrhein-westfalen.osm.pbf for import with libscout

And copying the resulted files (which are a lot) into /home/nemo/Documents/Maps

Can you help me please?

Have you got any converted-working map with OSM Scout Server to compare the results and seeing what I'm doing wrong?

rinigus 2016-11-13 12:26

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by ferlanero (Post 1518628)
What is what I'm doing wrong?

For importing maps from http://www.geofabrik.de/ I'm doing:

./build.sh nordrhein-westfalen.osm.pbf for import with libscout

And copying the resulted files (which are a lot) into /home/nemo/Documents/Maps

Can you help me please?

Have you got any converted-working map with OSM Scout Server to compare the results and seeing what I'm doing wrong?

You are very-very close. Just few touches and you should be fine.

I suspect that your problems are just due to version mismatch. The libosmscout is developing very fast these days. In particular, the server together with Poor Maps exposed routing component and there are quite a few bugs fixed over the last weeks. As a result, the database format is changing as well. So:

1. Import the maps using the libosmscout version against which the server has been built: https://github.com/rinigus/libosmsco...0.git.20161110

2. When importing, I suggest to use the command line as in https://github.com/rinigus/osmscout-server#maps . This would delete the temporary and debug files from the maps folder. To my knowledge, the build script does not do that and you could end up with 2x larger import. Just adjust the path of the typefile and your map folder/pbf. Expect to get the similar size for imported map as you had PBF.

3. Keep the maps that you use for the server and OSM Scout separate. This is to avoid the version mismatch. We will converge eventually, but just do it for now while the development is rapid. Since you can specify the path for your maps in the server easily, I suggest to put the maps used by server into some other folder. On my phone, I use /home/nemo/Maps and subfolders with countries. So, to get Estonian maps, I use /home/nemo/Maps/estonia in my settings.

As you noticed, the maps are converted into multiple smaller files. That is correct and you would need them.

Please let me know if that helped. If not, we'll look further into it.

rinigus 2016-11-14 12:56

Re: [Announce] Native offline maps: OSM Scout Server
 
Here is my first take on simplified import. I have compiled Import statically for Linux x86_64 and wrote a small script to assist while importing. This version supports ONLY PBF-formatted OpenStreetMap maps. You can get maps from

http://download.geofabrik.de/

download the importer from

/link removed, see the next post/

uncompress the import tool and see included README.

This is a first try and I hope it works well. Please report back on how did it work and if there are any errors. I plan to drop it from dropbox in future and add the importer binaries to the releases at github. That way you will be able to get the importer corresponding to the release that you use.

Import tool was built using https://github.com/rinigus/libosmsco...uild_import.sh and "import" branch of libosmscout.

At present, only Linux on x86_64 architecture is provided. I don't know how to make similar build for Windows or Mac. If someone could chip in that would be of great help. Alternatively, Windows and Mac users could setup a virtual machine (VirtualBox, for example) with Linux and use it to import maps.

rinigus 2016-11-25 10:20

Re: [Announce] Preparing for release of Import tool - help needed
 
I have been working on making import of the maps simpler. The state is a bit work in progress, but it should be close to full release.

Right now, on each release in GitHub, the Import tool is built automatically using Travis and AppVeyor. As a result, we can get compiled import tools for Linux and Windows. While Linux version includes a small script that simplifies import to command "import mapfilename_in_pbf" , Windows build requires a bit more sophisticated input.

So, I would like to ask users interested in offline maps:

1. Linux users: Please test whether import works for you. Please report even if its just OK and include your distro. Would like to get conformation that it works in general, not just on PCs available to me.

2. Windows users: Please test whether you could run Import.exe by double clicking on it. No import will run, just command line help should be shown. Again please give a feedback, even a short one. If you wish to use it for map import as it is then I strongly suggest to use the similar options as in https://github.com/rinigus/libosmsco...inux/import.sh . Otherwise, you will end up with lots of debug info and get very large map directories that you have to store on device.

3. (quick note - I am getting help with this script via libosmscout developers mailing list) Someone with Windows scripting skills (minimal should already be better than mine), please write a script that would confirm to the spec below. If you don't know how to make it in full, please start discussion and write some parts that you can. Maybe then Windows users could finalize this small script to make map import easy.

The compiled packages are available at

https://github.com/rinigus/libosmsco...0.git.20161118

Let's use this for current testing. As soon as we get Windows scripts ready, we could make a new server release with the corresponding Import tools.

What I expect the Windows script should do:

Aim: finalize a compiled Windows package for simple importing maps into libosmscout

* The compiled binary of Import can be grabbed at https://github.com/rinigus/libosmsco...0.git.20161118 . The new script will be a part of this package and would allow us to distribute importers to the users.

* The script could be similar to https://github.com/rinigus/libosmsco...inux/import.sh

* While in Linux, we are used to start commands from terminal, in Windows its more common to drag-and-drop. So, on dragging downloaded OSM or OSM.PBF file, the new script should:
  • open a terminal (or PowerShell). I suspect its done automatically in Windows, but I don't know for sure
  • make a directory with the simplified name based on the dragged map file name. I used to remove -latest.osm.pbf from the map fine name and make directory just on the basis of remaining name. See Linux script for details
  • start Import.exe from relative bin\Import.exe . The style map.ost is under stylesheets directory. See compiled binary for package structure. Its expected that users just unpack it and can use importer straight away (no install is needed).
  • keep terminal open after Import.exe is finished, so the user could examine output for errors
  • on any error (cannot make export directory), stop the script (set -e in bash).

Since I am not familiar with Windows scripting, I am just afraid that it would take too long time for me to make it. So, please help even if its really trivial for you.

ferlanero 2016-11-25 11:09

Re: [Announce] Preparing for release of Import tool - help needed
 
Quote:

Originally Posted by rinigus (Post 1519356)
I have been working on making import of the maps simpler. The state is a bit work in progress, but it should be close to full release.

Right now, on each release in GitHub, the Import tool is built automatically using Travis and AppVeyor. As a result, we can get compiled import tools for Linux and Windows. While Linux version includes a small script that simplifies import to command "import mapfilename_in_pbf" , Windows build requires a bit more sophisticated input.

So, I would like to ask users interested in offline maps:

1. Linux users: Please test whether import works for you. Please report even if its just OK and include your distro. Would like to get conformation that it works in general, not just on PCs available to me.

2. Windows users: Please test whether you could run Import.exe by double clicking on it. No import will run, just command line help should be shown. Again please give a feedback, even a short one. If you wish to use it for map import as it is then I strongly suggest to use the similar options as in https://github.com/rinigus/libosmsco...inux/import.sh . Otherwise, you will end up with lots of debug info and get very large map directories that you have to store on device.

3. Someone with Windows scripting skills (minimal should already be better than mine), please write a script that would confirm to the spec below. If you don't know how to make it in full, please start discussion and write some parts that you can. Maybe then Windows users could finalize this small script to make map import easy.

The compiled packages are available at

https://github.com/rinigus/libosmsco...0.git.20161118

Let's use this for current testing. As soon as we get Windows scripts ready, we could make a new server release with the corresponding Import tools.

What I expect the Windows script should do:

Aim: finalize a compiled Windows package for simple importing maps into libosmscout

* The compiled binary of Import can be grabbed at https://github.com/rinigus/libosmsco...0.git.20161118 . The new script will be a part of this package and would allow us to distribute importers to the users.

* The script could be similar to https://github.com/rinigus/libosmsco...inux/import.sh

* While in Linux, we are used to start commands from terminal, in Windows its more common to drag-and-drop. So, on dragging downloaded OSM or OSM.PBF file, the new script should:

* open a terminal (or PowerShell). I suspect its done automatically in Windows, but I don't know for sure

* make a directory with the simplified name based on the dragged map file name. I used to remove -latest.osm.pbf from the map fine name and make directory just on the basis of remaining name. See Linux script for details

* start Import.exe from relative bin\Import.exe . The style map.ost is under stylesheets directory. See compiled binary for package structure. Its expected that users just unpack it and can use importer straight away (no install is needed).

* keep terminal open after Import.exe is finished, so the user could examine output for errors

* on any error (cannot make export directory), stop the script (set -e in bash).

Since I am not familiar with Windows scripting, I am just afraid that it would take too long time for me to make it. So, please help even if its really trivial for you.

Wow! In Linux (Ubuntu 16.04) works really great!! :D It's a really improvement to handle offline maps. Thank you very much!!

Now I have a question: OSMScout will be handle offline maps from sdcard? I love this app and how it works! So I love to see its evolution.

Thank ypou very much for all your efforts!!

rinigus 2016-11-25 11:26

Re: [Announce] Preparing for release of Import tool - help needed
 
OSM Scout Server should handle maps from SD. You just have to specify the map folder which is on SD card (use Linux path). Its under Settings, in OSM Scout Server GUI.

ferlanero 2016-11-25 13:28

Re: [Announce] Preparing for release of Import tool - help needed
 
Quote:

Originally Posted by rinigus (Post 1519363)
OSM Scout Server should handle maps from SD. You just have to specify the map folder which is on SD card (use Linux path). Its under Settings, in OSM Scout Server GUI.

You are right. Everything is working great! :)

Thanks

rinigus 2016-11-28 21:12

Re: 0.4.0
 
The new release is out: 0.4.0, followed by 0.4.1

A highlight of this release is introduction of pre-compiled map importer that can be used to convert OSM maps to the libosmscout format using Linux or Windows PC. See https://github.com/rinigus/osmscout-server#maps for instructions. While in Linux all should be rather simple, in Windows, an additional script simplifying the import is needed. I've got some help on libosmscout mailing list and, hopefully, we could make such a script soon.

I have removed OpenMP requirement of the server which should reduce the problems with installing it. Please let me know if the resulting performance of the server is degraded. I have not noticed any slowdown.

I would like to thank Carmen F. B. @carmenfdezb for Spanish translation of GUI interface. Its a great initiative!

Several bugfixes were incorporated in this release. In particular, a problematic routing case found by @otsaloma has been fixed in the upstream library.

0.4.1: Updated Spanish translation allowing you to get routing instructions in Spanish. Work of Carmen F. B. @carmenfdezb, thank you!

Feathers McGraw 2016-11-29 10:04

Re: [Announce] Native offline maps: OSM Scout Server
 
I've been reading the documentation in preparation for giving this a go, it's all really clear and well presented! Thank you to whoever wrote it.

Documentation often gets overlooked in projects like this where development is fast, I believe having decent docs will really help build up a userbase quickly. Well done :)

Feathers McGraw 2016-11-29 23:37

Re: [Announce] Native offline maps: OSM Scout Server
 
Successfully processed a map of the British Isles with the Linux importer (https://github.com/rinigus/libosmsco...git.20161118.1).

Processing time was under an hour with an i5 Haswell, which I think is pretty damn good!

Loaded it onto an SD card, and it is all working great (testing with Poor Maps). I think I could probably play around with the OSMSS settings to get the maps to load a bit faster, because the FP2 has 2GB RAM (looking at the settings, I think they're probably designed to be suitable for Jolla 1 too i.e. 1GB RAM?).

The first route I asked it to calculate was from Guildford to the Isle of Skye, which worked(!) - for those of you who don't live here, that's about 1000km! I was actually quite surprised that the route calculation was successful because OsmAnd had trouble with the length of the route, or the bridge to Skye (maybe it was a map split in OsmAnd?), or both. Not only that, OSM Scout Server was quicker than OsmAnd taking just a couple of minutes. to calculate. So basically, you've beaten the best offline map & routing app for Android already...

I have one remaining question, and a two requests...

In the stylesheets folder, there's a file called map.ost. I haven't copied it to the phone... should I, or is it just used for importing?

Would it be possible to add a folder selection for choosing the folder containing maps? Typing out the UUID of an SD card is pretty tedious!

Can the routing speeds input boxes be converted to mph using a settings toggle in the app to save the hassle of converting them?

Thanks again guys, I'm seriously impressed!

karlos devel 2016-11-30 01:00

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by Feathers McGraw (Post 1519643)
Successfully processed a map of the British Isles with the Linux importer (https://github.com/rinigus/libosmsco...git.20161118.1).

Processing time was under an hour with an i5 Haswell, which I think is pretty damn good!

Loaded it onto an SD card, and it is all working great (testing with Poor Maps). I think I could probably play around with the OSMSS settings to get the maps to load a bit faster, because the FP2 has 2GB RAM (looking at the settings, I think they're probably designed to be suitable for Jolla 1 too i.e. 1GB RAM?).

The first route I asked it to calculate was from Guildford to the Isle of Skye, which worked(!) - for those of you who don't live here, that's about 1000km! I was actually quite surprised that the route calculation was successful because OsmAnd had trouble with the length of the route, or the bridge to Skye (maybe it was a map split in OsmAnd?), or both. Not only that, OSM Scout Server was quicker than OsmAnd taking just a couple of minutes. to calculate. So basically, you've beaten the best offline map & routing app for Android already...

I have one remaining question, and a two requests...

In the stylesheets folder, there's a file called map.ost. I haven't copied it to the phone... should I, or is it just used for importing?

Would it be possible to add a folder selection for choosing the folder containing maps? Typing out the UUID of an SD card is pretty tedious!

Can the routing speeds input boxes be converted to mph using a settings toggle in the app to save the hassle of converting them?

Thanks again guys, I'm seriously impressed!


- I TRY TO IMPORT USING THE STATIC PROGRAM https://github.com/rinigus/libosmsco...git.20161118.1 BUT I GET A ERROR> CAN U SHOW UR EXAMPLE to IMPORT?--------------------------------debian-9 OS

MartinK 2016-11-30 01:19

Re: [Announce] Native offline maps: OSM Scout Server
 
The OSM Scout Server Day & Night layers have been added to the default modRana layer definitions:

http://modrana.org/images/osmscout_s..._day_layer.jpg

http://modrana.org/images/osmscout_s...ight_layer.jpg

I also plan to (finally!) do a new modRana release in a few days to make this functionality available to users. :)

Thanks a lot yet again to rinigus and all others who made this possible! :)

Now I just need to add all support for all the other nice features OSM Scout Server provides, such as search and routing. ;)

rinigus 2016-11-30 07:04

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by Feathers McGraw (Post 1519643)
Loaded it onto an SD card, and it is all working great (testing with Poor Maps). I think I could probably play around with the OSMSS settings to get the maps to load a bit faster, because the FP2 has 2GB RAM (looking at the settings, I think they're probably designed to be suitable for Jolla 1 too i.e. 1GB RAM?).

I used to test it on Nexus 4 (2GB RAM) and now I just moved to OnePlus X (3GB RAM). Since I don't have access to Jolla 1, I have no idea how well it works on that. I have followed the RAM usage of the server using SystemDataScope earlier and it was about 300MB RSS, if I recall correctly.

However, while RAM is handy, it would not have a major impact on map loading, I think (unless its cached by FS). You are probably bound by I/O from SD card, unfortunately.

Quote:

Originally Posted by Feathers McGraw (Post 1519643)
The first route I asked it to calculate was from Guildford to the Isle of Skye, which worked(!) - for those of you who don't live here, that's about 1000km! I was actually quite surprised that the route calculation was successful because OsmAnd had trouble with the length of the route, or the bridge to Skye (maybe it was a map split in OsmAnd?), or both. Not only that, OSM Scout Server was quicker than OsmAnd taking just a couple of minutes. to calculate. So basically, you've beaten the best offline map & routing app for Android already...

No, I don't think we have beaten OsmAnd. Maybe in routing speed, but there is still a long way ahead - keep patches and issues coming!

There are many ways of contributing in addition to coding. For example: filing issues, working on map drawing styles (standard.oss on your phone) and making them better suited for Poor Maps and modRana.

I am wondering, how well does it cope with the wrong-sided roundabouts? Any other problems? Never tested it on UK maps, I must say. Please file issues in GitHub if you see any.

Quote:

Originally Posted by Feathers McGraw (Post 1519643)
In the stylesheets folder, there's a file called map.ost. I haven't copied it to the phone... should I, or is it just used for importing?

Yes, map.ost is just used for importing. You could actually tweak it to import or exclude the features you want. Although, let us know if you changed it when you stumble on some bugs.

Quote:

Originally Posted by Feathers McGraw (Post 1519643)
Would it be possible to add a folder selection for choosing the folder containing maps? Typing out the UUID of an SD card is pretty tedious!

To take the maximal positive out from the current situation, you could even try to memorize UUID while you type it in :) .

Yes, I plan to add file/directory selection dialogue. Not sure it will be in the next release, but it will come. You could probably use symbolic links for now to simplify the path. Issue filed: https://github.com/rinigus/osmscout-server/issues/42

Quote:

Originally Posted by Feathers McGraw (Post 1519643)
Can the routing speeds input boxes be converted to mph using a settings toggle in the app to save the hassle of converting them?

Will deal with it: https://github.com/rinigus/osmscout-server/issues/41

Quote:

Originally Posted by Feathers McGraw (Post 1519643)
Thanks again guys, I'm seriously impressed!

Thank you!

rinigus 2016-11-30 07:12

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by karlos devel (Post 1519644)
- I TRY TO IMPORT USING THE STATIC PROGRAM https://github.com/rinigus/libosmsco...git.20161118.1 BUT I GET A ERROR> CAN U SHOW UR EXAMPLE to IMPORT?--------------------------------debian-9 OS

Hi! Let's try to fix it.

Its not fully static since there are major issues involved if you have threads. So, its almost static in this release, but not fully.

Please paste the command that you run and its output.

In addition, please paste output to see whether dynamic linking is working - from importer dir run:

ldd bin/Import

and to know a bit about your machine:

uname -a

rinigus 2016-11-30 07:14

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by MartinK (Post 1519645)
The OSM Scout Server Day & Night layers have been added to the default modRana layer definitions

Thank you!

rinigus 2016-12-02 09:08

Re: [Announce] Native offline maps: OSM Scout Server
 
Windows users - great news for you! Michael Nenninger developed a script that simplifies the import of the maps to dragging and dropping action.

Get the latest importer package from https://github.com/rinigus/libosmsco...git.20161128.2 , unzip and follow the README.

The same version has been generated for Linux as well. There are small improvements in import script in Linux allowing you to run it from any path.

The importer link has been updated in GitHub osmscout-server README.md as well.

I have tested the imported maps against 0.4.1 release of the server. However, if you encourage any problems, please let me know.

Thank Michael Nenninger for developing the import script and I hope that Windows PC owners could enjoy offline maps on their Sailfish devices as well!

Fellfrosch 2016-12-02 13:29

Re: [Announce] Native offline maps: OSM Scout Server
 
At First - thanks to you all for your great work!!!

One question:
I use OSM Scout Server with maps of Germany Austria and Switzerland. I recognized, that working with it isn't really fast, espacially when using Modrana (which often even refuses to load some map tiles). Using poormaps is way faster. Why is that?
And, can I speed it up in general. (Maybe by using a smaller map area?)

Feathers McGraw 2016-12-02 14:07

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by rinigus (Post 1519649)
I am wondering, how well does it cope with the wrong-sided roundabouts? Any other problems? Never tested it on UK maps, I must say. Please file issues in GitHub if you see any.

Wrong sided! Lol. I can't see any problems with the test routes I've generated, but I haven't actually used it for driving yet - will let you know if I spot anything.

rinigus 2016-12-02 14:59

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by Fellfrosch (Post 1519829)
At First - thanks to you all for your great work!!!

One question:
I use OSM Scout Server with maps of Germany Austria and Switzerland. I recognized, that working with it isn't really fast, espacially when using Modrana (which often even refuses to load some map tiles). Using poormaps is way faster. Why is that?
And, can I speed it up in general. (Maybe by using a smaller map area?)

I must say that I use modRana on PC and Poor Maps on the device so far. So, I haven't tested modRana too much on device.

Assuming that you use the latest version of the server (there could be different problems with the versions earlier than 0.3)

* you could be hitting timeouts for connections in modRana. As a result, modRana might ask multiple times for the same tiles. To check if its true, look into the events log shown in the server's main screen. Check whether the same URL is loaded multiple times.

* its probable that modRana and Poor Maps use different tile sizes. This is visible in URLs requested to render the tiles. Notice if there is a difference in "shift" and "scale" parameters in URL. Poor Maps uses by default 1024x1024 or 512x512 tiles that are not very well supported by modRana (or not at all yet) with modRana using 256x256 tiles (shift=0 and scale=1 by default). As a result, tile rendering can be done in bigger chunks in Poor Maps which should produce faster overall rendering. Optimization of tile size vs parallel rendering is not a simple problem and it may vary between the devices.

* maybe modRana's rendering is slower. If its true you could try to follow when the server states "Idle" on its cover and see when the map is rendered in reality. For that, browse the map in Poor Maps or modRana and peak by sliding from the right on the state of the server. In Poor Maps delay is minimal if any, in modRana - I haven't checked.

As for general speeding up - Smaller maps do help to render faster. Also, don't forget that Poor Maps and modRana make tile caches. So, when you open the same place next time, it should be faster.

The main drawback of smaller maps is that you would have to change them in the server's settings. Also, you cannot calculate the routes between maps. But if you know, for sure, that you are in some area and would be using only that for some time, its better to have a map of just that area. It should speed up things a bit.

We'll probably have to work on speed optimization at some point. At present, I am mainly working on getting the maps accessible for all, expose functionality better (GUI enhancement), and on bug fixes. There are few ideas already voiced in this thread that need to be checked better and I'll try to do it as much as I can.

rinigus

PS: meaning of parameters in URL is described at https://github.com/rinigus/osmscout-server#tiles for tile rendering


All times are GMT. The time now is 11:29.

vBulletin® Version 3.8.8