View Full Version : [SailfishOS] [Announce] Native offline maps: OSM Scout Server
I would like to announce an offline maps solution: OSM Scout Server.
The server can be used as a drop-in replacement for online map services providing map tiles, search, and routing. As a result, together with the map client, a full offline solution is available for map search and navigation (car, bicycle, and walking).
The server supports:
search via geocoder-nlp (https://github.com/rinigus/geocoder-nlp) that is based on libpostal (https://github.com/openvenues/libpostal);
map rendering via mapnik (https://github.com/mapnik/mapnik);
providing vector tiles for Mapbox GL-based applications;
routing instructions via Valhalla (https://github.com/valhalla/valhalla);
map rendering, search, and routing via libosmscout (http://libosmscout.sourceforge.net/).
License: GPL
Source code: https://github.com/rinigus/osmscout-server
User's guide: https://rinigus.github.io/osmscout-server
Packaging: OpenRepos https://openrepos.net/content/rinigus/osm-scout-server
Starting from 1.4.3, distribution via Jolla Store stopped until systemd linking is allowed (https://github.com/sailfishos/sdk-harbour-rpmvalidator/issues/102)
Translations: https://www.transifex.com/rinigus/osm-scout-server
At present, the server can be used to provide:
map tiles for other applications;
search for locations and free text search;
search for POIs next to a reference area;
calculate the route between two or more locations.
Starting from version 0.7, the server uses maps distributed via online distribution network. It is expected that the users would download, update, and, when needed, remove maps via server's GUI. This distribution model allows users to specify which components are required (libosmscout, geocoder-nlp, for example) and download only the required components.
To use the server, you have to start it and configure the client to access it. Recent versions of Poor Maps and modRana cover the full functionality of the server without any additional steps needed to be done by the user. In addition, an example configurations for Poor Maps and modRana are provided under "thirdparty" folder (see Github source tree).
The server can be used as Sailfish application or running in console on regular Linux. By default, it is configured to accept connections from localhost only, but you can change the configuration and serve it on LAN or even provide a globally accessible service, if you wish.
On Sailfish device, route calculations could take some time. As a reference, route calculation for shorter distances takes few seconds on a device (I use it on Nexus 4). For example, Stockholm -> Malmö (600 km) is about 15 seconds. Route calculation from Stockholm to Aarhus (950 km, via bridges) would take about 45 seconds. These are times that are visible for end user when using libosmscout backend (2016) - from starting route calculation request in Poor Maps till Poor Maps displayed the result.
For detailed up-to-date description and map import instructions, please see https://github.com/rinigus/osmscout-server/blob/master/README.md
The server relies on the same library as an offline map viewing application OSM Scout by Lukáš Karas (@Karry). As the result, you could use the same maps when using the server or OSM Scout. However, at the current stage of development, the server is using the latest version of the upstream library without the compatibility with the older map formats, as done in OSM Scout. This may change in future, but at present the focus is on other aspects of the development.
Starting from 0.6.0 version, the server provides an additional geocoder that uses libpostal for normalization and parsing of the search requests.
When a new backend is added, such as with the release of 0.9.0, please ensure that you have subscription information updated. For that, in Map Manager, click on "Check for updates", decline updates, and unsubscribe/subscribe each territory that you have subscribed. After that, proceed with the downloads. This is not needed if you don't want to use the added backend(s).
Acknowledgments
The server is just a small part in large amount of work that have been done on mapping solutions by many other developers.
The underlying library was originally developed by Tim Teulings (@Framstag) who is the principal developer right now as well. Lukáš Karas (@Karry) has been working actively on the development of this library and Sailfish application OSM Scout. The server relies on their and other libosmscout developers code to provide the service.
On the other side, the server allows us to use excellent map applications developed over time for Sailfish and Linux. Already in the early development stage, I was helped by Poor Maps and modRana developers Osmo Salomaa (@otsaloma) and Martin Kolman (@M4rtinK), respectively. Thank you! While developing server, I could not cover all possible client applications and limited myself to these two. I am sure that if the server would become popular, the other plugins would be developed as well.
Changelog
Please see https://github.com/rinigus/osmscout-server/releases for changelog
otsaloma
2016-10-14, 20:15
I expect to ship the provider files in Poor Maps, so that if you have OSM Scout Server installed, it'll be listed in Poor Maps along rest of the providers. It just needs a bit of testing, so that release is maybe a week or so away.
I would like to ask the community developers to help with the aspects of this project. In particular, a vital part of using any offline maps solution is getting the maps on a device. At this stage, we would need an Import utility that would be compiled for different platforms (preferably statically linked, if possible) that users could download and use to convert OSM maps into libosmscout format. Maybe there is someone wishing to help with that?
Wouldn't it be easiest to just put the generated database files on a server for users to download? Bandwidth shouldn't be (https://aws.amazon.com/s3/pricing/) that expensive these days. It would mean writing conversion and upload scripts (preferrably in a separate repository) and having your app list what's on the server and download what the user chooses. Karry already has some kind of a download server (https://osmscout.karry.cz/).
Or are you looking to be able to import arbitrary smaller regions? Are full countries/states too large or slow?
I expect to ship the provider files in Poor Maps, so that if you have OSM Scout Server installed, it'll be listed in Poor Maps along rest of the providers. It just needs a bit of testing, so that release is maybe a week or so away.
Wouldn't it be easiest to just put the generated database files on a server for users to download? Bandwidth shouldn't be (https://aws.amazon.com/s3/pricing/) that expensive these days. It would mean writing conversion and upload scripts (preferrably in a separate repository) and having your app list what's on the server and download what the user chooses. Karry already has some kind of a download server (https://osmscout.karry.cz/).
Or are you looking to be able to import arbitrary smaller regions? Are full countries/states too large or slow?
Karry is working on it, indeed. He wrote some scripts that he uses to generate maps that are available on his server. Good news is that these maps are compatible with the server as well. From his server you could see the sizes of the files. Its a very important work that he does on it and I am looking forward to see where it would lead.
Unfortunately, I think that he used a bit older format of the map database, so right now you would still have to make the database yourself. Later, it would be possible to build the server using Karry's version of the library, so this problem would go away.
The second problem with pre-build maps is the cross-country routing. At present, libosmscout supports routing only within a map database. Whether the database covers 2, 3, or 1 countries does not matter. So, if you want to get from Sweden to Denmark, you would need a map covering the both countries. Since I'd like to get routing working as well, I see the need for letting user to decide the map region.
Rendering bigger maps are slower, sure. Fortunately, caching by Poor Maps and other applications would offset it a lot. There maybe parts in the library that would need to be optimized. But with the increased number of users, it should be easier to tackle the corner cases and fix them.
Your idea on getting lists of maps and downloading them is probably the way it is the easiest for the users and, as far as I know, Karry is planning to implement it for his application. I haven't thought about it too much yet and looked for bandwidth providers. My hope was that having a simple Import tool would at least postpone the need for a full-blown maps service. However, if someone wants to help Karry with the scripts and setting up the infrastructure, I am sure he will be very happy for all the help he can get!
rinigus
otsaloma
2016-10-15, 23:37
I had a chance to try your server. My observations:
libosmscout is no masterpiece of cartographic rendering -- upsidedown labels, all kinds of things overlapping -- but I guess that was to be expected.
There's a huge difference in rendering time between using a Mapzen metro extract for the Helsinki region and a Geofabrik extract for the whole Finland. So, your desire for users to be able to have databases of arbitrary regions instead of country extracts makes sense from a speed standpoint as well. With a Mapzen metro extract, the rendering time is surprisingly fast -- it's very usable!
The large tile size seems fine. I doubt we need the smaller ones.
The search seems very particular about the query format. e.g. "Erottaja" works, but "Erottaja, Helsinki" doesn't, "Tukholmankatu" works, but "Tukholmankatu 3" doesn't.
The "admin_region" field in search results and thus the description field in Poor Maps is often empty. I can't conclude from the results when or why.
The Nearby search seems to work fine, expect for the missing region data, which is a known problem. The raw OSM data is really bad for this though -- it's incomplete, outdated and very particular about how exactly POIs are tagged.
I had a chance to try your server. My observations:
libosmscout is no masterpiece of cartographic rendering -- upsidedown labels, all kinds of things overlapping -- but I guess that was to be expected.
There's a huge difference in rendering time between using a Mapzen metro extract for the Helsinki region and a Geofabrik extract for the whole Finland. So, your desire for users to be able to have databases of arbitrary regions instead of country extracts makes sense from a speed standpoint as well. With a Mapzen metro extract, the rendering time is surprisingly fast -- it's very usable!
The large tile size seems fine. I doubt we need the smaller ones.
The search seems very particular about the query format. e.g. "Erottaja" works, but "Erottaja, Helsinki" doesn't, "Tukholmankatu" works, but "Tukholmankatu 3" doesn't.
The "admin_region" field in search results and thus the description field in Poor Maps is often empty. I can't conclude from the results when or why.
The Nearby search seems to work fine, expect for the missing region data, which is a known problem. The raw OSM data is really bad for this though -- it's incomplete, outdated and very particular about how exactly POIs are tagged.
libosmscout as a masterpiece: To my understanding, its constant optimization of speed vs quality. Label rendering is considered as an expensive part in tile rendering and there might be issues with it. If you have upside-down labels then note that this is unexpected and we should raise an issue in libosmscout developers list or github. An example tile would help (if you could provide server URL that would be of help too).
difference in rendering times: Sometimes, there are corner cases that could make rendering slower than it should. For example, in Estonia, the rendering was majorly slowed down due to the handling of ferry lines with the ferry between Rostock and Helsinki increasing rendering time by order of magnitudes. This case was solved by @Karry. I don't know if you are hitting something similar. How long do you have to wait on the phone for a 1024x1024 tile to be rendered? There are also some messages printed in stdout by the library that are not displayed in Events of the server. I have converted Finland map and can check out on the phone if you would send me the zoom level and rough coordinates. Then we can see if there is something peculiar with Finland. But true, rendering is far from instant. That's were caching is helping a lot.
Tile size: yes, large are probably the way to go.
Search of Erottaja and admin_region: see below
POI: that's maybe can be improved if I'll manage to find a way to get admin_regions for that. But yes, OSM data limitations are inherited by the server as well.
The search is implemented in two parts: search for location and search for any free text in the map. While the first is case insensitive, the free text search is case-sensitive.
The search results can be distinguished easily right now by presence (location) or absence (free text) of the admin_region field in the reply.
For location search, you have to add frequently an area where you search. When you search for Erottaja, you'll get the first response from the location search with
"admin_region": "Erottaja, Iitti, Päijät-Häme, Etelä-Suomi, Etelä-Suomi, Suomi"
The replies from the free text search find all objects that have the search string in object title. So, if you were searching for a bus stop in Helsinki then you get it as a part of free text search that can find an object with the name Erottaja, but its not recorded as "Erottaja, Helsinki" in that database. I don't know why the bus stop is not available in location search, maybe we should ask in libosmscout list.
As for Tukholmankatu, again all can be explained and the search can be guided. When you just use Tukholmankatu, location search is not finding anything since you haven't specified the area where you search. So, you get replies from the free text search. In free text, you have an object with this name which is a road. All buildings in that road are not a part of the database, unless they enter it as POI. To get an address, just add an area. So, the searches
Tukholmankatu 3, Helsinki
or even Tukholmankatu 3, Suomi would probably give what you want.
I am not sure how to improve this - whether to educate us as users or try to fix it in the library or server.
i have problem installing server. libosmscout depends on libprotobuf which i have newer version. could it be possible to update to newer version, or i just back down. tried to make symlink, didnt work.
i have problem installing server. libosmscout depends on libprotobuf which i have newer version. could it be possible to update to newer version, or i just back down. tried to make symlink, didnt work.
You know what, wait a little bit. I have a suspicion that protobuf is not needed. As far as I know, its needed for import and I cannot see it in ldd of the server binary, which is a good sign. I'll check it out and will report back on it.
otsaloma
2016-10-16, 14:40
If you have upside-down labels then note that this is unexpected and we should raise an issue in libosmscout developers list or github.
I doubt it's unexpected, since I see it all over the place: regular roads, forks, labels wrapping around a U-turn, etc. Couple examples below, center of the screen.
https://postimg.org/gallery/d4dc0wbm/
difference in rendering times:
I'd say those Mapzen metro extract tiles render in about half the time of tiles from whole Finland data. And those tiles from whole Finland data are about what I'd expect, so it was not to report a problem, but a pleasant surprise when using a Mapzen metro extract. I don't have much use for offline maps myself, so I'll leave any possible debugging and timing for others.
I am not sure how to improve this [search] - whether to educate us as users or try to fix it in the library or server.
I understand that it's a very difficult problem to parse and match search queries when people give them in many different formats and sometimes you might have partial matches, e.g. street found, but not the the full address, etc. But I also think that search should work with the same format queries as with online providers. Especially in an app like Poor Maps, which supports several providers and has a common history. No-one is going to read a long manual -- if normal search queries don't work, users will just conlude your software is broken.
Since it's a big problem, I wonder if there's a suitable external library you could use instead of trying to solve the problem on your own? For example, Mapzen has recently released Libpostal -- "a fast, multilingual, international street address parser trained on OpenStreetMap data".
https://mapzen.com/blog/inside-libpostal/
i have problem installing server. libosmscout depends on libprotobuf which i have newer version. could it be possible to update to newer version, or i just back down. tried to make symlink, didnt work.
@kaari, please try again. I have changed RPM packaging script and dropped import library from the package. As a result, protobuf dependency is not needed. The new package for the library is uploaded to OpenRepos.
Please report back if it breaks anything. For those, who want to clean their devices:
* uninstall protobuf: pkcon remove protobuf . This should prompt for removal of the server and the libosmscout-qt.
* install the server again.
Everything is working on my device, but please report back and tell if anything is OK after this change. Sorry for this extra dependency, I seem to have left it by mistake.
I doubt it's unexpected, since I see it all over the place: regular roads, forks, labels wrapping around a U-turn, etc. Couple examples below, center of the screen.
https://postimg.org/gallery/d4dc0wbm/
In general, it looks similar to what I see. However, I haven't spotted upside-down labels as you did yet.
I understand that it's a very difficult problem to parse and match search queries when people give them in many different formats and sometimes you might have partial matches, e.g. street found, but not the the full address, etc. But I also think that search should work with the same format queries as with online providers. Especially in an app like Poor Maps, which supports several providers and has a common history. No-one is going to read a long manual -- if normal search queries don't work, users will just conlude your software is broken.
Since it's a big problem, I wonder if there's a suitable external library you could use instead of trying to solve the problem on your own? For example, Mapzen has recently released Libpostal -- "a fast, multilingual, international street address parser trained on OpenStreetMap data".
https://mapzen.com/blog/inside-libpostal/
libpostal seems to be an interesting option. It looks that this is a preprocessor for the database query (see https://github.com/openvenues/libpostal#so-its-not-a-geocoder ). Its an interesting option and certainly has to be tested. However, they do state that use on a mobile is "conceivable" and that its not designed with that in mind. I'll add libpostal to the todo (issues in github) and would take a look on it in future.
In case of Erottaja, Helsinki, libpostal would not help, I think. I'll have to check what is included into the search database as location and what is not. Hopefully, we can fix it.
Everything is working on my device, but please report back and tell if anything is OK after this change. Sorry for this extra dependency, I seem to have left it by mistake.
Thanks, working fine now.
I have just published a new release of the server. This version introduces the support for route calculations. Thus, together with a client, you could use it for offline navigation by car, bicycle, or while walking. The changelog will be added to the first post when it will be updated.
As a reference, route calculation for shorter distances takes few seconds on a device (I use it on Nexus 4). For example, Stockholm -> Malmö (600 km) is about 15 seconds. Route calculation from Stockholm to Aarhus (950 km) would take about 45 seconds.
Since it can take significant amount of time, the client program has to be a bit more patient with timeouts. See https://github.com/otsaloma/poor-maps/issues/23 for a reference and how to patch Poor Maps to work around it.
There are also changes that allow you to tweak label rendering (thanks to @Karry!). These are under advanced settings. Use caution while experimenting with them.
With the latest release of Poor Maps, there is no need anymore to add plugins from the thirdparty folder into your .local/shared/harbour-poor-maps folder. To avoid confusion, its probably better to use official Poor Maps plugins to interact with the server.
After exposing routing functionality of the libosmscout library, there are more issues with the stability of the server. For example, @otsaloma has submitted the cases where either the server crashed https://github.com/rinigus/osmscout-server/issues/31 or was not able to find the route https://github.com/rinigus/osmscout-server/issues/32 .
It seems to me that there were not that many users of routing functionality of the library so far and we are one of the first ones. This is, in general, not a problem as long as we help to fix the bugs and make the library better. The principal developer of the library - Tim Teulings (@Framstag) - is very active and is resolving the issues rather quickly. For example, during a week he was able to: [1] fix the issue related to suggesting illegal turns and the routes were only public transport can go; [2] propose a fix to an issue leading to segfault of the server in a case reported by @otsaloma. These fixes would be released with the next version of the server.
So, I would like to ask you to file the issues that you have. Please try to get an URL that leads to a problem. I will try to react in timely manner and either fix them myself or report upstream in a way that would allow Tim to reproduce and fix it.
In addition, please keep the library installed on your PCs. Some fixes, such as the ones related to taking illegal turns, would require regeneration of the maps. So, you would have to get a newer version of the library (git pull), compile and install it again, and then reimport the maps. One of the tasks that I am planning to resolve is related to simplifying this cycle by providing an Import tool, but such cycle is needed until its done.
We are not yet in the stage where everything is working as it should. But, with all of our help, the library and the server could be improved significantly.
The current issues and my todo list are shown under https://github.com/rinigus/osmscout-server/issues
KylliOrvokki
2016-11-02, 13:49
I am getting following error when trying to install from Warehouse.
Any ideas/help what I am doing wrong?
Jolla-C and 2.0.4.14
You are the second user who has this problem - I have received earlier report via OpenRepos.
I have just tested it on my Nexus 4 and I can install the server without a problem. This was tested on an SFOS image that I built earlier today for 2.0.4.13 (for ported devices we have a scratchbox that corresponds to .13 version). I have libgomp-4.8.3-1.2.5 installed as a dependency. [I don't own any official device and I cannot test it on jolla].
I suspect that your repos need to be refreshed (maybe Jolla refreshed gomp recently?). Try to run from command line:
pkcon refresh
pkcon search name libgomp
and see if the library is available.
I installed without problems from Warehouse. It's not easy to import maps for this moment :-)
JollaC(exAquaFish), v.2.0.4.14.
I am getting following error when trying to install from Warehouse.
Any ideas/help what I am doing wrong?
Jolla-C and 2.0.4.14
I had the same problem. I believe you need to refresh all repos first ('pkcon ref' should do) - at least I think that solved it for me.
KylliOrvokki
2016-11-02, 20:14
pkcon refresh
pkcon search name libgomp
Thanks!
Refresh helped and after that i could see libgomp and able to install OSM Scout Server from Warehouse!
otsaloma
2016-11-02, 20:31
So, I would like to ask you to file the issues that you have. Please try to get an URL that leads to a problem. I will try to react in timely manner and either fix them myself or report upstream in a way that would allow Tim to reproduce and fix it.
Just to add to this, if you're using Poor Maps, the way to get the URL, is to run Poor Maps via the terminal with command
sailfish-qml harbour-poor-maps
Among the output will be something like
Requesting http://localhost:8553/v1/route?type=car&p[0][search]=Helsinki&p[1][lng]=24.811749&p[1][lat]=60.218380
and similar for search and nearby venues. That's the URL to look for. You can also save the output to file by using command
sailfish-qml harbour-poor-maps > poor-maps.log 2>&1
I have just published a new release: 0.3.0.
0.3.0 release is built against libosmscout release https://github.com/rinigus/libosmscout/releases/tag/0.0.git.20161110 . I am going to tag libosmscout versions, so you could always build corresponding Import tool.
For end users, the main change is in fixing multiple bugs related to routing in the libosmscout library by its authors: segfault at certain conditions reported by @otsaloma (thank you!), taking the wrong turns, and few others.
There are several important changes "under the hood". I am using now GNU Libmicrohttpd to provide HTTP backend. As a result, the clients can use up to 100 connections served using as many threads as there are CPUs on your device. The limit of connections is hard-coded, but if there is a huge interest in specifying it as an option, I will add it. This change should simplify the client's side of communication since it would not have to worry about exceeding the limit.
The server is now built using static linking to libosmscout, marisa, and libmicrohttpd. Its mainly done for two reasons: libosmscout API is changing as it is developed leading to incompatibilities between the versions; there is no need for other RPMs as you install the server. After an upgrade, I suggest to remove libosmscout-qt and libmarisa from your devices. I am planning to drop these packages from OpenRepos a bit later when users would move to 0.3.0 or later release.
NB! When you start using a new version, please update the maps! Few bugs were fixed in libosmscout library by fixing an importer. So, to get these fixes, you would have to update the maps with the latest version.
I have added a session log (see Settings to enable it). You could use it when reporting bugs to get the corresponding URL.
If your favorite client does not support routing yet, but supports GPX, you could now calculate the route and save the result in GPX format (see URL parameters in server GitHub project page).
Feathers McGraw
2016-11-12, 20:36
Guys, I just wanted to say thank you for working on this. Looking forward to testing it when there are prebuilt maps or an import tool!
ferlanero
2016-11-13, 11:16
Hi guys!
Thank you very much for this piece of gold for our Sailfish OS!
I'm trying to making OSM Scout Server working on a Jolla 1, but after installing correctly OSM Scout Server, import and convert maps correctly with libosmscout and copying it to
/home/nemo/Domcuments/Maps
After opening OSM Scout Server I only can see this error on main screen:
https://3.bp.blogspot.com/-YZAX6D-lpA8/WChHySgt1GI/AAAAAAAAGRY/cdZUkWslUz85rE2GDIC6pWNWO0_z2WWdQCLcB/s1600/Captura_20161113_001.png
I have to say that with same imported maps, and in same folder, OSM Scout aplicattion from Warehouse https://openrepos.net/content/karry/osm-scout everything is working as spectated:
https://1.bp.blogspot.com/-NChJQX8HXeQ/WChKC7oP2YI/AAAAAAAAGRk/bm_viX5787g5T956BF_7TXjja9XBr3xqACLcB/s1600/Captura_20161113_002.png
What is what I'm doing wrong?
For importing maps from http://www.geofabrik.de/ I'm doing:
./build.sh nordrhein-westfalen.osm.pbf for import with libscout
And copying the resulted files (which are a lot) into /home/nemo/Documents/Maps
Can you help me please?
Have you got any converted-working map with OSM Scout Server to compare the results and seeing what I'm doing wrong?
What is what I'm doing wrong?
For importing maps from http://www.geofabrik.de/ I'm doing:
./build.sh nordrhein-westfalen.osm.pbf for import with libscout
And copying the resulted files (which are a lot) into /home/nemo/Documents/Maps
Can you help me please?
Have you got any converted-working map with OSM Scout Server to compare the results and seeing what I'm doing wrong?
You are very-very close. Just few touches and you should be fine.
I suspect that your problems are just due to version mismatch. The libosmscout is developing very fast these days. In particular, the server together with Poor Maps exposed routing component and there are quite a few bugs fixed over the last weeks. As a result, the database format is changing as well. So:
1. Import the maps using the libosmscout version against which the server has been built: https://github.com/rinigus/libosmscout/releases/tag/0.0.git.20161110
2. When importing, I suggest to use the command line as in https://github.com/rinigus/osmscout-server#maps . This would delete the temporary and debug files from the maps folder. To my knowledge, the build script does not do that and you could end up with 2x larger import. Just adjust the path of the typefile and your map folder/pbf. Expect to get the similar size for imported map as you had PBF.
3. Keep the maps that you use for the server and OSM Scout separate. This is to avoid the version mismatch. We will converge eventually, but just do it for now while the development is rapid. Since you can specify the path for your maps in the server easily, I suggest to put the maps used by server into some other folder. On my phone, I use /home/nemo/Maps and subfolders with countries. So, to get Estonian maps, I use /home/nemo/Maps/estonia in my settings.
As you noticed, the maps are converted into multiple smaller files. That is correct and you would need them.
Please let me know if that helped. If not, we'll look further into it.
Here is my first take on simplified import. I have compiled Import statically for Linux x86_64 and wrote a small script to assist while importing. This version supports ONLY PBF-formatted OpenStreetMap maps. You can get maps from
http://download.geofabrik.de/
download the importer from
/link removed, see the next post/
uncompress the import tool and see included README.
This is a first try and I hope it works well. Please report back on how did it work and if there are any errors. I plan to drop it from dropbox in future and add the importer binaries to the releases at github. That way you will be able to get the importer corresponding to the release that you use.
Import tool was built using https://github.com/rinigus/libosmscout/blob/import/packaging/import/build_import.sh and "import" branch of libosmscout.
At present, only Linux on x86_64 architecture is provided. I don't know how to make similar build for Windows or Mac. If someone could chip in that would be of great help. Alternatively, Windows and Mac users could setup a virtual machine (VirtualBox, for example) with Linux and use it to import maps.
I have been working on making import of the maps simpler. The state is a bit work in progress, but it should be close to full release.
Right now, on each release in GitHub, the Import tool is built automatically using Travis and AppVeyor. As a result, we can get compiled import tools for Linux and Windows. While Linux version includes a small script that simplifies import to command "import mapfilename_in_pbf" , Windows build requires a bit more sophisticated input.
So, I would like to ask users interested in offline maps:
1. Linux users: Please test whether import works for you. Please report even if its just OK and include your distro. Would like to get conformation that it works in general, not just on PCs available to me.
2. Windows users: Please test whether you could run Import.exe by double clicking on it. No import will run, just command line help should be shown. Again please give a feedback, even a short one. If you wish to use it for map import as it is then I strongly suggest to use the similar options as in https://github.com/rinigus/libosmscout/blob/sailfish/packaging/import/linux/import.sh . Otherwise, you will end up with lots of debug info and get very large map directories that you have to store on device.
3. (quick note - I am getting help with this script via libosmscout developers mailing list) Someone with Windows scripting skills (minimal should already be better than mine), please write a script that would confirm to the spec below. If you don't know how to make it in full, please start discussion and write some parts that you can. Maybe then Windows users could finalize this small script to make map import easy.
The compiled packages are available at
https://github.com/rinigus/libosmscout/releases/tag/0.0.git.20161118
Let's use this for current testing. As soon as we get Windows scripts ready, we could make a new server release with the corresponding Import tools.
What I expect the Windows script should do:
Aim: finalize a compiled Windows package for simple importing maps into libosmscout
* The compiled binary of Import can be grabbed at https://github.com/rinigus/libosmscout/releases/tag/0.0.git.20161118 . The new script will be a part of this package and would allow us to distribute importers to the users.
* The script could be similar to https://github.com/rinigus/libosmscout/blob/sailfish/packaging/import/linux/import.sh
* While in Linux, we are used to start commands from terminal, in Windows its more common to drag-and-drop. So, on dragging downloaded OSM or OSM.PBF file, the new script should:
open a terminal (or PowerShell). I suspect its done automatically in Windows, but I don't know for sure
make a directory with the simplified name based on the dragged map file name. I used to remove -latest.osm.pbf from the map fine name and make directory just on the basis of remaining name. See Linux script for details
start Import.exe from relative bin\Import.exe . The style map.ost is under stylesheets directory. See compiled binary for package structure. Its expected that users just unpack it and can use importer straight away (no install is needed).
keep terminal open after Import.exe is finished, so the user could examine output for errors
on any error (cannot make export directory), stop the script (set -e in bash).
Since I am not familiar with Windows scripting, I am just afraid that it would take too long time for me to make it. So, please help even if its really trivial for you.
ferlanero
2016-11-25, 11:09
I have been working on making import of the maps simpler. The state is a bit work in progress, but it should be close to full release.
Right now, on each release in GitHub, the Import tool is built automatically using Travis and AppVeyor. As a result, we can get compiled import tools for Linux and Windows. While Linux version includes a small script that simplifies import to command "import mapfilename_in_pbf" , Windows build requires a bit more sophisticated input.
So, I would like to ask users interested in offline maps:
1. Linux users: Please test whether import works for you. Please report even if its just OK and include your distro. Would like to get conformation that it works in general, not just on PCs available to me.
2. Windows users: Please test whether you could run Import.exe by double clicking on it. No import will run, just command line help should be shown. Again please give a feedback, even a short one. If you wish to use it for map import as it is then I strongly suggest to use the similar options as in https://github.com/rinigus/libosmscout/blob/sailfish/packaging/import/linux/import.sh . Otherwise, you will end up with lots of debug info and get very large map directories that you have to store on device.
3. Someone with Windows scripting skills (minimal should already be better than mine), please write a script that would confirm to the spec below. If you don't know how to make it in full, please start discussion and write some parts that you can. Maybe then Windows users could finalize this small script to make map import easy.
The compiled packages are available at
https://github.com/rinigus/libosmscout/releases/tag/0.0.git.20161118
Let's use this for current testing. As soon as we get Windows scripts ready, we could make a new server release with the corresponding Import tools.
What I expect the Windows script should do:
Aim: finalize a compiled Windows package for simple importing maps into libosmscout
* The compiled binary of Import can be grabbed at https://github.com/rinigus/libosmscout/releases/tag/0.0.git.20161118 . The new script will be a part of this package and would allow us to distribute importers to the users.
* The script could be similar to https://github.com/rinigus/libosmscout/blob/sailfish/packaging/import/linux/import.sh
* While in Linux, we are used to start commands from terminal, in Windows its more common to drag-and-drop. So, on dragging downloaded OSM or OSM.PBF file, the new script should:
* open a terminal (or PowerShell). I suspect its done automatically in Windows, but I don't know for sure
* make a directory with the simplified name based on the dragged map file name. I used to remove -latest.osm.pbf from the map fine name and make directory just on the basis of remaining name. See Linux script for details
* start Import.exe from relative bin\Import.exe . The style map.ost is under stylesheets directory. See compiled binary for package structure. Its expected that users just unpack it and can use importer straight away (no install is needed).
* keep terminal open after Import.exe is finished, so the user could examine output for errors
* on any error (cannot make export directory), stop the script (set -e in bash).
Since I am not familiar with Windows scripting, I am just afraid that it would take too long time for me to make it. So, please help even if its really trivial for you.
Wow! In Linux (Ubuntu 16.04) works really great!! :D It's a really improvement to handle offline maps. Thank you very much!!
Now I have a question: OSMScout will be handle offline maps from sdcard? I love this app and how it works! So I love to see its evolution.
Thank ypou very much for all your efforts!!
OSM Scout Server should handle maps from SD. You just have to specify the map folder which is on SD card (use Linux path). Its under Settings, in OSM Scout Server GUI.
ferlanero
2016-11-25, 13:28
OSM Scout Server should handle maps from SD. You just have to specify the map folder which is on SD card (use Linux path). Its under Settings, in OSM Scout Server GUI.
You are right. Everything is working great! :)
Thanks
The new release is out: 0.4.0, followed by 0.4.1
A highlight of this release is introduction of pre-compiled map importer that can be used to convert OSM maps to the libosmscout format using Linux or Windows PC. See https://github.com/rinigus/osmscout-server#maps for instructions. While in Linux all should be rather simple, in Windows, an additional script simplifying the import is needed. I've got some help on libosmscout mailing list and, hopefully, we could make such a script soon.
I have removed OpenMP requirement of the server which should reduce the problems with installing it. Please let me know if the resulting performance of the server is degraded. I have not noticed any slowdown.
I would like to thank Carmen F. B. @carmenfdezb for Spanish translation of GUI interface. Its a great initiative!
Several bugfixes were incorporated in this release. In particular, a problematic routing case found by @otsaloma has been fixed in the upstream library.
0.4.1: Updated Spanish translation allowing you to get routing instructions in Spanish. Work of Carmen F. B. @carmenfdezb, thank you!
Feathers McGraw
2016-11-29, 10:04
I've been reading the documentation in preparation for giving this a go, it's all really clear and well presented! Thank you to whoever wrote it.
Documentation often gets overlooked in projects like this where development is fast, I believe having decent docs will really help build up a userbase quickly. Well done :)
Feathers McGraw
2016-11-29, 23:37
Successfully processed a map of the British Isles with the Linux importer (https://github.com/rinigus/libosmscout/releases/tag/0.0.git.20161118.1).
Processing time was under an hour with an i5 Haswell, which I think is pretty damn good!
Loaded it onto an SD card, and it is all working great (testing with Poor Maps). I think I could probably play around with the OSMSS settings to get the maps to load a bit faster, because the FP2 has 2GB RAM (looking at the settings, I think they're probably designed to be suitable for Jolla 1 too i.e. 1GB RAM?).
The first route I asked it to calculate was from Guildford to the Isle of Skye, which worked(!) - for those of you who don't live here, that's about 1000km! I was actually quite surprised that the route calculation was successful because OsmAnd had trouble with the length of the route, or the bridge to Skye (maybe it was a map split in OsmAnd?), or both. Not only that, OSM Scout Server was quicker than OsmAnd taking just a couple of minutes. to calculate. So basically, you've beaten the best offline map & routing app for Android already...
I have one remaining question, and a two requests...
In the stylesheets folder, there's a file called map.ost. I haven't copied it to the phone... should I, or is it just used for importing?
Would it be possible to add a folder selection for choosing the folder containing maps? Typing out the UUID of an SD card is pretty tedious!
Can the routing speeds input boxes be converted to mph using a settings toggle in the app to save the hassle of converting them?
Thanks again guys, I'm seriously impressed!
karlos devel
2016-11-30, 01:00
Successfully processed a map of the British Isles with the Linux importer (https://github.com/rinigus/libosmscout/releases/tag/0.0.git.20161118.1).
Processing time was under an hour with an i5 Haswell, which I think is pretty damn good!
Loaded it onto an SD card, and it is all working great (testing with Poor Maps). I think I could probably play around with the OSMSS settings to get the maps to load a bit faster, because the FP2 has 2GB RAM (looking at the settings, I think they're probably designed to be suitable for Jolla 1 too i.e. 1GB RAM?).
The first route I asked it to calculate was from Guildford to the Isle of Skye, which worked(!) - for those of you who don't live here, that's about 1000km! I was actually quite surprised that the route calculation was successful because OsmAnd had trouble with the length of the route, or the bridge to Skye (maybe it was a map split in OsmAnd?), or both. Not only that, OSM Scout Server was quicker than OsmAnd taking just a couple of minutes. to calculate. So basically, you've beaten the best offline map & routing app for Android already...
I have one remaining question, and a two requests...
In the stylesheets folder, there's a file called map.ost. I haven't copied it to the phone... should I, or is it just used for importing?
Would it be possible to add a folder selection for choosing the folder containing maps? Typing out the UUID of an SD card is pretty tedious!
Can the routing speeds input boxes be converted to mph using a settings toggle in the app to save the hassle of converting them?
Thanks again guys, I'm seriously impressed!
- I TRY TO IMPORT USING THE STATIC PROGRAM https://github.com/rinigus/libosmscout/releases/tag/0.0.git.20161118.1 BUT I GET A ERROR> CAN U SHOW UR EXAMPLE to IMPORT?--------------------------------debian-9 OS
The OSM Scout Server Day & Night layers have been added (https://github.com/M4rtinK/modrana/commit/6261c314f188561a7be65c30218b2b53a25e965f) to the default modRana layer definitions:
http://modrana.org/images/osmscout_server/modrana_osm_scout_server_day_layer.jpg
http://modrana.org/images/osmscout_server/modrana_osm_scout_server_night_layer.jpg
I also plan to (finally!) do a new modRana release in a few days to make this functionality available to users. :)
Thanks a lot yet again to rinigus and all others who made this possible! :)
Now I just need to add all support for all the other nice features OSM Scout Server provides, such as search and routing. ;)
Loaded it onto an SD card, and it is all working great (testing with Poor Maps). I think I could probably play around with the OSMSS settings to get the maps to load a bit faster, because the FP2 has 2GB RAM (looking at the settings, I think they're probably designed to be suitable for Jolla 1 too i.e. 1GB RAM?).
I used to test it on Nexus 4 (2GB RAM) and now I just moved to OnePlus X (3GB RAM). Since I don't have access to Jolla 1, I have no idea how well it works on that. I have followed the RAM usage of the server using SystemDataScope earlier and it was about 300MB RSS, if I recall correctly.
However, while RAM is handy, it would not have a major impact on map loading, I think (unless its cached by FS). You are probably bound by I/O from SD card, unfortunately.
The first route I asked it to calculate was from Guildford to the Isle of Skye, which worked(!) - for those of you who don't live here, that's about 1000km! I was actually quite surprised that the route calculation was successful because OsmAnd had trouble with the length of the route, or the bridge to Skye (maybe it was a map split in OsmAnd?), or both. Not only that, OSM Scout Server was quicker than OsmAnd taking just a couple of minutes. to calculate. So basically, you've beaten the best offline map & routing app for Android already...
No, I don't think we have beaten OsmAnd. Maybe in routing speed, but there is still a long way ahead - keep patches and issues coming!
There are many ways of contributing in addition to coding. For example: filing issues, working on map drawing styles (standard.oss on your phone) and making them better suited for Poor Maps and modRana.
I am wondering, how well does it cope with the wrong-sided roundabouts? Any other problems? Never tested it on UK maps, I must say. Please file issues in GitHub if you see any.
In the stylesheets folder, there's a file called map.ost. I haven't copied it to the phone... should I, or is it just used for importing?
Yes, map.ost is just used for importing. You could actually tweak it to import or exclude the features you want. Although, let us know if you changed it when you stumble on some bugs.
Would it be possible to add a folder selection for choosing the folder containing maps? Typing out the UUID of an SD card is pretty tedious!
To take the maximal positive out from the current situation, you could even try to memorize UUID while you type it in :) .
Yes, I plan to add file/directory selection dialogue. Not sure it will be in the next release, but it will come. You could probably use symbolic links for now to simplify the path. Issue filed: https://github.com/rinigus/osmscout-server/issues/42
Can the routing speeds input boxes be converted to mph using a settings toggle in the app to save the hassle of converting them?
Will deal with it: https://github.com/rinigus/osmscout-server/issues/41
Thanks again guys, I'm seriously impressed!
Thank you!
- I TRY TO IMPORT USING THE STATIC PROGRAM https://github.com/rinigus/libosmscout/releases/tag/0.0.git.20161118.1 BUT I GET A ERROR> CAN U SHOW UR EXAMPLE to IMPORT?--------------------------------debian-9 OS
Hi! Let's try to fix it.
Its not fully static since there are major issues involved if you have threads. So, its almost static in this release, but not fully.
Please paste the command that you run and its output.
In addition, please paste output to see whether dynamic linking is working - from importer dir run:
ldd bin/Import
and to know a bit about your machine:
uname -a
The OSM Scout Server Day & Night layers have been added (https://github.com/M4rtinK/modrana/commit/6261c314f188561a7be65c30218b2b53a25e965f) to the default modRana layer definitions
Thank you!
Windows users - great news for you! Michael Nenninger developed a script that simplifies the import of the maps to dragging and dropping action.
Get the latest importer package from https://github.com/rinigus/libosmscout/releases/tag/0.0.git.20161128.2 , unzip and follow the README.
The same version has been generated for Linux as well. There are small improvements in import script in Linux allowing you to run it from any path.
The importer link has been updated in GitHub osmscout-server README.md as well.
I have tested the imported maps against 0.4.1 release of the server. However, if you encourage any problems, please let me know.
Thank Michael Nenninger for developing the import script and I hope that Windows PC owners could enjoy offline maps on their Sailfish devices as well!
Fellfrosch
2016-12-02, 13:29
At First - thanks to you all for your great work!!!
One question:
I use OSM Scout Server with maps of Germany Austria and Switzerland. I recognized, that working with it isn't really fast, espacially when using Modrana (which often even refuses to load some map tiles). Using poormaps is way faster. Why is that?
And, can I speed it up in general. (Maybe by using a smaller map area?)
Feathers McGraw
2016-12-02, 14:07
I am wondering, how well does it cope with the wrong-sided roundabouts? Any other problems? Never tested it on UK maps, I must say. Please file issues in GitHub if you see any.
Wrong sided! Lol. I can't see any problems with the test routes I've generated, but I haven't actually used it for driving yet - will let you know if I spot anything.
At First - thanks to you all for your great work!!!
One question:
I use OSM Scout Server with maps of Germany Austria and Switzerland. I recognized, that working with it isn't really fast, espacially when using Modrana (which often even refuses to load some map tiles). Using poormaps is way faster. Why is that?
And, can I speed it up in general. (Maybe by using a smaller map area?)
I must say that I use modRana on PC and Poor Maps on the device so far. So, I haven't tested modRana too much on device.
Assuming that you use the latest version of the server (there could be different problems with the versions earlier than 0.3)
* you could be hitting timeouts for connections in modRana. As a result, modRana might ask multiple times for the same tiles. To check if its true, look into the events log shown in the server's main screen. Check whether the same URL is loaded multiple times.
* its probable that modRana and Poor Maps use different tile sizes. This is visible in URLs requested to render the tiles. Notice if there is a difference in "shift" and "scale" parameters in URL. Poor Maps uses by default 1024x1024 or 512x512 tiles that are not very well supported by modRana (or not at all yet) with modRana using 256x256 tiles (shift=0 and scale=1 by default). As a result, tile rendering can be done in bigger chunks in Poor Maps which should produce faster overall rendering. Optimization of tile size vs parallel rendering is not a simple problem and it may vary between the devices.
* maybe modRana's rendering is slower. If its true you could try to follow when the server states "Idle" on its cover and see when the map is rendered in reality. For that, browse the map in Poor Maps or modRana and peak by sliding from the right on the state of the server. In Poor Maps delay is minimal if any, in modRana - I haven't checked.
As for general speeding up - Smaller maps do help to render faster. Also, don't forget that Poor Maps and modRana make tile caches. So, when you open the same place next time, it should be faster.
The main drawback of smaller maps is that you would have to change them in the server's settings. Also, you cannot calculate the routes between maps. But if you know, for sure, that you are in some area and would be using only that for some time, its better to have a map of just that area. It should speed up things a bit.
We'll probably have to work on speed optimization at some point. At present, I am mainly working on getting the maps accessible for all, expose functionality better (GUI enhancement), and on bug fixes. There are few ideas already voiced in this thread that need to be checked better and I'll try to do it as much as I can.
rinigus
PS: meaning of parameters in URL is described at https://github.com/rinigus/osmscout-server#tiles for tile rendering
Fellfrosch
2016-12-02, 18:09
Holy Moly, thanks for this really detailed explanations!
Of course it would be nice if maps would render faster, but anyway its so nice to have finally native offline maps!
I must say that I use modRana on PC and Poor Maps on the device so far. So, I haven't tested modRana too much on device.
Assuming that you use the latest version of the server (there could be different problems with the versions earlier than 0.3)
* you could be hitting timeouts for connections in modRana. As a result, modRana might ask multiple times for the same tiles. To check if its true, look into the events log shown in the server's main screen. Check whether the same URL is loaded multiple times.
Maybe it could be also related to the old bundled urrlib3 (issue 146 (https://github.com/M4rtinK/modrana/issues/146)) ? I've updated it in the master branch, but it is not (yet!) in any released version. So users packages from OpenRepos & Jolla Store still run with the old urrlib3 and might be getting those errors.
That should hopefully not be an issue soon as I plan to release updated modRana packages today or tomorrow. :)
* its probable that modRana and Poor Maps use different tile sizes. This is visible in URLs requested to render the tiles. Notice if there is a difference in "shift" and "scale" parameters in URL. Poor Maps uses by default 1024x1024 or 512x512 tiles that are not very well supported by modRana (or not at all yet) with modRana using 256x256 tiles (shift=0 and scale=1 by default). As a result, tile rendering can be done in bigger chunks in Poor Maps which should produce faster overall rendering. Optimization of tile size vs parallel rendering is not a simple problem and it may vary between the devices.
Map tile scaling is unfortunately currently broken in modRana - and even if it worked, it is at present only meant to upscale current 256x256 pixel tiles - mainly to make text and small map features easier to read (at the cost of making them more blurry). The map scaling code (once fixed) should be relatively easy to adapt to supporting larger tile sizes - even though there might be some issues with overlays to overcome when some layers provide bigger tiles and others not.
* maybe modRana's rendering is slower. If its true you could try to follow when the server states "Idle" on its cover and see when the map is rendered in reality. For that, browse the map in Poor Maps or modRana and peak by sliding from the right on the state of the server. In Poor Maps delay is minimal if any, in modRana - I haven't checked.
You can also check the modRana debug logs: Go to Options -> Debug, enable the tile related debug logging & logging to a file and then restart modRana. On next start modRana should create a log file in ~/Documents/modrana_debug_logs/
The main drawback of smaller maps is that you would have to change them in the server's settings.
This is something I have been wondering about - is that really needed ? Couldn't OSM Scout Server check incoming tile requests against the bounding boxes/shapefiles of multiple data packs and return data for the first match ? Or, depending on cost, just run the query on one pack after another until there is a hit ?
Some heuristic can be applied for this to run all further queries after a hit on the same data pack until there is another miss. Users could also enable just some packs in the OSM Scout Server UI to reduce the number of lookups.
Also a related idea - client side pack specification. OSM Scout Server clients could specify data pack name for the lookup. They could get the available pack names either from OSM Scout Server via a special query or by other means (listing all folders the OSM Scout Server data folder or even by downloading data packs themselves and pointing OSM scout to them).
Just a couple of ideas. :)
Also, you cannot calculate the routes between maps.
Yeah, that's certainly not an easy problem to solve. I wonder how the commercial/proprietary navigation systems handle that ? Or do they just have larger areas (whole Europe, etc.) so they don't have to care about it ?
BTW, a naive idea how inter pack routing might be done: :)
0. assuming two adjacent packs with a relatively straight border between them & some overlap + simple start-destination routing
1. draw a straight line from start to destination
2. note where it intersects the border
3. search for suitable (way) nodes around the intersection points in both map-packs
4. find a node near the border that is in both packs
5. compute two routes: start - shared border node, shared border node - destination
6. return combined route as the result
This is of course very simplified and would probably returns some pretty inefficient routes. It also does not handle complex border geometries, routes over multiple packs, non overlapping packs or having only packs that have no shared border at all. :)
Again - just a couple of ideas. :)
Maybe it could be also related to the old bundled urrlib3 (issue 146 (https://github.com/M4rtinK/modrana/issues/146)) ? I've updated it in the master branch, but it is not (yet!) in any released version. So users packages from OpenRepos & Jolla Store still run with the old urrlib3 and might be getting those errors.
That should hopefully not be an issue soon as I plan to release updated modRana packages today or tomorrow. :)
I already forgot about this bug, thank you for reminding about it! Yes, that's probably the reason behind the slower response. It got exposed when I switched from Qt-based HTTP server to microhttpd and is fixed in modRana's master branch. As far as I remember, the symptoms were very similar to what was described.
I am pretty confident that this is a reason for significant slow-down. We just have to wait 1-2 days to test it :)
This is something I have been wondering about - is that really needed ? Couldn't OSM Scout Server check incoming tile requests against the bounding boxes/shapefiles of multiple data packs and return data for the first match ? Or, depending on cost, just run the query on one pack after another until there is a hit ?
Some heuristic can be applied for this to run all further queries after a hit on the same data pack until there is another miss. Users could also enable just some packs in the OSM Scout Server UI to reduce the number of lookups.
Something like this has been implemented by Lukas (@Karry) in his OSM Scout application, if I understand correctly. There are some significant hurdles to overcome for server if we want to do that.
Server renders the map in as many threads as you have CPUs. For that, it actually shares database access. If we want to render in parallel, we may end up having the same number of databases open. I think this may end up with too much RAM consumed by the server. So, we could start writing in some logic and lock/unlock threads depending on the accessed databases. Its all possible, but not trivial.
Alternative is to limit to a single thread and render one tile in a time. This may slow down it too much taking into account that the mobile devices may start lowering CPU frequency and switching off some cores when the load is about 1.
We can return to it a bit later. Maybe Lukas will implement it that way that it will be exposed via libosmscout anyway :-). Ideally, it would be solved on the library level and all applications using the library would benefit from it (server included).
We can try to formulate it as an issue and see when and how to resolve it.
Also a related idea - client side pack specification. OSM Scout Server clients could specify data pack name for the lookup. They could get the available pack names either from OSM Scout Server via a special query or by other means (listing all folders the OSM Scout Server data folder or even by downloading data packs themselves and pointing OSM scout to them).
I was thinking at one point to add database and the stylesheet as a part of URL. Its possible - we just get the same problems as I was pointing out above.
I would suggest to get other issues solved first: better GUI, search, rendering, and routing come up into my mind on server side. Better routing support on the client part (voice commands?).
Yeah, that's certainly not an easy problem to solve. I wonder how the commercial/proprietary navigation systems handle that ? Or do they just have larger areas (whole Europe, etc.) so they don't have to care about it ?
Routing between maps is possible. Many have solved it with what seems to be separate data packages: OSMAnd, MAPS.ME for example. From Valhalla's docs and few test runs, it seems that they also partition internally all the maps. So, its possible, just have to work on it. Corresponding issue is https://github.com/Framstag/libosmscout/issues/57
Again - just a couple of ideas. :)
Keep them coming. Let's just hope that we can implement at least a fraction of them.
We can return to it a bit later. Maybe Lukas will implement it that way that it will be exposed via libosmscout anyway :-). Ideally, it would be solved on the library level and all applications using the library would benefit from it (server included).
Yep, that definitely seems like the most sensible way. :)
I would suggest to get other issues solved first: better GUI, search, rendering, and routing come up into my mind on server side. Better routing support on the client part (voice commands?).
A significant usability issue is also IMHO the constantly changing data format - if it was more stable there could be data repositories users could use instead of having to generate their own data packs. It might be even possible to download the data directly from the GUI (from the OSM Scout Server UI or via clients).
I might be even able to arrange something with the Masaryk University NLP lab (https://nlp.fi.muni.cz/web3/en/NLPCentre) which is already kindly hosting the 100+ GB of Monav offline routing data (http://modrana.org/data/monav/) modRana can make use of (currently mainly on the N900).
But at the moment people might even have to re-generate their data files even due to am OSM Scout Server updates due to using a newer version of libosmscout.
Does it look like there might also be a solution for this issue ? I understand designing and supporting an unchanging binary data format is no easy task for libosmscout developers.
But at the same time I'm afraid the current unstable-data format can block many users from using OSM Scout Server & client apps - especially less advanced users who could make use of pre-generated map data.
Routing between maps is possible. Many have solved it with what seems to be separate data packages: OSMAnd, MAPS.ME for example. From Valhalla's docs and few test runs, it seems that they also partition internally all the maps. So, its possible, just have to work on it. Corresponding issue is https://github.com/Framstag/libosmscout/issues/57
Sounds promising! :)
Keep them coming. Let's just hope that we can implement at least a fraction of them.
Yeah, that's always the biggest problem! :D
A significant usability issue is also IMHO the constantly changing data format - if it was more stable there could be data repositories users could use instead of having to generate their own data packs. It might be even possible to download the data directly from the GUI (from the OSM Scout Server UI or via clients).
I might be even able to arrange something with the Masaryk University NLP lab (https://nlp.fi.muni.cz/web3/en/NLPCentre) which is already kindly hosting the 100+ GB of Monav offline routing data (http://modrana.org/data/monav/) modRana can make use of (currently mainly on the N900).
But at the moment people might even have to re-generate their data files even due to am OSM Scout Server updates due to using a newer version of libosmscout.
Does it look like there might also be a solution for this issue ? I understand designing and supporting an unchanging binary data format is no easy task for libosmscout developers.
But at the same time I'm afraid the current unstable-data format can block many users from using OSM Scout Server & client apps - especially less advanced users who could make use of pre-generated map data.
The last time we had to regenerate the maps was due to the bug in map importer. As far as I remember, the format stayed the same - just data was imported wrongly. This bug was exposed during testing of the routing and I suspect that there maybe more bugs to come. Eventually, it will stabilize, we just have to accept some fluidity at this time.
To offset the map format changes I worked on the importers. Hopefully, it would be of help for users.
Its great that you have such a host - it could really become handy. Note that the maps do have to be regenerated relatively frequently. I presume that once a month should be a target. And then we would have to see how to split the maps - countries, regions, ... Maybe you should get in touch with Lukas. He was working on getting maps distributed via downloads and he may be interested in expanding on that part of the project.
But the supply of the maps to the users is, indeed, a major problem that we would have to resolve.
The last time we had to regenerate the maps was due to the bug in map importer. As far as I remember, the format stayed the same - just data was imported wrongly.
I for some reason though it was much more unstable, so it's good to hear it's not the case. :)
This bug was exposed during testing of the routing and I suspect that there maybe more bugs to come. Eventually, it will stabilize, we just have to accept some fluidity at this time.
Sounds good. :)
To offset the map format changes I worked on the importers. Hopefully, it would be of help for users.
I've tried the Windows importer a while ago and it seem to work fine - nice trick with drag-and-drop support on the import script. :)
Its great that you have such a host - it could really become handy. Note that the maps do have to be regenerated relatively frequently. I presume that once a month should be a target.
That should be doable - I'm using one of the machines in the NLP compute cluster to generate the Monav data. It's a rather powerful computer (48 CPU cores & ~250 GB RAM) and can run a full update (download planet.osm.pbf, split to regional files, convert to Monav data, compress the results to tar.gz archives) in less than 24 hours. The conversion itself takes about 5-6 hours, the rest is "planet splitting".
BTW, the software used to generate the repository lives in the modrana-data-repository (https://github.com/M4rtinK/modrana-data-repository) repository and it should not be that hard to extend it to generate also libosmscout-compatible data once the format is considered stable enough. :)
And then we would have to see how to split the maps - countries, regions, ...
For the Monav data packs I've just used the same region definitions Geofabrik uses on their OSM data download page (http://download.geofabrik.de/). They even have polygon files for each area - this is what I use to generate the regional *.osm.pbf files with osmosis from a fresh planet.osm.pbf file. No need to download all their PBF files every time the repository is updated. :)
That should be doable - I'm using one of the machines in the NLP compute cluster to generate the Monav data. It's a rather powerful computer (48 CPU cores & ~250 GB RAM) and can run a full update (download planet.osm.pbf, split to regional files, convert to Monav data, compress the results to tar.gz archives) in less than 24 hours. The conversion itself takes about 5-6 hours, the rest is "planet splitting".
BTW, the software used to generate the repository lives in the modrana-data-repository (https://github.com/M4rtinK/modrana-data-repository) repository and it should not be that hard to extend it to generate also libosmscout-compatible data once the format is considered stable enough. :)
For the Monav data packs I've just used the same region definitions Geofabrik uses on their OSM data download page (http://download.geofabrik.de/). They even have polygon files for each area - this is what I use to generate the regional *.osm.pbf files with osmosis from a fresh planet.osm.pbf file. No need to download all their PBF files every time the repository is updated. :)
It would be great if you could subscribe to https://lists.sourceforge.net/lists/listinfo/libosmscout-development and introduce your options to distribute maps to the list. I'll mention this discussion over there as well. Lukas has been working on generating the maps for distribution and maybe you together could come up with some great solution. Through direct communication between you and Lukas we would ensure that your experience is not lost in translation :)
Hi all. I read this thread completely and I have to say thanks to all developers for your work and to all people who test our applications!
@rinigus you should consider to submit server to Coding Competition (https://wiki.maemo.org/Maemo.org_Coding_Competition_2016) :-) There are some interesting prices like Jolla C. Other apps are welcome too!
To map distribution, I just recap what I send to libosmscout mailing list: server part for OSM Scout is ready, scripts are available here (https://github.com/Karry/osmscout-scripts). I host it on Rasperry PI (https://osmscout.karry.cz/) with 1 TiB disk dedicated for maps. It is dwarf to compare to your machine @MartinK :-) I will focus on client side next days/weeks. It will be available for osmscout server too off course. When everything will be ready, we can start working on map delivery network (MDN ?). What is university status to your server Martin. It is officially supported or it is tolerated "punk activity"?
To map distribution, I just recap what I send to libosmscout mailing list: server part for OSM Scout is ready, scripts are available here (https://github.com/Karry/osmscout-scripts). I host it on Rasperry PI (https://osmscout.karry.cz/) with 1 TiB disk dedicated for maps. It is dwarf to compare to your machine @MartinK :-) I will focus on client side next days/weeks. It will be available for osmscout server too off course. When everything will be ready, we can start working on map delivery network (MDN ?).
Sounds good! We could try to create some simple osmscout data repo spec - for example how a JSON document describing available map packs looks like. Like this repos/repo mirrors would be independent from how they are created (your scripts, the modrana-data-repository generator, manual generation, etc.) and people will still be able to use them in a unified way from applications supporting the spec. :)
What is university status to your server Martin. It is officially supported or it is tolerated "punk activity"?
My bachelors & masters thesis were led by a supervisor from the NLP lab. We are still in touch after I've finished my studies and the current Monav data repository (and the whole modrana.org website) is hosted by the lab with his & labs consent. And I do credit them (https://nlp.fi.muni.cz) for this where appropriate this. :)
I will of course first ask them if they are OK with generating & hosting another data set, but I kinda expect it should not be an issue. In comparison to some of the corpus processing they do our data sets are very very small. :)
Map downloader is ready for merge to upstream library: https://github.com/Framstag/libosmscout/pull/185
Now, Silica UI...
@karry - great work, thank you very much! This is surely a step in the right direction!
I have just published a new release: 0.5.0
This release addresses the GUI shortcoming highlighted by @Feathers McGraw:
its possible to specify units (Metric or Imperial). Since we use only km and mi., there is no need to separate british from american, as far as I understand
to select the map (and few other settings), you are shown directory or file selection dialog. Should work for SD cards as well.
Unfortunately, my changes break the Spanish translation in few places. Maybe we will get an update a bit later and I'll publish new point releases.
If there are some other requirements with GUI, let me know. For me, its OK and I am planning to focus on some aspects of the library. There were suggestions by @otsaloma that were given early on and I'd like to see what I can improve.
@rinigus, I tested v.5 with Poor Maps.
- Great work, but it's not simple for ordinary user for this moment to import maps. NokiaMaps for N9 is good exemple for me.
- I need 2 districts - ural and volga. I haven't problem with ural-fed-district-latest.osm.pbf, but I don't see map after import from volga-fed-district-latest.osm.pbf(I configured path for server). And I see in folder 186.5MB instead 359.9MB for volga-fed-district-latest.osm.pbf. Or it's good?
- is it possible to navigate between 2 districts?
@rinigus, I tested v.5 with Poor Maps.
- Great work, but it's not simple for ordinary user for this moment to import maps. NokiaMaps for N9 is good exemple for me.
- I need 2 districts - ural and volga. I haven't problem with ural-fed-district-latest.osm.pbf, but I don't see map after import from volga-fed-district-latest.osm.pbf(I configured path for server). And I see in folder 186.5MB instead 359.9MB for volga-fed-district-latest.osm.pbf. Or it's good?
- is it possible to navigate between 2 districts?
Thank you!
Sure, import of the maps can be improved. As you could see from yesterday's post by @karry, he is working on it :) . See also earlier discussion between @karry and @MartinK on the subject. We could expect that the devices would be able to download maps directly from some server, but there maybe some time before it will be up and running.
So, at present, an ordinary user would have to download maps to PC, drag-and-drop them on importer tool, and transfer imported folders to device. I think its reasonably simple and, taking into account that @karry's work, there is no point in making fancy GUI on PC side of things.
Re Ural and Volga: I tried to import Volga district and it leads to 328MB folder. When moved to device, all worked fine. So, I think something went wrong during import or while you copied to the device. Note that you should have got all the files that you have for Ural, maybe something is missing. If the importer finishes with such small import, please tell me what OS do you use, importer version, and maybe you could also provide Importer output? Otherwise its hard to debug the issue.
Routing between 2 maps is not possible (yet). What you can do is to join the regions that you are interested in. Use OSM-provided tools for that, as explained at https://github.com/rinigus/osmscout-server#maps . In particular, look at https://wiki.openstreetmap.org/wiki/Osmconvert#Parallel_Processing
You may consider to reduce the maps to the part that you actually use using osmconvert.
Thank you!
... What you can do is to join the regions that you are interested in.
Btw, you can define more OSM (osm.pbf) files for one import. When they are overlapping, routing should be possible.
Btw, you can define more OSM (osm.pbf) files for one import. When they are overlapping, routing should be possible.
@karry - thanks for a tip, didn't know that! Note that the import script that is distributed with pre-compiled importer is not aware of that. So, I don't think it would work with it and the included import executable would have to be called directly.
Feathers McGraw
2016-12-11, 23:48
@karry - great work, thank you very much! This is surely a step in the right direction!
I have just published a new release: 0.5.0
This release addresses the GUI shortcoming highlighted by @Feathers McGraw:
its possible to specify units (Metric or Imperial). Since we use only km and mi., there is no need to separate british from american, as far as I understand
to select the map (and few other settings), you are shown directory or file selection dialog. Should work for SD cards as well.
That was quick! Thank you very much for making those changes.
Since you seem to be on a roll, I noticed another thing the other day, when I was searching for pubs near my current location from within Poor Maps. No pubs were returned in the results despite there being several nearby, but there were a few public transport platforms in there for some reason...not sure if poor maps is requesting the wrong POI type, or if it's a server side thing? The map package was the one I created using the script as we discussed previously in this thread.
@rinigus I understand there is no other way for this moment.
volga. It's not problem with copy from PC to phone. I use ubuntu 16.04 and last importer. I'll try to send Importer output, but how to do it? to get from terminal(it's long) and send after that.
Thanks for help.
P.S. It's not simple to replay by SfOS from phone :-)
@rinigus I understand there is no other way for this moment.
volga. It's not problem with copy from PC to phone. I use ubuntu 16.04 and last importer. I'll try to send Importer output, but how to do it? to get from terminal(it's long) and send after that.
Thanks for help.
P.S. It's not simple to replay by SfOS from phone :-)
Great, Linux makes it all easer for me. I would suggest to run importer again and see
du -sh volga-fed-district
For me it resulted in 328MB.
You could copy the output and paste it into https://ptpb.pw/f . For that, go to that web-site, copy your text into the large text box (central), set the expiry date (bottom right) for ~week from now, and press paste. As a result, you will get a light blue box with "status: created". Click on "created" and send me the URL via this forum. Then we can see whether import was fine.
Is it possible that you run out of storage? RAM is also an option, how much do you have in Ubuntu?
That was quick! Thank you very much for making those changes.
Since you seem to be on a roll, I noticed another thing the other day, when I was searching for pubs near my current location from within Poor Maps. No pubs were returned in the results despite there being several nearby, but there were a few public transport platforms in there for some reason...not sure if poor maps is requesting the wrong POI type, or if it's a server side thing? The map package was the one I created using the script as we discussed previously in this thread.
Thanks!
Its surely not Poor Maps issue. The problem could be either the wrong POI type or absence of the pubs nearby.
To see the available types, use the URL given https://github.com/rinigus/osmscout-server#list-of-available-poi-types . For that, start the server and visit
http://localhost:8553/v1/poi_types
in the browser. [There is no better way right now]. When you look for nearby POIs, you are searching for objects which are within the given distance (with some precision) and are of the corresponding type. The type comparison is done by lowercasing the type you enter and assuming that you entered a substring of a type.
If the type was correct and pubs are still missing, try to do the same search in Poor Maps using some other search provider. You could also go to http://www.openstreetmap.org and see if you can find the pubs over there. Try to figure out what type they are and see if you could find them.
Let me know. If something is wrong and you see all these pubs in OSM, we would debug it further. Then I would need an example of a missing pub :)
@rinigus I'll do this slightly later.
RAM = 8 GB. I don't see warning "out of space".
du -sh volga-fed-district
For me it resulted in 328MB.
You could copy the output and paste it into https://ptpb.pw/f . For that, go to that web-site, copy your text into the large text box (central), set the expiry date (bottom right) for ~week from now, and press paste. As a result, you will get a light blue box with "status: created". Click on "created" and send me the URL via this forum. Then we can see whether import was fine.
- after du: "373M volga-fed-district"
- Importer output: https://ptpb.pw/Uyha , but it's only part. the beginning of the process is missing in terminal :-(
- 2 differencies today:
I deleted volga-fed-district.poly from folder
I enlarged free space from 4GB to 6GB.
- I see in Poor Maps ural-district as OSM_Scout_day_@1x and volga-district as OSM_Scout_day_@2x, OSM_Scout_night_@1x, OSM_Scout_night_@2x. And I don't see the difference after modification of path in server config.
- after du: "373M volga-fed-district"
- Importer output: https://ptpb.pw/Uyha , but it's only part. the beginning of the process is missing in terminal :-(
- 2 differencies today:
I deleted volga-fed-district.poly from folder
I enlarged free space from 4GB to 6GB.
- I see in Poor Maps ural-district as OSM_Scout_day_@1x and volga-district as OSM_Scout_day_@2x, OSM_Scout_night_@1x, OSM_Scout_night_@2x. And I don't see the difference after modification of path in server config.
@XOleg, all looks good with import - its finished. See "Import OK!" before deleting tmp files :)
Now regarding the jiggling with the maps, and that's important for all of us as users. Poor Maps and modRana do all they can to reduce the downloads. So, they cache the map tiles. Which is great, unless you are as @XOleg jumping from one map to another. In this case, when you change the map in the server, I suggest that you delete the cache in the client, close it for a good measure and reopen. That way new regions will be loaded.
There are ways to resolve it, but I am a bit on a fence to see what happens with the library. @Karry was working on support of multiple maps and maybe that's gonna be merged. At present, I am addressing other issues and let's see if we can get back to it.
@XOleg, please let me know if the cleaning of the cache helped.
On the side note: to celebrate opening of the store to the ported devices, I published OSM Scout Server in the store as well. You could choose your favorite provider now. I'll continue to publish it in OpenRepos as well.
@rinigus thanks for help. yes, cleaning of the cache helped.
hmm, Russia is not Monaco :-)
I have just published a new release: 0.5.2 followed by 0.5.3
The main change introduced in this release is addition of new translations - Czech by @Karry and Swedish by @eson57 (thank you very much!). I have updated also the coordinates returned on search of the street to ensure that they are on the street.
At present, I mainly work on improving the search engine. I am trying to adapt libpostal (thank you @otsaloma for suggestion!) for normalization and parsing of the search request. The results look very promising, but there are quite a few challenges with pushing it into mobile frame (RAM usage) and distribution of search databases. I will give an update when a bit more details are known.
OK, this is for getting feedback from the users and developers.
I have been working on integrating libpostal into the search functionality, as suggested by @otsaloma during his testing of OSM Scout Server. The development so far has been documented and discussed via mailing list https://sourceforge.net/p/libosmscout/mailman/libosmscout-development/thread/CA+4hNMHv9Eiodn1cSPuvJnWHnt65-ZxmeU+eXROP4g8oPsXOBQ@mail.gmail.com/ . Below, is a summary of the thread.
Libpostal is a library that parses user input and splits the string into hierarchy: house number, street, city, ... In addition, it normalizes the strings to make it simple to use them for search. So, in theory, it would allow you to look for addresses without worrying about case, commas, accents and few other things. In practice, as much as I have tried, it works quite well. There are few cases were it fails and it would have to be addressed.
It turned out that there were no geocoders written so far around libpostal with an exception of integration it into Mapzen services (read: big iron). So, I wrote one: https://github.com/rinigus/geocoder-nlp . Its a small library that allows you to get coordinates and full transversal of an address/POI using a SQLite database.
When testing on mobile (OnePlus X), I was able to get search results for Germany (one of the worse cases due to huge OSM import) within 10s, usually within 5s (depends on how many levels you give). For that, RAM usage was <250MB RSS. So, I think it is possible to use this on mobile, especially taking into account quality of the results.
To make it mobile-worthy, I had to work around libpostal huge default memory requirements: 1GB RSS. This is due to the approach of libpostal - covering the planet in one go. So, it parses lots of languages and should be able to parse any address in the world. To reduce the requirements, I split the databases by country. Work is documented at https://github.com/openvenues/libpostal/issues/132 . In short, to make it work, custom databases are needed.
And here I come to the main issues with this approach:
1. Storage space requirements These come from libpostal databases and SQLite database for geocoding. Note that the numbers below are in addition to what is needed for libosmscout.
Libpostal requires language parser (I haven't split that, so it covers all with 690MB. In addition, country-specialized databases are
28M CZ
146M DE
10M DK
5.1M EE
9.4M FI
16M SE
So, if you want to cover FI and SE, you would need 690+9.4+16 MB.
In addition, SQLite database that is generated to get geocoding working. For Estonia, its 14MB (compare that with 69MB import for libosmscout). For Germany, its 1.1GB (compare it to 2.6 GB of libosmscout import). So, this is significant.
2. Database generation. Now, to generate libpostal databases, you would have to engage in machine learning. In short, that is not for an average user. It requires server-grade hardware, jiggling with 30GB files and significant CPU requirements. Good news is that these databases don't change frequently. So, its possible to make them using and extending scripts that I wrote already. Since its NLP in its pure form applied for postal addresses, probably Martin's servers in university could be considered? In short, to use this approach, distribution of the databases has to be provided.
That's all for now. I wonder, whether you as a user would be interested in such approach taking into account the storage requirements? I will surely be using it on my device, but if noone is interested ...
cheers,
rinigus
Feathers McGraw
2017-01-15, 13:56
I wonder, whether you as a user would be interested in such approach taking into account the storage requirements? I will surely be using it on my device, but if noone is interested ...
I will absolutely use this, and the storage requirements seem entirely reasonable to me. Thank you for your continued work on this!
otsaloma
2017-01-15, 16:40
The device requirements look about OK to me, and if they look about OK now, they'll be a complete non-issue in two or three years. Design for the future! Regarding databases, my impression is that server resources (both computation and storage + traffic) are quite cheap these days, and if the data doesn't change often, you only pay for computation time you need. So, if Martin can provide servers that's good, if not, it shouldn't be a problem either. You should be able to collect server fees as donations, or maybe charge for the app if that's one day possible. So, if the results are good, I'd say go for it.
I'm not likely to use offline maps myself though, with or without this change, as online is not a problem for me. But, if I did, I'd want to use something that works.
Feathers McGraw
2017-01-15, 16:49
I'm not likely to use offline maps myself though, with or without this change, as online is not a problem for me. But, if I did, I'd want to use something that works.
Out of interest, why not? When I generated a map for the whole of the UK it only took up 1GB of space.
otsaloma
2017-01-15, 17:11
Out of interest, why not? When I generated a map for the whole of the UK it only took up 1GB of space.
In Finland unlimited data plans are the norm, so using online providers doesn't bring any cost. And online providers tend to be much better quality in terms of map rendering, search query parsing, navigations instructions, etc.
When travelling in Europe, I tend to just download Google map tiles and copy them to the device. After that, only searches and navigation cause data traffic, and those are just small JSON files, it's such little traffic that it's not a problem in Europe these days.
When travelling outside Europe, offline maps could be useful, but that's rare for me.
The device requirements look about OK to me, and if they look about OK now, they'll be a complete non-issue in two or three years. Design for the future! Regarding databases, my impression is that server resources (both computation and storage + traffic) are quite cheap these days, and if the data doesn't change often, you only pay for computation time you need. So, if Martin can provide servers that's good, if not, it shouldn't be a problem either. You should be able to collect server fees as donations, or maybe charge for the app if that's one day possible. So, if the results are good, I'd say go for it.
I'm not likely to use offline maps myself though, with or without this change, as online is not a problem for me. But, if I did, I'd want to use something that works.
Thank you and @Feathers McGraw for positive support. As for servers, I guess you are right. Just don't want to engage right now into organizing financial aspects of it and would prefer to focus on making the server that works well.
karlos devel
2017-01-15, 22:23
The device requirements look about OK to me, and if they look about OK now, they'll be a complete non-issue in two or three years. Design for the future! Regarding databases, my impression is that server resources (both computation and storage + traffic) are quite cheap these days, and if the data doesn't change often, you only pay for computation time you need. So, if Martin can provide servers that's good, if not, it shouldn't be a problem either. You should be able to collect server fees as donations, or maybe charge for the app if that's one day possible. So, if the results are good, I'd say go for it.
I'm not likely to use offline maps myself though, with or without this change, as online is not a problem for me. But, if I did, I'd want to use something that works.
for me offline it is much better that online,Both have advantages and disadvantages!!!
what about REroute in PoorMaps @otsaloma There is possibility to fix it?
otsaloma
2017-01-16, 21:42
what about REroute in PoorMaps @otsaloma There is possibility to fix it?
Rerouting is planned, yes, and possible, in time.
karlos devel
2017-01-16, 21:49
perfect! @otsaloma
Rerouting is planned, yes, and possible, in time.
Just in case this is the code modRana uses to check if the current route is being followed:
https://github.com/M4rtinK/modrana/blob/master/modules/mod_turnByTurn/mod_turnByTurn.py
If should be fairly efficient as it works fine even on the N900 for fairly long roads. Feel free to (re)use. :)
This is announcement for a new release: 0.6.0
This release introduces libpostal (https://github.com/openvenues/libpostal ) based geocoder that is expected to significantly improve the search functionality of the server. To my knowledge, its the first time that libpostal has been used on mobile. For background and the requirements, see earlier post http://talk.maemo.org/showpost.php?p=1521995&postcount=65
In addition to libpostal-based geocoding, the server is compiled with the newer libosmscout version that addresses few drawing issues, to name the few changes. Also, Swedish and Spanish translations have been updated.
This release should be considered as a "technology preview" with the distribution of the required databases and simplification of selections planned to be addressed in future.
The new search functionality is disabled by default. To enable, you would first need to get few databases in addition to the regular libosmscout map databases. For distribution, I am using mega.nz for now:
http://tiny.cc/geocodernlpdata
I may have to update the corresponding link on mega from time to time, to get decryption in sync.
The required databases are listed below with the file location at the distributed folder:
* libpostal language parser: postal/postal-global.tar.bz2
* libpostal country-specific database: postal/countries/<SELECT THE NEEDED ONES>
* geocoder database: geocoder/<SELECT THE NEEDED ONES>
Country-specific databases are generated in accordance with http://download.geofabrik.de/ distribution. Exceptions were:
* Russia was added as a part of Europe;
* Kosovo is missing since I could not find any records from this country in libpostal training set.
Some countries are joined with the others. For example, Ireland and Northern Ireland are joined. In this exceptional case, I had to join Ireland and UK records to provide libpostal country-specific database.
All the archives (tar.bz2 or .bz2) have to be extracted before use. I suggest to download them to PC, extract and then move to the device.
As soon as you have all databases on device, enable new geocoder by
* Go to menu Geocoder
* Select "Use geocoder-nlp"
* Select location of libpostal language parser (extracted postal-global.tar.bz2)
* Select libpostal country-specific database (extracted postal/countries/AA.tar.bz2, for example)
* Select geocoder database (for example AAland.sqlite)
* Select the languages that you want to use for parsing. Don't leave "All languages selected", that may lead to too large memory requirements! At present, select languages used in OSM map for that region.
* Accept new settings and enjoy
Searches (while using search in Poor Maps) and routing-related searches would run then via new geocoder.
As every intelligence, AI of libpostal does make mistakes. To help you out in these cases, the server logs how was the search partitioned (house number, street, ...). As an additional parser, I added a primitive parser that constructs hierarchy by taking your input and splitting it into parts assuming that parts are split by comma. So, in these cases, use the following notation "house number, street, city, ..." (i.e. move from inner to outer admin region). To use primitive parser, enable it in Geocoder options. Primitive parser shows hierarchy with h-number ids in logs.
For map application developers: I have introduced a new version of search results available via v2, see https://github.com/rinigus/osmscout-server#location-search-version-2 . In addition to search results, it adds information on how the search string was parsed. As far as I have seen, something similar is expected or already done by Mapzen to give the feedback on libpostal parsing.
I have just released a new update: 0.6.1
This is a work of the translators who updated the Swedish and Spanish translations - thank you @carmenfdezb and @eson57!
Please note that this version uses exactly the same libosmscout library as 0.6.0. Due to the changes in libosmscout, there is a bug that can cause server crash when you use older maps with the border style (https://github.com/rinigus/osmscout-server/issues/61). @Karry has found the cause and its fixed upstream. However, since it would require generation of new maps, I decided that such rather rare case (use of borders style) does not require the update of underlying library. This update is planned for future major release.
Together with others, I have been working on improving import into libosmscout databases. We are testing one patch right now that should make location hierarchies more accurate on imports. As soon as its ready, I am planning to regenerate geocoder-nlp world databases (sqlite).
I am curious if anyone has tried geocoder-nlp/libpostal search functionality in 0.6.x? If you did, how where the results? So far there has been no feedback on this.
seiichiro0185
2017-02-04, 11:20
I did a quick test with the new libpostal/geocoder stuff, here are my thougts so far:
Search results are a lot more usefull, putting in <Street> <Number>, <ZIP-Code> <City> gives the correct result almost 100% of the time, without libpostal it often would find anything but the Address that was searched for
Time for an address search seems to depend a lot on the "comonness" of the street name:
- searching for an address with "Parkstraße", which is a very common street name in Germany, takes several Minutes to complete, even when full ZIP-Code and City is given
- In contrast a search with an unique (or uncommon) street name yields a result in under 5 seconds.
I don't know how the search code works, but it seems to me it searches for the Street Name first, maybe it would be more efficient to first search City / ZIP-Code and then the Street (just a wild guess, I have not looked at the code so far)
Since I just played around with the new functionality a bit, thats it for my first impressions. All tests where done on a JollaC with all the data on the SD-Card
Search results are a lot more usefull, putting in <Street> <Number>, <ZIP-Code> <City> gives the correct result almost 100% of the time, without libpostal it often would find anything but the Address that was searched for
Time for an address search seems to depend a lot on the "comonness" of the street name:
- searching for an address with "Parkstraße", which is a very common street name in Germany, takes several Minutes to complete, even when full ZIP-Code and City is given
- In contrast a search with an unique (or uncommon) street name yields a result in under 5 seconds.
I don't know how the search code works, but it seems to me it searches for the Street Name first, maybe it would be more efficient to first search City / ZIP-Code and then the Street (just a wild guess, I have not looked at the code so far)
Thank you very much for the profiling and an example!
I will use your case for optimization of the search. It does search first for the city (or county) and then moves over to within searches, but I should probably change SQL query slightly to apply these refinements earlier. I should be able to make it faster :)
Unfortunately, ZIP codes are not used in the search. So, there is no point in entering them. I don't know yet how to apply ZIP codes and it has been rather low priority for now. I may look into it a bit later (or anyone interested can look into implementing zip code searches as well).
Thanks again, its quite helpful to get your feedback!
@rinigus
access to http://tiny.cc is blocked in Russia :-(
@seiichiro0185, I made a new SQLite database that should speedup the searches. On OPX, the search time for Parkstraße 5 frankfurt dropped from 20+s to 1-2s. There is an additional index in the database for that which made it bigger. I will look into how to reduce the size of it.
Please test whether it works for you. Using the same server code, just get the new database from "testing/europe" subfolder in mega.nz share.
@XOleg, that's really strange. Current link is
https://mega.nz/#F!8EsxXRTT!arad2NJ9rEhpXXn-YXsF8w , but maybe mega.nz is blocked as well. I will work on database distribution soon and maybe we can find servers that are possible to connect to from Russia. Although, I cannot really check it out from here.
@rinigus
Thanks :-) I downloaded from Mega.
this works, but...
1. I have for Street,8: Street,8-Street,18-Street,81-Street,181-Street,22/8 etc... It's OK?
2. I request Shop_A, MyCity and I have CityB-MyCity,highway...
3. I have for Shop_B: monument "Shop_B" only(not Shop_B. because names are the same).
I have good answers for alternative method. I tested only for my city, cyrillic, offline maps.
P.S. It's possible to download maps for russian regions from http://gis-lab.info/projects/osm_dump/
this works, but...
1. I have for Street,8: Street,8-Street,18-Street,81-Street,181-Street,22/8 etc... It's OK?
2. I request Shop_A, MyCity and I have CityB-MyCity,highway...
3. I have for Shop_B: monument "Shop_B" only(not Shop_B. because names are the same).
I have good answers for alternative method. I tested only for my city, cyrillic, offline maps.
1: In this case it should search for Street* first, and 8* after that. So, I would expect the houses which start with 8, but not 18 and 181. For 22/8, it may have split it into two parts by libpostal normalization, so that could probably happen. Fortunately, I sort it according to the string length => you should get 8 as a first hit. Please verify 18 and 181 houses in your example. If they are there, would be good to get an example which I can specifically test.
2: If the search cannot find a specific reply (judged by resolved number of hierarchies), the result for a search with the latest resolved level dropped is returned. I presume the Shop_A was not found and then it just looked for anything with MyCity.
3: It looks to me that Shops A and B are missing. Let's try to fix it. I presume its due to the fact that these shops are not included in local search database in libosmscout. For fix, I would need their type in OSM.
So, in general, if you miss some POIs in the search, let me know. Find the type as follows:
a. go to http://www.openstreetmap.org
b. find that missing POI, preferably in search
c. see what's its type. For example, for http://www.openstreetmap.org/node/2558987540 its "amenity" "restaurant".
d. let me know about it (github at https://github.com/rinigus/geocoder-nlp or this thread)
I would have to include that by modifying import data type file and regenerate planet databases and push them out.
I presume we would have to do few iterations and let's hope it would stabilize at some point :)
P.S. It's possible to download maps for russian regions from http://gis-lab.info/projects/osm_dump/
Thank you for suggestion. Its not a problem with getting PBFs - I use geofabrik for that. Its serves whole planet chopped by countries. Problem is mainly how to distribute my databases :)
I am now regenerating a planet using a newest libosmscout importer that should be able to resolve admin hierarchies much better. As soon as its ready, I'll push it into mega.nz and would update the links.
seiichiro0185
2017-02-06, 05:21
@seiichiro0185, I made a new SQLite database that should speedup the searches. On OPX, the search time for Parkstraße 5 frankfurt dropped from 20+s to 1-2s. There is an additional index in the database for that which made it bigger. I will look into how to reduce the size of it.
Please test whether it works for you. Using the same server code, just get the new database from "testing/europe" subfolder in mega.nz share.
Just did a quick Test with the new sqlite DB and the search is a lot faster now. Searching for the "Parkstraße" address which took 3-4 minutes before now takes about 10 seconds.
I'll test some more and report back if I find anything else ;)
Thanks for your amazing work on this!
monkeyisland
2017-02-06, 06:27
Hi there is it possible, to integrate german language to Scout Server ?
so that i have german driving instructions with offline support ?
:o
@seiichiro0185: great to hear that it helped!
I have just uploaded the new version of the geocoder databases to mega.nz. As earlier, please use
http://tiny.cc/geocodernlpdata
since I can change the link after each upload. For tiny.cc-blocking countries, link is
https://mega.nz/#F!1RtTyQwS!r5QDqjwQKB5TU3YtEu9Q7Q!8Zt1RaQB
Databases are under geocoder subfolder. Postal part is unchanged.
Changes:
1. Shops are listed. Reported missing by @XOleg, please check if its ok now.
2. I have changed the internal database index and it should search much faster now. That's important for countries like Germany. As a result, database increased a bit in a size, but its probably reasonable compromise.
3. Resolution of admin relationships should be much better than before. So, if there was an error in earlier version, check if its fixed. I expect it to be as good as on http://www.openstreetmap.org, with just few misses (I'll let you find them if you can).
@seiichiro0185, would be great if you could test the speed with the latest version. It should be as fast as the version you had, but the file size is smaller.
Re localization, aka in German, @monkeyisland:
Sure, its possible. Its just German users would have to translate the strings. See https://github.com/rinigus/osmscout-server/tree/master/translations for instructions and example translations. I'll be happy to merge PRs with translations.
If someone knows how to setup easier system for users, would be great if you could submit patch with its implementation. Or open an issue and try to give me detailed instructions at https://github.com/rinigus/osmscout-server . The easier it is for me, the faster it would be implemented. :)
Thank you all for using / testing it!
1: In this case it should search for Street* first, and 8* after that. So, I would expect the houses which start with 8, but not 18 and 181. For 22/8, it may have split it into two parts by libpostal normalization, so that could probably happen. Fortunately, I sort it according to the string length => you should get 8 as a first hit. Please verify 18 and 181 houses in your example. If they are there, would be good to get an example which I can specifically test.
2: If the search cannot find a specific reply (judged by resolved number of hierarchies), the result for a search with the latest resolved level dropped is returned. I presume the Shop_A was not found and then it just looked for anything with MyCity.
3: It looks to me that Shops A and B are missing. Let's try to fix it. I presume its due to the fact that these shops are not included in local search database in libosmscout. For fix, I would need their type in OSM.
1. rinigus, excuse me for *8*,but */8 exists. I see *8* for alternative method.
2, 3. It's OK after update.
Thank you for suggestion. Its not a problem with getting PBFs - I use geofabrik for that. Its serves whole planet chopped by countries. Problem is mainly how to distribute my databases :)
Geofabrik provides for Russia federal districts only. This requires volume... gis-lab provides most of 89 regions separately.
1. rinigus, excuse me for *8*,but */8 exists. I see *8* for alternative method.
OK, I see. */8 exists due to the way libpostal normalizes the strings. Its a side effect of allowing to search for streets with the names like Dr. John Smith by using "Smith" in the search string. How to get rid of */8 in response in a general way, I don't know. Since the database is in an open form (SQLite has several tools to work with it) and SQL queries are all in geocoder source code, anyone is welcome to analyse the alternative search commands. Just note that w need to use database indexes to make it fast - otherwise anyone with a bigger country would complain.
Geofabrik provides for Russia federal districts only. This requires volume... gis-lab provides most of 89 regions separately.
I see. Note that I am not distributing the libosmscout maps, just the databases required for geocoding. Libpostal part would be hard to reduce even further by taking smaller part of a country without chopping languages part that I don't have data to do (and probably time either).
SQLite geocoding database would reduce, indeed, if I take these 89 regions. Full Russia is now 720MB. I guess that by dividing it into 9 federal districts should make it reasonable. Don't forget that we have to have ~700MB libpostal dataset already anyway to make it work with the supported languages.
At present, I'll try mainly to work on organizing these databases into a bit of an order first and try to make it simply downloadable. Then I could look into chopping the planet into additional subregions. Surely that would be needed for Germany, Russia, and few others.
@rinigus
Thanks for huge work
I have just published an updated version (0.6.2) which includes German translation - thank you @monkeyisland!
To simplify the work on translations, a project has been opened at https://www.transifex.com/rinigus/osm-scout-server/ . @monkeyisland has been of great help and we tested it on his translation.
In addition, when you ask for a route somewhere by giving the target by name, the calculated route should display the search result under the "Target" message.
Now back to developing it further...
monkeyisland
2017-02-16, 15:37
I dont know if the issue comes from poor maps or osm scout server.
Now i can choose maps from osm. But when i try navigate with osm scout in poor maps osm scoit server will crash.
Offline search is also not possible.
I dont know if the issue comes from poor maps or osm scout server.
Now i can choose maps from osm. But when i try navigate with osm scout in poor maps osm scoit server will crash.
Offline search is also not possible.
If OSM Scout Server crashes - its a problem with the server, not Poor Maps.
I suspect that the crash is related to the change in the map format that was not tracked. See https://github.com/rinigus/osmscout-server/issues/61 . I have not updated libosmscout in the latest builds and the bug is still present in the current server builds.
To check if you are hit with the same bug, please reimport the maps using importers from https://github.com/rinigus/libosmscout/releases/tag/0.0.git.20170126 . Let me know if it helped.
monkeyisland
2017-03-02, 15:56
Hi there i tested to import a little map i take berlin-latest from geofabrik.
OSM cant open the map it shows an error.
so i dont have an x64 systems i compiled the importer tool by myself for powerpc.
the compiling works great and the map would be converted without errors but osmscout cant open this test map...:(
Hi there i tested to import a little map i take berlin-latest from geofabrik.
OSM cant open the map it shows an error.
so i dont have an x64 systems i compiled the importer tool by myself for powerpc.
the compiling works great and the map would be converted without errors but osmscout cant open this test map...:(
Hi. What error you get? Can you share your generated map please?
Hi there i tested to import a little map i take berlin-latest from geofabrik.
OSM cant open the map it shows an error.
so i dont have an x64 systems i compiled the importer tool by myself for powerpc.
the compiling works great and the map would be converted without errors but osmscout cant open this test map...:(
The last version of OSM Scout Server was released against buggy (in terms of version control) libosmscout. Bug was found by @Karry and got fixed.
I would suggest to wait few days, since I am preparing a new release that would resolve this issue as well. Its progressing very nicely with only few issues to get resolved (milestone 0.7 at project's github page).
If you cannot wait, please use https://github.com/rinigus/libosmscout/releases/tag/0.0.git.20170126 for compilation of the importer. If you have used some other version, you would either get crashes or refusal to load the map.
cheers,
rinigus
karlos devel
2017-03-02, 18:16
Hello @rinigus
i updated all my maps with libosmscout-importer.Linux.x86_64.tar.gz > 0.0.git.20170221 it is not work with osmscoutserver MSSG:ERROR: 13:12:47 It is not possible to open the database. work ok with karry-osmscout-sailfish
Hello @rinigus
i updated all my maps with libosmscout-importer.Linux.x86_64.tar.gz > 0.0.git.20170221 it is not work with osmscoutserver MSSG:ERROR: 13:12:47 It is not possible to open the database. work ok with karry-osmscout-sailfish
Hi @karlos devel:
please use only importer based on 0.0.git.20170126 for 0.6.x server.
@karry's osmscout is much smarter than a server in terms of version checks and I am not surprised it works over with that application. Its also newer and supports the last version format as produced by the importer based on 0.0.git.20170221 .
I am using libosmscout release 0.0.git.20170221 for 0.7.x development, I probably made a mistake by tagging it too early before release of the new version of the server. Just wanted to be sure that I don't forget to tag it this time.
This is announcement for a new release: 0.7.0
This release introduces Map Manager and incorporates maps distribution via CDN. This is a major change in approach and should make the use of the server simple. In particular, use of geocoder-nlp should be easy and all dependencies are resolved automatically. I hope that you would enjoy using it!
Starting from this release, maps are supposed to be used only through download. This is to ensure that its simple to update and cleanup free space. It should also help with the incompatibilities between the versions. So, even if you had maps prepared for your devices, you would have to download new maps.
Distribution model is as follows. I have a file in the project github source that has a list of provided countries/regions/features and URL to CDN base. Location of the file will stay and you could always update the provided list via GUI/Map Manager. Then you would have to decide which countries do you want (through subscription) and whether you want to store files needed for geocoder-nlp. After that, start to download from CDN.
When you start new server, you would be asked to allocate a separate folder that OSM Scout Server would manage. Note that you are expected to let it perform all operations within that folder (or follow its lead when shown the files that it doesn't use anymore, for example). I am not allocating the folder automatically to give you more control over proceedings (~nemo, SD card, ...). If its too confusing, I can allocate .local/shared/harbour-osmscout-server for it. But that may lead to filling up of internal storage for some users.
The files downloaded by server are all tracked in a small database, so it knows the versions and timestamps for each of them. If you download files using @Karry's OSM Scout into the same folder, they would be unknown by the server. We would have to discuss with @Karry how to proceed and ensure that compatible maps can be used in the both applications, but that may take some time to figure it out.
With geocoder-nlp easily accessible, don't forget to specify which languages do you want to use for parsing addresses. Otherwise, libpostal may eat lots of RAM!
I will gather your questions and probably add also small Q&A into the first post of the thread, if needed.
I will work on making maps distribution simpler. For that, I plan to use @MartinK scripts to chop the planet into regions and import all polygons into github. It would take some time to make it all work, but should allow us to extend provided regions through github development process.
At present, I added a small credit for CDN. I do wonder for how long would it last, probably it would cover the first TB of downloads.
Now its a great time to propose your server space for distribution of maps. Or, propose a model where we could collect money for distribution. In the money collecting case, I would prefer to have it done via some organization, to stay away from all possible issues with taxes.
Its been lots of work on making it possible and I might have forgotten to mention something important.
Enjoy!
Top menu "Map Manager" and Settings/Maps storage "Select" is both unavailable for me.
This is output when starting Settings...
,---
| SailfishOS 2.1.0.9 (Iijoki) (armv7hl)
'---
[nemo@Sailfish ~]$ harbour-osmscout-server
[D] unknown:0 - Using Wayland-EGL
[W] unknown:38 - file:///usr/lib/qt5/qml/Sailfish/Silica/private/Util.js:38: TypeError: Cannot read property 'parent' of null
[W] unknown:194 - file:///usr/lib/qt5/qml/Sailfish/Silica/DialogHeader.qml:194: TypeError: Cannot read property 'backIndicatorDown' of null
[W] unknown:189 - file:///usr/lib/qt5/qml/Sailfish/Silica/DialogHeader.qml:189: TypeError: Cannot read property 'backIndicatorDown' of null
[W] unknown:247 - file:///usr/lib/qt5/qml/Sailfish/Silica/DialogHeader.qml:247: TypeError: Cannot read property 'forwardIndicatorDown' of null
[W] unknown:242 - file:///usr/lib/qt5/qml/Sailfish/Silica/DialogHeader.qml:242: TypeError: Cannot read property 'forwardIndicatorDown' of null
Top menu "Map Manager" and Settings/Maps storage "Select" is both unavailable for me.
This is output when starting Settings...
,---
| SailfishOS 2.1.0.9 (Iijoki) (armv7hl)
'---
[nemo@Sailfish ~]$ harbour-osmscout-server
[D] unknown:0 - Using Wayland-EGL
[W] unknown:38 - file:///usr/lib/qt5/qml/Sailfish/Silica/private/Util.js:38: TypeError: Cannot read property 'parent' of null
[W] unknown:194 - file:///usr/lib/qt5/qml/Sailfish/Silica/DialogHeader.qml:194: TypeError: Cannot read property 'backIndicatorDown' of null
[W] unknown:189 - file:///usr/lib/qt5/qml/Sailfish/Silica/DialogHeader.qml:189: TypeError: Cannot read property 'backIndicatorDown' of null
[W] unknown:247 - file:///usr/lib/qt5/qml/Sailfish/Silica/DialogHeader.qml:247: TypeError: Cannot read property 'forwardIndicatorDown' of null
[W] unknown:242 - file:///usr/lib/qt5/qml/Sailfish/Silica/DialogHeader.qml:242: TypeError: Cannot read property 'forwardIndicatorDown' of null
Map Manager should be disabled until Map storage folder is selected. So, that's OK.
What is very strange is that you cannot select Maps storage under Settings. Frankly, I cannot reproduce it. Would you mind to move away (don't delete - it maybe its needed in future for fixing this bug) OSM Scout Server settings and then try without your old settings?
mv .config/harbour-osmscout-server/harbour-osmscout-server.conf /home/nemo/
As for the messages, it doesn't seem that anything is unusual, at least at the brief look.
Top menu "Map Manager" and Settings/Maps storage "Select" is both unavailable for me.
This is output when starting Settings...
I rushed with the response, I think its much easier.
To "Select" some folder, you would have to go in FileManager widget into a parent folder, make a long press on the folder that you want to choose and then the "Select" will appear as a context menu.
For example, let's say we want to choose /home/nemo/Maps:
* You navigate to /home/nemo
* Long press on "Maps"
* Context menu "Select" appears, choose that
* Accept in Settings
* Maps storage is selected, Map Manager would be available on the main window.
Let me know if it helped.
seiichiro0185
2017-03-05, 16:34
Just tried the new version, selecting the storage folder and subscribing to a map works fine, but the download stops with an error after about 200-300 MB (I'm trying to download the germany map with geocoder data). Error is always "Connection Closed". My internet connection is stable without any interruption during the download.
Any idea what could be the problem?
I rushed with the response, I think its much easier.
To "Select" some folder...
Ahhh... got it! This is what I saw, and found very confusing, or maybe I'm just stupid. :confused:
Thanks!
.
At present, I added a small credit for CDN. I do wonder for how long would it last, probably it would cover the first TB of downloads.
Now its a great time to propose your server space for distribution of maps.
Sure! I've talked a bit more with the NLP lab (https://nlp.fi.muni.cz/web3/en/NLPCentre) people and they are indeed fine with both hosting and generating data for OSM Scout Server as long as the data is hosted as static files that are batch-updated once in a while (every few days, etc.). As that condition is satisfied it should be possible to host at least ~100 GB, without transferred bandwidth limitations.
So now I need to know how can I "join" the OSM Scout Server data CDN with these resources. :) Are there some mirroring scripts I could add to cron or something similar ?
I can also help generate the data sets if needed, on one of the NLP compute machines, a fairly beefy system (220 GB RAM & 48 CPU core) running Fedora 24 with some 200-300 GB of local storage that could be used for the data generation run.
Just tried the new version, selecting the storage folder and subscribing to a map works fine, but the download stops with an error after about 200-300 MB (I'm trying to download the germany map with geocoder data). Error is always "Connection Closed". My internet connection is stable without any interruption during the download.
Any idea what could be the problem?
So far, the biggest map I have tried is Spain and that downloaded without any issues. However, in my place, it wasn't happy about mobile net. That's not surprising since we have a place/operator combination that resets connections quite frequently. Worked fine on WiFi.
One suspicion would be that maybe phone enters a deep sleep while downloading leading to "Connection closed". Namely, I have not used any keepalive and the both of my devices (N4 and OPX) worked just fine to my surprise. So, I didn't even look into keepalive locks. Maybe its not working for you.
Try to put your device on charging and download then.
On the technical part: many of the databases are split into several parts. As long as you see INFO messages on the main screen with "File downloaded" you are progressing even if after restart of the download the apparent size remains the same. So, you should be able to download by these batches... However if you are stuck on some bigger file (larger than 200-300 MB) then it will be trying to get the same file again.
Let me know if putting on charger helped. If that''s a problem, I'll add keepalive lock and we should be OK.
Ahhh... got it! This is what I saw, and found very confusing, or maybe I'm just stupid. :confused:
Thanks!
.
What you were selecting is actually a parent folder (".."). That's disabled to prevent users selecting /home/nemo instead of /home/nemo/Maps and later "cleaning up" their whole home dir :) [you have to authorize deleting, unless you do that will not happen]
Sure! I've talked a bit more with the NLP lab (https://nlp.fi.muni.cz/web3/en/NLPCentre) people and they are indeed fine with both hosting and generating data for OSM Scout Server as long as the data is hosted as static files that are batch-updated once in a while (every few days, etc.). As that condition is satisfied it should be possible to host at least ~100 GB, without transferred bandwidth limitations.
So now I need to know how can I "join" the OSM Scout Server data CDN with these resources. :) Are there some mirroring scripts I could add to cron or something similar ?
I can also help generate the data sets if needed, on one of the NLP compute machines, a fairly beefy system (220 GB RAM & 48 CPU core) running Fedora 24 with some 200-300 GB of local storage that could be used for the data generation run.
Brilliant! I'll think how to do that and contact a bit later tonight
seiichiro0185
2017-03-05, 18:15
[....]
Let me know if putting on charger helped. If that''s a problem, I'll add keepalive lock and we should be OK.
I tried with having the device charging(and device was on wifi for all tries so far), but the download stopped with the same error again (This time at about 400MB). Unfortunately the areas.dat for germany is bigger than that, so it doesn't get further than trying to download this file with every new start. Btw downloading the map for germany with osmscout worked fine last time I tried (same wifi, device not charging).
I tried with having the device charging(and device was on wifi for all tries so far), but the download stopped with the same error again (This time at about 400MB). Unfortunately the areas.dat for germany is bigger than that, so it doesn't get further than trying to download this file with every new start. Btw downloading the map for germany with osmscout worked fine last time I tried (same wifi, device not charging).
Not that it helps you - it worked for me when I downloaded Germany. I presume you mean the whole country, 4GB+ unpacked.
I presume you do have enough space to download full Germany.
Since you are on the charger already, I don't have to look into keepalive yet - it seems that this is not of a source of error.
Frankly, I don't know what to suggest. Its most probably something on CDN provider and I don't know how to debug it. I use S3-compatible service and, as far as I have seen in the documents, they do talk about large files and chopping them to pieces to ensure better redundancy. But how to use it, I have not really looked.
Maybe your connection is too fast/too slow and then some error on CDN provider backend is triggered?
In the case of areas.dat, you are downloading about 700MB bzip2'ed file. So, you are getting closer. Would you mind to try some part of Germany and later try to get full set again a bit later?
Let me know how it goes. We can always complain and ask for advice from the provider.
PS: OSM Scout uses @Karry's own server and has also different download implementation.
Sure! I've talked a bit more with the NLP lab (https://nlp.fi.muni.cz/web3/en/NLPCentre) people and they are indeed fine with both hosting and generating data for OSM Scout Server as long as the data is hosted as static files that are batch-updated once in a while (every few days, etc.). As that condition is satisfied it should be possible to host at least ~100 GB, without transferred bandwidth limitations.
So now I need to know how can I "join" the OSM Scout Server data CDN with these resources. :) Are there some mirroring scripts I could add to cron or something similar ?
I can also help generate the data sets if needed, on one of the NLP compute machines, a fairly beefy system (220 GB RAM & 48 CPU core) running Fedora 24 with some 200-300 GB of local storage that could be used for the data generation run.
Now a proper reply: Thank you very much for an offer! It would be of great help and I hope that this service would not be abused. We have now some amount of time to figure it out while my current CDN is up.
As it is, I can process the planet reasonably fast on the system available to me. Its not as beefy, but made for serious calculations as well :) . Maybe later we could move data processing as well, but there is no need now for data processing. Unless they are interested on working on address parsing and NLP approach behind libpostal. Then we can discuss it separately.
At present, I use ~40GB for all datasets. I presume that during dataset version changes (version incompatibilities) we would need double of that space for a week or two. If the other backends would be added, the data requirement may increase, but let's see about that.
Datasets are served statically. In essence, we have a tree with different backends (osmscout, postal, geocoder-nlp) and a JSON file that describes them. By changing a line at https://github.com/rinigus/osmscout-server/blob/master/scripts/import/provided/countries_provided.json#L7112 , I could move all download requests to a new location (when you hit "Update list" in Map Manager you download that file by OSM Scout Server from github).
I presume we would keep conditions human and update once in a month?
So, it would boil down to the way we move the data between the servers. Or if its too complicated, we would just need cron jobs running. But that we could just discuss via PM/email. I'll get in touch with you tomorrow, maybe rather late though.
Feathers McGraw
2017-03-05, 22:48
Just trying out the new version, all the dialogs seem to work as they should.
Might be nice to suggest some locations for data with the ability to select somewhere arbitrary as well - most people probably won't care what the folder is called... you could suggest ~/OSMServer or sdcard/OSMServer and create the chosen folder too, which would prevent the user having to leave the app and come back.
I also got the connection closed error downloading the map data for Great Britain (on Fairphone 2, if that matters). I'll try again whilst charging and see if that helps.
Feathers McGraw
2017-03-05, 22:50
Also, in the download counter what do the D and W in brackets stand for?
seiichiro0185
2017-03-06, 07:07
Not that it helps you - it worked for me when I downloaded Germany. I presume you mean the whole country, 4GB+ unpacked.
I presume you do have enough space to download full Germany.
[...]
Let me know how it goes. We can always complain and ask for advice from the provider.
PS: OSM Scout uses @Karry's own server and has also different download implementation.
I seems there is some kind of incompatiblility between my Jolla C, my WIFI at home, my ISP, the CDN provider and the Download implementation in osmscout-server (maybe not all of them, but no idea what the real problem is here).
Downloading maps for parts of Germany does work for some, but not for the bigger ones. Several tries for the whole country would always fail at some point downloading areas.dat.
Today I tried the download with the WIFI at work, and it worked perfectly. No interruptions, no errors, just downloaded the whole map in one try..
I have no idea what the problem might be, since every other Download so far worked without any Issues on my home WIFI with my Jolla C.
Just trying out the new version, all the dialogs seem to work as they should.
Might be nice to suggest some locations for data with the ability to select somewhere arbitrary as well - most people probably won't care what the folder is called... you could suggest ~/OSMServer or sdcard/OSMServer and create the chosen folder too, which would prevent the user having to leave the app and come back.
Great idea, I opened an issue at github regarding it and would implement it when I get there
I also got the connection closed error downloading the map data for Great Britain (on Fairphone 2, if that matters). I'll try again whilst charging and see if that helps.
OK, it looks that I need to make a more robust downloader. I think I have an idea on how to do it, it may just take some time. Let me know if your next tries worked and if the charger helped.
Issues to address it have been opened as well.
Also, in the download counter what do the D and W in brackets stand for?
When I was implementing it, I was wondering if someone would ask.
The downloads are organized as follows: you download bz2 archive (the expected remaining part is shown as (D) in MBs) and write it to the disk unpacked (shown as W).
@seiichiro0185: I am glad it worked in other place. I'll work on implementation to make it possible to download it at your home as well.
Feathers McGraw
2017-03-06, 13:15
I retried the download on charger, managed go download the map fine but the other data failed (several times). Will have another go this evening.
I retried the download on charger, managed go download the map fine but the other data failed (several times). Will have another go this evening.
I'll work on fixing those broken downloads as the first priority. @Karry already gave a great suggestion on an aspect of its implementation, now just have to work on it.
@Feathers McGraw, Would you mind to test downloading with RPM provided at
https://mega.nz/#F!cdcEDbpb!i4A1GwKM_TXeRYBaahYgAg
It should resume downloads for few times (you would see RANGE messages in terminal if you start it over there). After testing and (hopefully fully) downloading, remove this RPM and install the one as in OpenRepos. This RPM has debug messages in it and not compiled in release mode.
If its all fine, I'll hook messages into GUI, merge translations that are ready and release 0.7.1. If it fails, then would be great to know how (as much as you know what to tell)
Feathers McGraw
2017-03-07, 11:05
@Feathers McGraw, Would you mind to test downloading with RPM provided at
https://mega.nz/#F!cdcEDbpb!i4A1GwKM_TXeRYBaahYgAg
It should resume downloads for few times (you would see RANGE messages in terminal if you start it over there). After testing and (hopefully fully) downloading, remove this RPM and install the one as in OpenRepos. This RPM has debug messages in it and not compiled in release mode.
If its all fine, I'll hook messages into GUI, merge translations that are ready and release 0.7.1. If it fails, then would be great to know how (as much as you know what to tell)
I will give it a try as soon as I can, but I'm moving house tomorrow and unfortunately I won't have internet access for a week afterwards! Any other volunteers available this week?
I will give it a try as soon as I can, but I'm moving house tomorrow and unfortunately I won't have internet access for a week afterwards! Any other volunteers available this week?
Thanks for giving a timeline. I'll continue with hooking it into GUI and would release based on my testing unless some negative feedback would come in.
My tests so far have been promising, but the conditions could be different in other places.
@Feathers McGraw, Would you mind to test downloading with RPM provided at
https://mega.nz/#F!cdcEDbpb!i4A1GwKM_TXeRYBaahYgAg
It should resume downloads for few times (you would see RANGE messages in terminal if you start it over there). After testing and (hopefully fully) downloading, remove this RPM and install the one as in OpenRepos. This RPM has debug messages in it and not compiled in release mode.
If its all fine, I'll hook messages into GUI, merge translations that are ready and release 0.7.1. If it fails, then would be great to know how (as much as you know what to tell)
@rinigus I try this version. I don't know how to make better maps for Russia. For ex., I have interest for two regions(from 85), but for this I should to download two districts(from 8)... It's difference in space.
@rinigus I try this version. I don't know how to make better maps for Russia. For ex., I have interest for two regions(from 85), but for this I should to download two districts(from 8)... It's difference in space.
@XOleg, did you have problems with downloads using 0.7.0?
I am planning to approach planet chopping via polygons and when (if) I manage to set it up, it would be possible to add our own polygons into hierarchy. See for example POLY files at geofabric. As long as the number of polygons is not too large, I would be happy to add more with the preferences of early adopters high in priority.
I have in my mind something like @MartinK has done for modRana: https://github.com/M4rtinK/modrana-data-repository/tree/master/tools/polys . Except, in addition to poly files, I plan to have few others to describe relationship between libosmscout, libpostal, and geocoder-nlp datasets.
As soon as initial import is transferred to a new system, I will be able to get additional chopping as well. So, you would just have to wait a bit...
@XOleg, did you have problems with downloads using 0.7.0?
I have in my mind something like @MartinK has done for modRana: https://github.com/M4rtinK/modrana-data-repository/tree/master/tools/polys . Except, in addition to poly files, I plan to have few others to describe relationship between libosmscout, libpostal, and geocoder-nlp datasets.
@rinigus I don't tried :-)
the download(Ural+Volga) by Wi-Fi is finished(from my last message).
One question. maybe it's possible to combine efforts of @rinigus, @MartinK, @karry to create one good maps/navigation app? It's not simple(to combine), but...
karlos devel
2017-03-07, 20:02
THNKS @rinigus For its genuine continuous development of this.
- would like to know if after that version 0.7.0 it not enable to update the maps manual anymore?
- would be a good idea to help a maps navigation app among all of us = )
@rinigus I don't tried :-)
the download(Ural+Volga) by Wi-Fi is finished(from my last message).
Good! I will proceed with the update - just have to ensure that the timeout for connection would work and then get a new release out.
One question. maybe it's possible to combine efforts of @rinigus, @MartinK, @karry to create one good maps/navigation app? It's not simple(to combine), but...
In reality, these efforts are combined already. Together with @Karry we work on the offline part of the maps/navigation app. We use the same backend linbrary - libosmscout - and improve it together with several others. Recently, we worked on different aspects with me focusing on search and @Karry just finished a great work on rendering speedup. Corresponding changelog is at https://github.com/Framstag/libosmscout/commits/master .
I would still have to work on search and make it faster and, hopefully, occupy less storage space. That would be my focus as soon as we sort out the distribution of the maps, with @MartinK offering a lot of help. Before that, @Karry actually implemented and showed how such distribution can be done for his OSM Scout.
Notice that we also incorporate developers that are not working/using Sailfish. Main developers of libosmscout and libpostal are working on Linux, so there are even more people involved.
While @Karry works on GUI of his application, OSM Scout Server relies on Poor Maps (@otsaloma) and/or modRana (@MartinK). The whole idea behind working on the server was to combine the efforts of different developers and bring parts developed for offline maps to the map applications that we had already. I think all in all there is a fast progress on the front of maps/navigation [wanted to continue using a word soon, but it should be probably banned in SFOS universe].
THNKS @rinigus For its genuine continuous development of this.
- would like to know if after that version 0.7.0 it not enable to update the maps manual anymore?
- would be a good idea to help a maps navigation app among all of us = )
Yes, my idea is that, after 0.7, having self imported maps would be relatively hard (read impractical). This is due to difficulties on keeping dependencies between libosmscout/libpostal/geocoder-nlp and whatever fancy backend we would throw into the mix. I would suggest to try without self-imported maps for few releases and see if it is OK. If users would not like it after that, I would have to reconsider and make a simplish way to use custom imported maps.
Navigation app - help @MartinK and @otsaloma. OSM Scout Server does provide routes and Poor Maps / modRana can already show them. At present, Poor Maps is the best for navigation with its full navigation GUI during a drive. For me, the main missing part is rerouting in Poor Maps. After that, voice guidance. As far as I understood, rerouting code is already somewhere in modRana, but its not exposed yet. So, we have to be patient and help the devs as much as we could.
seiichiro0185
2017-03-08, 05:06
@Feathers McGraw, Would you mind to test downloading with RPM provided at
https://mega.nz/#F!cdcEDbpb!i4A1GwKM_TXeRYBaahYgAg
It should resume downloads for few times (you would see RANGE messages in terminal if you start it over there). After testing and (hopefully fully) downloading, remove this RPM and install the one as in OpenRepos. This RPM has debug messages in it and not compiled in release mode.
If its all fine, I'll hook messages into GUI, merge translations that are ready and release 0.7.1. If it fails, then would be great to know how (as much as you know what to tell)
Just a quick feedback from my side: Using the linked test version I could succesfully download the Germany Map while being on my home wifi, so the improved download code seems to help for my case :)
I have just released 0.7.1.
This time, the main changes are updated translations. I would like to thank all the translators for the great job on updating the messages!
In addition, the downloading has been made more robust and should be able to recover from smaller hiccups. During my testing I've got once "Connection closed" but that might have been since I was in the area without coverage.
Please let me know if you have more issues with downloads. I hope that those as we had earlier with some of you in Germany/UK are now resolved.
The new code still does not have keepalive incorporated into it. I would prefer to keep the server without keepalive if possible. So far all my downloads have worked fine without being on charger suggesting that in this case keepalive is not required. I guess if we will not notice any major issues with downloads, I'll keep it as it is.
This time, the main changes are updated translations. I would like to thank all the translators for the great job on updating the messages!I will try to help with French, which I just requested on transifex. If you have time to add it to the list, I'll try to work out most of it this weekend.
I'll also take a look on the Linux-side code (not Sailfish for now), to see if I can help there or not.
I will try to help with French, which I just requested on transifex. If you have time to add it to the list, I'll try to work out most of it this weekend.
I'll also take a look on the Linux-side code (not Sailfish for now), to see if I can help there or not.
Thanks for letting me know re transifex request - for some reason they notify me by email a day or two later. All accepted and should be ready.
Re Linux: Any help would be great! Few questions regarding it:
* Do you use server in Linux? If so, then I have to expose map downloading via command line options. Otherwise, its impossible to use it right now.
* On Linux side of things, there is a lot to be done on the backend libraries - libosmscout and geocoder. Its probably easier to start work on libosmscout since its a bigger community with different OS support and more help on the lists. geocoder though is a small code base, but it maybe not as well documented.
Additional note for users: @ade has been hit by a bug leading to OOM-killer on J1. Current suspicion is that the writing and uncompressing operations are slower that network download leading to accumulation of data in the buffers. Bug was exposed during libpostal dataset download which does contain a huge file. Issue was opened at https://github.com/rinigus/osmscout-server/issues/89 and I hope to resolve it within few days.
Thanks for letting me know re transifex request - for some reason they notify me by email a day or two later. All accepted and should be ready.Thanks. I have already done a bunch of the translations. 100 remaining that I will try to complete before Sunday.
* Do you use server in Linux? If so, then I have to expose map downloading via command line options. Otherwise, its impossible to use it right now.I intend to use it on Linux, yes. I have also installed it on my Jolla through you packages on OpenRepos, to have more context while doing the translations, and as a reference.
So far I am trying to see what the prerequisites are to build the different parts (libosmscout, osmscout-server, ...), and have a lot of thins to read to understand better what every part is doing. But I will try to finish the translation before spending more time on that front.
* On Linux side of things, there is a lot to be done on the backend libraries - libosmscout and geocoder. Its probably easier to start work on libosmscout since its a bigger community with different OS support and more help on the lists. geocoder though is a small code base, but it maybe not as well documented.Ok. I assume the reference website is http://libosmscout.sourceforge.net/ with the mailing lists at https://sourceforge.net/p/libosmscout/mailman/libosmscout-development/ ?
Are there other channels I need to read ?
Thanks for your work and help by the way.
Thanks. I have already done a bunch of the translations. 100 remaining that I will try to complete before Sunday.
I intend to use it on Linux, yes. I have also installed it on my Jolla through you packages on OpenRepos, to have more context while doing the translations, and as a reference.
So far I am trying to see what the prerequisites are to build the different parts (libosmscout, osmscout-server, ...), and have a lot of thins to read to understand better what every part is doing. But I will try to finish the translation before spending more time on that front.
Ok. I assume the reference website is http://libosmscout.sourceforge.net/ with the mailing lists at https://sourceforge.net/p/libosmscout/mailman/libosmscout-development/ ?
Are there other channels I need to read ?
Thanks for your work and help by the way.
Assuming that we'll figure it out this weekend with @ade what's going on with memory during download, translations by the end of weekend would fit perfectly with the next point release.
Using the server on Linux - great! I used it earlier in development with modRana, but haven't tried lately. To make it possible, I'll have to fix https://github.com/rinigus/osmscout-server/issues/90 , but that shouldn't be too complicated. Let me fix OOM-killer bug first and finish small scripts to move downloads to @MartinK server. Then I should be able to get issue #90 resolved in a night or two.
Dependencies on Linux are actually much simpler. Assuming that you do have RAM (more than 2-3GB), you can skip all these country-specific libpostal datasets. You would need to install libpostal as at https://github.com/openvenues/libpostal , libosmscout, and sqlite.
To use with the server downloadable datasets, please use https://github.com/rinigus/libosmscout Sailfish branch, latest release. For fixing / development of libosmscout, please use https://github.com/Framstag/libosmscout. Sorry for a mess, but that way I can keep versions consistent (upstream does not provide releases). libosmscout can be also used without the server using their own QML/Qt application which is available at github. Note that the work that you would do on backend features of libosmscout (map drawing, routing, and several others) would be pulled into the server and OSM Scout by @Karry as well. So, you would have to see what annoys you most and find the easiest way to work on it. Ask/propose project(s) if you like. Note that the mailing list is reasonably active and we all try to help each other as much as we could.
Geocoder-nlp is available as a submodule for osmscout-server. So, that's easy dependency.
In Linux, all configuration is done via INI configuration file, under .config/osmscout-server . At present, translations are not exposed to Linux to avoid double work by translators. We should probably just modify build scripts to copy harbour-osmscout-*ts files to osmscout-*.ts files.
In Linux, if you run server on a proper server without GUI, you may prefer using Cairo drawing backend of libosmscout. For that, just uncomment https://github.com/rinigus/osmscout-server/blob/master/osmscout-server_console.pro#L20 and comment out a line above. Its possible to use Qt for drawing on a server hardware as well, but you would have struggle a bit with some environment vars, if I remember correctly.
I hope I remembered all what is relevant to this case. If you have any trouble, just ask.
Quick note: we have just resolved issue of OOM-killer with @ade. I will publish a new release with this fix, unless someone will see degradation in their download stability. Note that downloads could be slower after it, but they should not be degraded in any major way.
Test build with the fix enabled:
https://mega.nz/#F!IQ0nmYZJ!RQgn0Lo13NwmtixZ-2am7w
I'll wait a day and maybe there will be new translations to incorporate into it as well.
@ade: thank you for fast testing and reporting a bug!
In reality, these efforts are combined already.
wonderful! :-)
@rinigus, I'll try to translate to Russian if necessary. I have an account for transifex but I don't understand how change a team.
Thanks in advance for help.
wonderful! :-)
@rinigus, I'll try to translate to Russian if necessary. I have an account for transifex but I don't understand how change a team.
Thanks in advance for help.
Its not necessary - but helpful, I presume. Assuming that you are XOleg @ transifex, I just invited you.
For me, this transifex is a bit mystery. But I am trying to navigate there with minimal reading and clicking the links rather randomly, I must confess. So far, it seems to be working.
Its not necessary - but helpful, I presume. I just invited you.
So far, it seems to be working.
Yes. it's working :-)
Quick note: we have just resolved issue of OOM-killer with @ade. I will publish a new release with this fix, unless someone will see degradation in their download stability. Note that downloads could be slower after it, but they should not be degraded in any major way.
Test build with the fix enabled:
https://mega.nz/#F!IQ0nmYZJ!RQgn0Lo13NwmtixZ-2am7w
I'll wait a day and maybe there will be new translations to incorporate into it as well.
"Failed to download transliteration.dat.
Timeout"
"Failed to download transliteration.dat.
Timeout"
Try again. If it persists then that's a problem. If it resumes and is able to download it on further tries, I think its OK. So far, the fixed problems were for cases where it was just impossible to download a huge file since connection was always dropped in the middle (Germany & UK) or server was crashing with OOM. Let's see if you could download on retry.
Edit: I can also increase timeout to 5 minutes and see if it helps. XOleg, should I try it? Do you get plenty of timeouts? At present, its 1 minute
Try again. If it persists then that's a problem. If it resumes and is able to download it on further tries, I think its OK. So far, the fixed problems were for cases where it was just impossible to download a huge file since connection was always dropped in the middle (Germany & UK) or server was crashing with OOM. Let's see if you could download on retry.
Edit: I can also increase timeout to 5 minutes and see if it helps. XOleg, should I try it? Do you get plenty of timeouts? At present, its 1 minute
I don't know :-) Phone works himself. I restart 1h ago, phone download maps(second district) for this moment.
P.S. I translate, but this not very quickly. Maps it's not my theme.
I don't know :-) Phone works himself. I restart 1h ago, phone download maps(second district) for this moment.
P.S. I translate, but this not very quickly. Maps it's not my theme.
I'll look into the code again, maybe I missed something. As for translation - take your time. We can do as many point releases as needed. I don't think the next series 0.8.x would change strings too much, I plan to work later on backend features.
@rinigus Download is terminated without errors. 1h20' max after restart
@rinigus Download is terminated without errors. 1h20' max after restart
Does it mean it was OK or program terminated?
Does it mean it was OK or program terminated?
Download is OK. Server continues to work.
Assuming that we'll figure it out this weekend with @ade what's going on with memory during download, translations by the end of weekend would fit perfectly with the next point release.Done for the French translation.
It is far from perfect, and I'm not happy with every translation, but it can be released as a first try.
If another French speaking fellow wants to take a look, this webpage can be really useful for the terms related to roads names : http://wiki.openstreetmap.org/wiki/FR:France_roads_tagging
There are lots of details in roads characterizations terms that are not that easy to understand at first sight without this reference.
Using the server on Linux - great! I used it earlier in development with modRana, but haven't tried lately. To make it possible, I'll have to fix https://github.com/rinigus/osmscout-server/issues/90 , but that shouldn't be too complicated. There is no hurry in this.
Done for the French translation.
It is far from perfect, and I'm not happy with every translation, but it can be released as a first try.
If another French speaking fellow wants to take a look, this webpage can be really useful for the terms related to roads names : http://wiki.openstreetmap.org/wiki/FR:France_roads_tagging
There are lots of details in roads characterizations terms that are not that easy to understand at first sight without this reference.
There is no hurry in this.
Thank you for the translation, its a big work!
I have just published a new point release: 0.7.2
In addition to fixing a nasty bug imposed by fast download and slow writing to the media reported by @ade, this release introduces 4 more languages added by translators. While French and Russian are completely translated, there is also a great progress with Polish and Norwegian. So, I would like to thank the translators that step up to support these new languages in our list:
- @Sagittarii (@Zeta at TMO): French
- Oleg Artobolevsky @XOleg: Russian
- A @atlochowski: Polish
- Peer-Atle Motland @Pam: Norwegian
If you see something strange going on with the downloads (or some other functionality), let me know. I hope most of the issues with the downloads are fixed now, but there could be still some corner case lurking around.
We are working with @MartinK behind the scenes to get the distribution moved over to his server. Also I have to rearrange the scripts that are generating the maps, so we could easily add new regions and make updates automatic.
@rinigus I agree with @Zeta: "There are lots of details in roads characterizations terms that are not that easy to understand" and
"It is far from perfect, and I'm not happy with every translation, but it can be released as a first try."
@Zeta, thank for link :-)
@rinigus I agree with @Zeta: "There are lots of details in roads characterizations terms that are not that easy to understand" and
"It is far from perfect, and I'm not happy with every translation, but it can be released as a first try."
@Zeta, thank for link :-)
Yes, its not very simple, sure! The link is added to Translation README at https://github.com/rinigus/osmscout-server/blob/master/translations/README.md and available via "Help pages" in Transifex project page
To all users:
Together with @MartinK, we have just finished syncing the whole maps dataset into its new home - data.modrana.org. To use this new location, please open Map Manager and press
"Update lists"
button. This would load an updated list pointing to the new location.
In few days I am planning to stop the current distribution via CDN that I used for launching the service assuming that we don't run into any technical difficulties.
I think a former acknowledgment is appropriate, so you would all know who to thank for the service: Maps are hosted by Natural Language Processing Centre (https://nlp.fi.muni.cz/en/ , Faculty of Informatics, Masaryk University, Brno, Czech Republic) through modRana (http://modrana.org)
I'll work on automation of the updates and we can then later look into adding few other regions as well
Together with @MartinK, we have just finished syncing the whole maps dataset into its new home - data.modrana.org. To use this new location, please open Map Manager and press
"Update lists"
Seem to work perfect. I just downloaded two more countries.
@rinigus and @MartinK, many thanks for great job.
P.S. food for thought :-) maybe will be possible to use http://gis-lab.info/projects/osm_dump/ as source of maps for Russia? regions, not districts.
@rinigus and @MartinK, many thanks for great job.
P.S. food for thought :-) maybe will be possible to use http://gis-lab.info/projects/osm_dump/ as source of maps for Russia? regions, not districts.
I am working on making it possible. Namely, chopping the regions by poly-files from planet.pbf. It would take some evenings to get the scripts done. After that, it would be possible for you to merge polygons with the names of the regions into the osmscout-server repository in github.
@XOleg, I have finished a first version of the scripts last night and it seems that all is going quite well.
You can insert the regions under
https://github.com/rinigus/osmscout-server/tree/master/scripts/import/hierarchy
as explained in https://github.com/rinigus/osmscout-server/tree/master/scripts/import README.
In short, you would have to fork my repo, make a patch, and submit it back. Even if you haven't worked with GitHub, its not too hard.
As you could see, its possible to add custom regions as well. I am preparing for the first update of the maps, but there will be more details regarding it a bit later
@XOleg, I have finished a first version of the scripts last night and it seems that all is going quite well.
You can insert the regions under
https://github.com/rinigus/osmscout-server/tree/master/scripts/import/hierarchy
as explained in https://github.com/rinigus/osmscout-server/tree/master/scripts/import README.
In short, you would have to fork my repo, make a patch, and submit it back. Even if you haven't worked with GitHub, its not too hard.
As you could see, its possible to add custom regions as well. I am preparing for the first update of the maps, but there will be more details regarding it a bit later
@rinigus thank for your very fast job, but... This is Chinese letter for me :-( I would prefer either download districts with new version of server or import regions for old version if there is no more simple solution.
@rinigus thank for your very fast job, but... This is Chinese letter for me :-( I would prefer either download districts with new version of server or import regions for old version if there is no more simple solution.
:) sorry ,would have been complicated for me couple of month as well . So, let's get to the bottom of this .
Aim is to get regions prepared into the system so you and all others could download them into the phone using OSM Scout Server. In addition, the regions would be updated with new OSM data once in a while and follow OSM Scout Server versions.
To get maps to the phone, you need to know a little bit about how they are prepared. In short, I download current Planet PBF and process it to get PBF for each region that you see available in you're Map Manager. After getting region PBFs, I convert them into libosmscout and geocoder-nlp formats. All this is later uploaded into @MartinK server from where you download them.
Now all regions are described using simple tree of hierarchies that I linked earlier. Just try to go and browse into Russia. You would see districts, if I call them correctly. Each district has two files, name and poly. Name is simple: just the name of the region. Poly is a file that describes a line on the map surrounding that region. This file is used when regional PBFs are made and later imported.
So, to add a new region, you have to make a directory under corresponding district. The hierarchy has to be followed, otherwise we will very fart get into too slow imports and huge mess.
In case of Russia, the link that you provided for smaller regions also had Poly files. So, it's a matter of inserting them into corresponding folders and following our convention. I'll be happy if you or someone else would arrange it accordingly.
As far as I remember, you mentioned that you needed region which is between normal hierarchies. I think you could add that as well, just position it accordingly. For that, you will need to make own Poly file. See Barcelona for example of such simple region.
One question for Russian users : do you need full Russia map or are the regions fine? I am planning to remove full Russia import and just leave parts, as done for US. So, unless we have lots of users using full Russia map, there is no reason to keep it in this way .
Now all regions are described using simple tree of hierarchies that I linked earlier. Just try to go and browse into Russia. You would see districts, if I call them correctly. Each district has two files, name and poly. Name is simple: just the name of the region. Poly is a file that describes a line on the map surrounding that region. This file is used when regional PBFs are made and later imported.
So, to add a new region, you have to make a directory under corresponding district. The hierarchy has to be followed, otherwise we will very fart get into too slow imports and huge mess.
In case of Russia, the link that you provided for smaller regions also had Poly files. So, it's a matter of inserting them into corresponding folders and following our convention. I'll be happy if you or someone else would arrange it accordingly.
As far as I remember, you mentioned that you needed region which is between normal hierarchies. I think you could add that as well, just position it accordingly. For that, you will need to make own Poly file. See Barcelona for example of such simple region.
One question for Russian users : do you need full Russia map or are the regions fine? I am planning to remove full Russia import and just leave parts, as done for US. So, unless we have lots of users using full Russia map, there is no reason to keep it in this way .
@rinigus this is explanation https://en.wikipedia.org/wiki/Federal_districts_of_Russia
I'll try to read yet again, understand and make. Yes, two regions are interested for me, one from one district and second from other. IMHO: maybe it's simple to download full Russia map, but space and time for dowload are required... Full map(not for Russia only, Monaco is exception) is required maybe for football funs or for vacation on wheels via country...
Hi! There will be disruption of the service due to updates of the maps today. Its mainly caused by a small flaw in the design that is corrected already and will be released as a part of the next OSM Scout Server release.
After the maps are updated, you would have to "Update list" before you could download again. In general, if you see download errors related to missing file on a server, try to update the list in the manager and try to download again.
I have finished an update of the maps. Since the hierarchy was created a bit differently, few notes:
* Update the lists of countries before downloading anything. URL has changed to ensure that you do get a version that your app is expecting to get
* Some regions have different internal IDs (Russia, US/Regions, and possibly few others). To fix it, unsubscribe and then resubscribe again. For re-subscription, its better if you leave the country page and select it again for subscription in the dialogue covering all countries. Its a problem with the current update that should not be there in future, so let's live with this inconvenience.
* For all other regions, its sufficient to open a country page, press Unsubscribe and then Subscribe to initiate an update. Note that the corresponding country will be marked as an unavailable before you download a new version.
* All updates are done when you authorize the download.
* There are much more regions now with subdivisions in UK, Germany and few other countries. This is done by following Geofabrik.
* Full Russia, Canada (US, as before) are not provided. If you use them, let me know and we can try to put them back. I assumed that you cannot drive during one day through them.
In the next release, I plan to make updates easier by adding a corresponding dialogue. Then this manual Unsubscribe/Subscribe cycle would disappear.
Fellfrosch
2017-03-16, 15:36
Hi, I've upgraded from an older version without map management. Now OSM-Server doesn't work anymore. I have reinstalled it. I've made a new empty folder and chosen that. But after that I get the message:
Failed to open the database for tracking downloaded files.
Map Manager functionality would be disturbed.
I can't choose countries or anything to download and even my old Map database doesn't work.
Any advice?
Hi, I've upgraded from an older version without map management. Now OSM-Server doesn't work anymore. I have reinstalled it. I've made a new empty folder and chosen that. But after that I get the message:
Failed to open the database for tracking downloaded files.
Map Manager functionality would be disturbed.
I can't choose countries or anything to download and even my old Map database doesn't work.
Any advice?
That's new. It corresponds to the case where it cannot create or open SQLite database in the folder that you allocated. Its either some problem with the file permissions, my bug, Qt bug or combination of these.
Let's try to track it down.
* If you go to the allocated folder and try to create subfolder, does it work? Do you have permissions as user "nemo" to do it?
* Close OSM Scout Server, go to the allocated folder. Do you have "files.sqlite" in that folder? If you have it, delete this file and start application again.
* If it all fails, please try to allocate folder just in /home/nemo . Like /home/nemo/Maps.OSM
I assume you used some regular filename, without äöõ or some other fancy character in the path.
If we are still in trouble - let me know version of your SFOS, where did you try to allocate it. Please also open then new issue at github and paste the configuration file for OSM Scout Server from
/home/nemo/.config/harbour-osmscout-server/harbour-osmscout-server.conf
Please let me know how it went.
@rinigus It's OK for me. I saw "Timeout" after 10' of downloading accidentally, then I continue and download is finished after 30' from restart. 1GB for 2 districts.
I made:
- unsubscribe
- cleanup
- update list
- subscribe
- start download
Hi, I've upgraded from an older version without map management. Now OSM-Server doesn't work anymore. I have reinstalled it. I've made a new empty folder and chosen that. But after that I get the message:
Failed to open the database for tracking downloaded files.
Map Manager functionality would be disturbed.
I can't choose countries or anything to download and even my old Map database doesn't work.
Any advice?
I have added an error description as provided by Qt's SQL engine in the new build. You could get a build from
<LINK TAKEN DOWN SINCE THE ISSUE IS RESOLVED>
However, the error message in the case of read-only folder was rather bizarre "out of memory Error opening database". So, let's see, maybe you have better luck and error message in your case would be clearer.
Fellfrosch
2017-03-16, 18:58
That's new. It corresponds to the case where it cannot create or open SQLite database in the folder that you allocated. Its either some problem with the file permissions, my bug, Qt bug or combination of these.
Let's try to track it down.
* If you go to the allocated folder and try to create subfolder, does it work? Do you have permissions as user "nemo" to do it?
* Close OSM Scout Server, go to the allocated folder. Do you have "files.sqlite" in that folder? If you have it, delete this file and start application again.
* If it all fails, please try to allocate folder just in /home/nemo . Like /home/nemo/Maps.OSM
I assume you used some regular filename, without äöõ or some other fancy character in the path.
If we are still in trouble - let me know version of your SFOS, where did you try to allocate it. Please also open then new issue at github and paste the configuration file for OSM Scout Server from
/home/nemo/.config/harbour-osmscout-server/harbour-osmscout-server.conf
Please let me know how it went.
Ohhhhhhh .....
Stupid me!!!! Of course it was a permission problem. Sorry for bothering you with my thoughtlessness :o
Changed permission, everything works as expected. Thanx for your great work!!!
Ohhhhhhh .....
Stupid me!!!! Of course it was a permission problem. Sorry for bothering you with my thoughtlessness :o
Changed permission, everything works as expected. Thanx for your great work!!!
Thank you for reporting it. Its better to get it iron out and ensure that the error message is actually correct. Also, since I didn't anticipate it - I found a condition where the server was responding incorrectly. So, thank you for report :)
So, to add a new region, you have to make a directory under corresponding district. The hierarchy has to be followed, otherwise we will very fart get into too slow imports and huge mess.
In case of Russia, the link that you provided for smaller regions also had Poly files. So, it's a matter of inserting them into corresponding folders and following our convention. I'll be happy if you or someone else would arrange it accordingly.
Hi, @rinigis. if I understand correctly it's possible to replace files in folder for district by files(after import from PBF) for region only? if I understood correctly description(from site) maps for regions are made with Poly files. I don't see Poly files in folders for OSM_SCout_server maps on device.
Hi, @rinigis. if I understand correctly it's possible to replace files in folder for district by files(after import from PBF) for region only? if I understood correctly description(from site) maps for regions are made with Poly files. I don't see Poly files in folders for OSM_SCout_server maps on device.
Yes, we can make that way that you get smaller regions instead of the larger ones. Let's keep both options possible, so you would place poly files under hierarchy folder in GitHub.
Poly files are used when I generate the regions, not on device. So, you would have to make it in GitHub and send over to me via forking/pull request in GitHub. Then I can run scripts on my computer, send it to @MartinK system and download them to your phone. Easy, isn't it :)
Yes, we can make that way that you get smaller regions instead of the larger ones. Let's keep both options possible, so you would place poly files under hierarchy folder in GitHub.
Poly files are used when I generate the regions, not on device. So, you would have to make it in GitHub and send over to me via forking/pull request in GitHub. Then I can run scripts on my computer, send it to @MartinK system and download them to your phone. Easy, isn't it :)
@rinigus good idea. maybe this will be convenient not for me only. I'll read manual and will try to send files.
@rinigus good idea. maybe this will be convenient not for me only. I'll read manual and will try to send files.
@XOleg, try to do with only one sub-division first. Then I can give the feedback on your pull request and we can continue after that on other parts. That way we'll do it as efficient as possible.
@rinigus. I'm sorry, but don't understand this technology.
I see regions here http://gis-lab.info/projects/osm_dump/ (for exemple http://data.gis-lab.info/osm_dump/dump/latest/RU-AD.osm.pbf), Poly files here http://data.gis-lab.info/osm_dump/poly/ (for this exemple RU-AD.poly).
And?.. what to do next with this?..
@rinigus. I'm sorry, but don't understand this technology.
I see regions here http://gis-lab.info/projects/osm_dump/ (for exemple http://data.gis-lab.info/osm_dump/dump/latest/RU-AD.osm.pbf), Poly files here http://data.gis-lab.info/osm_dump/poly/ (for this exemple RU-AD.poly).
And?.. what to do next with this?..
So, we would like to get it under
https://github.com/rinigus/osmscout-server/tree/master/scripts/import/hierarchy/russia/south-fed-district
in a separate folder. I think I've got it right and its under South Federal District, right? In particular, we would like then to make a subfolder with the name "adygea" and place 2 files into that subfolder
"poly" - this will contain RU-AD.poly . Please rename that into just "poly"
name - this will have Adygea or Republic of Adygea . I guess, you will have to choose how to call it. The name you put into this file, will be the one that you will see on your phone.
As a result, we will have subfolder with files:
adygea
adygea/poly
adygea/name
Now, to get these files in, you would have to go through typical GitHub development process:
1. Register at github.com
2. Go to https://github.com/rinigus/osmscout-server
3. Under https://github.com/rinigus/osmscout-server , press "Fork" to make your copy of it.
4. In YOUR copy, I presume under https://github.com/XOleg/osmscout-server , go to https://github.com/XOleg/osmscout-server/tree/master/scripts/import/hierarchy/russia/south-fed-district and press "Create new file".
5. With "New file name", you can type adygea/name and type Adygea or Republic of Adygea in the contents .
6. When happy with the file, go to the bottom, insert in a smaller box "adding adygea/name" and press Commit
7. Repeat 5 & 6 for poly
8. Go to your copy root (https://github.com/XOleg/osmscout-server) and submit your changes via Pull Request (press "New pull request" and go through the changes). Wait a bit till you get my reply :)
When you get to practice it a bit more, it would be better to create a branch when you work on Pull Requests, but maybe its fine without it for now
I am sure there are lots of tutorials on GitHub development online. Since there are lots of Russians developing open source projects, I am sure you even have a luxury to read the help/tutorials in Russian. Main thing is to not get scared away in the beginning, its actually rather simple.
Good luck and just ask if you have any problems.
@rinigus. thanks for help. I'll try...
I made. This is OK?
How about translate to Russian for name of regions?
@XOleg, just looked into your GitHub. Its going fine, except you added a name file as
adygea/name
not just as adygea. To change the name:
1. go to https://github.com/XOleg/osmscout-server/blob/master/scripts/import/hierarchy/russia/south-fed-district/adygea
2. press pencil buttom (edit)
3. you would have the name on the top part of the page, where you can edit "adegea". Edit it by adding "/" and then "name"
4. Commit the changes
1. It's impossible:
"Sorry, a file exists where you’re trying to create a subdirectory. Choose a new path and try again."
2. It's OK for poly in district folder? This file is for Adygea
@rinigus I corrected. I restored district folder with poly for district from geofabrik and created region folder
@rinigus I corrected. I restored district folder with poly for district from geofabrik and created region folder
It looks great now! That's exactly as it is supposed to be. Please send it over and we could test whether all works. For that, go to your github project page and create new pull request, as explained earlier.
I'll try to merge and test it ASAP. After accepted, you could just continue with the other regions at your own speed.
How about translate to Russian for name of regions?
This is actually hard question. We have 360 regions to translate - that's a huge task! I do wonder how to automate that.
There is also another issue - how to organize translations? Do we do it via transifex? Then all regions would be translated not when inserted but through the code.
Alternative is to add name.ru , for example, and put russian translation into it. But the issue would be that it would be hard for translators to go through the maze of files and get it translated. If its too hard - noone would do it
If anyone knows a good solution, speak up.
A bit off-topic, so please don't reply:
I just wanted to point out the great job rinigus is doing here, both for support and for development. I don't use Scout Server (yet), but still felt the urge to click 'thanks' to the last pages of his support/change comments – please consider doing exactly that if he helped you… I think this effort calls for a few more thanks below his name. ;)
It looks great now! That's exactly as it is supposed to be. Please send it over and we could test whether all works. For that, go to your github project page and create new pull request, as explained earlier.
I'll try to merge and test it ASAP. After accepted, you could just continue with the other regions at your own speed.
@rinigus I'm sorry. I don' understood. I pressed "create pull request" after correction.
This is not enough?
P.S. Transifex it's not bad for me. It's not very difficult to translate name of regions. 360 names. It's possible to reduce? ;-)
@rinigus I'm sorry. I don' understood. I pressed "create pull request" after correction.
This is not enough?
P.S. Transifex it's not bad for me. It's not very difficult to translate name of regions. 360 names. It's possible to reduce? ;-)
Strange, I haven't received anything. You might have not confirmed it.
1. Go to https://github.com/XOleg/osmscout-server
2. Press "Pull request"
3. You will get a page called "Comparing changes". Press big green button "Create pull request"
Re translations: let's see if we can somehow at least prefill some of them
Strange, I haven't received anything. You might have not confirmed it.
1. Go to https://github.com/XOleg/osmscout-server
2. Press "Pull request"
3. You will get a page called "Comparing changes". Press big green button "Create pull request"
Re translations: let's see if we can somehow at least prefill some of them
After 1-3 I see https://github.com/rinigus/osmscout-server and "Name is requested"
Name is requested
That's right, put there something. Like "adding adygea"
That's right, put there something. Like "adding adygea"
this is made
this is made
@XOleg - just tested - it all worked nicely. Now, how do we do? Do you want to make more regions and then we make all the updates later tonight? It takes 2-4 hours to get it into the server, due to the technical solution used.
Translators: I am finishing an intermediate release that would expose updates to GUI and also would help on the communication with the maps distribution server. Unfortunately, one of the longer help strings has been changed and was dropped by lang generator. I hope its OK and you could update it. Your older version can be seen at https://github.com/rinigus/osmscout-server/commit/747c96f8fed82857cc3f67aa4d7c06756226482a among all other changes in line numbers.
@rinigus. Yes I'll continue now.
I see translation for this moment
@XOleg - just tested - it all worked nicely. Now, how do we do? Do you want to make more regions and then we make all the updates later tonight? It takes 2-4 hours to get it into the server, due to the technical solution used.
Translators: I am finishing an intermediate release that would expose updates to GUI and also would help on the communication with the maps distribution server. Unfortunately, one of the longer help strings has been changed and was dropped by lang generator. I hope its OK and you could update it. Your older version can be seen at https://github.com/rinigus/osmscout-server/commit/747c96f8fed82857cc3f67aa4d7c06756226482a among all other changes in line numbers.
Done.
One strings seems strange : "Creates directory<br>%1<br>and configures it for storing maps". Should really this string extend on several lines (br), despites the "and" that would indicate it is a unique sentence ?
Also can you explain the meaning of the "%L1 (D) / %L2 (W)" substring that is found in some strings. What do the "D" and "W" represent ? I forgot to ask last time, and have kept them as is in the French Translation.
Thank you!
One strings seems strange : "Creates directory<br>%1<br>and configures it for storing maps". Should really this string extend on several lines (br), despites the "and" that would indicate it is a unique sentence ?
In practice, its "Creates directory<br>/home/nemo/Maps.OSM<br>and configures it for storing maps". Without breaking lines, you get a long path that is split into two lines on probably all devices. With the breaks, its clearer, I think.
Also can you explain the meaning of the "%L1 (D) / %L2 (W)" substring that is found in some strings. What do the "D" and "W" represent ? I forgot to ask last time, and have kept them as is in the French Translation.
:) . That's used during downloads. For example, 100.0 (D) / 200 (W) and tells you how much is left to download (D) and write (W) to the storage after unpacking.
@rinigus. I finished South-FD. to make Full request after each district or to continue with others to end?
Crimea region is part of South-FD for this moment. maybe delete Crimea district?
P.S. I corrected translation.
With the breaks, its clearer, I think. Ok. Thanks. I've translated it like you expected, so it's good for me.
:) . That's used during downloads. For example, 100.0 (D) / 200 (W) and tells you how much is left to download (D) and write (W) to the storage after unpacking.Ok. I'll see in the next translation batch if I can come up with something clear in French, in the meantime, it will stay like in English.
@rinigus. I finished South-FD. to make Full request after each district or to continue with others to end?
Crimea region is part of South-FD for this moment. maybe delete Crimea district?
P.S. I corrected translation.
I must say I was very surprised seeing Crimea as a part of Russia as shown by Geofabrik. However, I decided not to enter into any political debate at that stage and just propagate regions as done by Geofabrik, hence the statement regarding hierarchies in README.
Situation with Crimea is a pain for many and quite well characterized by Wikipedia (de jure vs de facto) and its status. In OSM, they have "Boundary relation for territories occupied by Russia", if I remember correctly.
I would prefer to stay away from this sore subject. I think we could follow Geofabrik lead and position it according to de facto and you could go ahead and correct that as it is now and position it under South-FD.
Please note that if we get argumentation regarding this positioning, we would have to address it probably. Its just extremely sad that there is this war and the countries cannot continue focusing on well-being of each other
@rinigus.
1. my question is about maps service, not political. We have 2 datasets for 1 region. server will work correct for voyagers?..
2. I make all regions without Pull request? Pull request will then?
@rinigus.
1. my question is about maps service, not political. We have 2 datasets for 1 region. server will work correct for voyagers?..
2. I make all regions without Pull request? Pull request will then?
1.correct , that's best way to approach . let's put only one for the same region . if the region moved , them let's move it accordingly. so, please delete old
option and insert where it's moved now in hierarchy .
2. please submit when ready . it's not related to release , we can update it separately
1.correct , that's best way to approach . let's put only one for the same region . if the region moved , them let's move it accordingly. so, please delete old
option and insert where it's moved now in hierarchy .
2. please submit when ready . it's not related to release , we can update it separately
1. OK. I'll delete crimea-district.
2.OK. Two districts remain
I'll see in the next translation batch if I can come up with something clear in French, in the meantime, it will stay like in English.
I have the same problem in Swedish. There's no clear one-character translation, so I just keep the original letters. Optimal linguistic would be using the whole word. Like "left" and "written". Don't know if it is graphically feasible though.
1. OK. I'll delete crimea-district.
2.OK. Two districts remain
Huge work, thank you! @XOleg, don't forget to insert the "special" region that you wanted. See how I made Barcelona, as an example. You basically just have to make poly file yourself by writing in coordinates of 4 points. Just remember to do it in the right order for this format - longitude, latitude. Format description here, if you need it: http://wiki.openstreetmap.org/wiki/Osmosis/Polygon_Filter_File_Format . Put some nice name to that region and we are good to go.
I have the same problem in Swedish. There's no clear one-character translation, so I just keep the original letters. Optimal linguistic would be using the whole word. Like "left" and "written". Don't know if it is graphically feasible though.
Actually, both numbers correspond to remaining part, i.e. left to download, left to write. Its probably not too important, just gives some visual feedback to the user that things are moving.
@rinigus. I finished by Pull request
@rinigus. I finished by Pull request
All imported. I have started pushing it to the cloud and later @MartinK server would get the added regions. If all goes well we'll have all new regions available late tonight, probably around 20:00 GMT.
As soon as the upload on my side is done, I'll push new regions to github from which you could see the updated list of regions (you'll be able to get them by "Update list"). Just the download of the region will not work until the @MartinK's server will fetch it (~20:00 GMT).
Thank you again!
@rinigus. but without your help... :-)
I would hope the work was not in vain...
@rinigus. but without your help... :-)
I would hope the work was not in vain...
Surely not in vain. I missed the target (too much to upload) so updates will take 2 more hours, ~22:00 GMT as it looks for now.
Surely not in vain. I missed the target (too much to upload) so updates will take 2 more hours, ~22:00 GMT as it looks for now.
Looks like the update went fine & the new regions are in. :)
I have just published 0.7.5 release. It exposes map update functionality to GUI, fixes a race condition on map updates by loading list of provided countries/features from data.modrana.org, and updates the translations.
Unless there are some issues coming up with the Map Manager, I am planning now to continue the work on geocoder-nlp performance and database size.
@rinigus appli and maps updated without problem for me. but I have a question. Routing is impossible between separate regions(countries...). Server selects one region. For exemple, it's necessary to travel from St-Petersbourg to Moscow. I have two maps for cities and two maps for regions. And how to be? I suppose it's impossible to pass from Moscow to Moscow region also...
This is other side of maps fragmentation...
I found one string untranslated. This one is missing in translation file.
Number of parallel worker threads: %1
@rinigus appli and maps updated without problem for me. but I have a question. Routing is impossible between separate regions(countries...). Server selects one region. For exemple, it's necessary to travel from St-Petersbourg to Moscow. I have two maps for cities and two maps for regions. And how to be? I suppose it's impossible to pass from Moscow to Moscow region also...
This is other side of maps fragmentation...
Routing, search , and drawing are all impossible between different maps . it's an important limitation thwart we will address in future. I have these issues filed in GitHub, so we'll surely work on it. My plan is to finish geocoder first , so I can property fix one component before working on the next one .
I found one string untranslated. This one is missing in translation file.
Thank you! Fixed in the latest commit and uploaded for translation.
Norwegian translation is now complete. -->Norsk oversettelse er nå komplett
Norwegian translation is now complete. -->Norsk oversettelse er nå komplett
Thank you very much! Its a big work.
With the map distribution server back online (thank you @MartinK!), I am preparing for a new release with an updated geocoder-nlp implementation that reduces the used storage size and should be faster as well. Unfortunately, it requires a new data storage format. While I did my best with the testing, its a major change and could lead to some breakage.
So, if you are planning a trip and depend on OSM Scout Server then I suggest to get the required maps today/tomorrow and store them in a safe place before the upgrade. If all goes as planned, I will publish new version on Wednesday morning with the World maps uploaded to the maps distribution server during a night between Tuesday and Wednesday.
I am happy to announce 0.8.0 release.
This release is mainly about geocoder-nlp polishing. Searches using geocoder-nlp should be faster and the datasets are about 2 times smaller. The libpostal part is still as large as before, but that is out of my hands.
The database format has changed and, as a result, you would have to get new version of the maps. The maps have been uploaded, so all should be ready. If you get the server from Harbour, it should be published on 29th, to sync with the new maps update.
To update the maps, open Map Manager and click Check for updates. Note that due to incompatibility of geocoder-nlp part, you cannot use 0.7.x maps on 0.8.x if you subscribed to geocoder-nlp. Update the maps as soon as you move to 0.8.x, please.
In this version, Linux CLI is supported again. Use command line options to subscribe and download the maps. Some configuration is done via INI configuration file that is populated with defaults on the first run. If you have any questions, please ask and I will help you out.
There are several translations updated. In addition, several fixes are carried over via update of libosmscout version.
I consider geocoder-nlp as ready for use. Please let me know if you experience slow response, too much RAM used, missing or not found data in search. I'll be happy to improve it as much as I can.
Unless major bug reports will start flying in, I will focus now on other aspects of the map server.
@rinigus I updated without problems. Maps (77MB + 81MB) have been updated via 3G in 8 minutes.
carmenluci
2017-03-29, 15:55
I am happy to announce 0.8.0 release.
This release is mainly about geocoder-nlp polishing. Searches using geocoder-nlp should be faster and the datasets are about 2 times smaller. The libpostal part is still as large as before, but that is out of my hands.
The database format has changed and, as a result, you would have to get new version of the maps. The maps have been uploaded, so all should be ready. If you get the server from Harbour, it should be published on 29th, to sync with the new maps update.
To update the maps, open Map Manager and click Check for updates. Note that due to incompatibility of geocoder-nlp part, you cannot use 0.7.x maps on 0.8.x if you subscribed to geocoder-nlp. Update the maps as soon as you move to 0.8.x, please.
In this version, Linux CLI is supported again. Use command line options to subscribe and download the maps. Some configuration is done via INI configuration file that is populated with defaults on the first run. If you have any questions, please ask and I will help you out.
There are several translations updated. In addition, several fixes are carried over via update of libosmscout version.
I consider geocoder-nlp as ready for use. Please let me know if you experience slow response, too much RAM used, missing or not found data in search. I'll be happy to improve it as much as I can.
Unless major bug reports will start flying in, I will focus now on other aspects of the map server.
Thanks for the update!! I've installed it on my tablet, but I can't open it. Is this a bug?
Thanks for the update!! I've installed it on my tablet, but I can't open it. Is this a bug?
Sounds like a bug. I presume you use it on i486, right? Do you have any output when you start it from command line? did the earlier versions work ?
@rinigus I updated without problems. Maps (77MB + 81MB) have been updated via 3G in 8 minutes.
Great to hear!
carmenluci
2017-03-29, 16:53
Sounds like a bug. I presume you use it on i486, right? Do you have any output when you start it from command line? did the earlier versions work ?
Yeah, I installed on i486 device. If I open the app via Terminal, it shows this message: 'Instrucción ilegal' (in English 'Illegal instruction').
0.7.5 version works fine, but I can't download maps because they aren't compatible and I must update the app.
Yeah, I installed on i486 device. If I open the app via Terminal, it shows this message: 'Instrucción ilegal' (in English 'Illegal instruction').
0.7.5 version works fine, but I can't download maps because they aren't compatible and I must update the app.
Yes, its kind of a limbo with the change in the map versions. You could use 0.7.5 without geocoder-nlp, but let's try to fix it for 0.8.0.
I don't know what could be the cause for Illegal instruction (broken build, wrong SFOS version, bug). So, I just recompiled the server and uploaded a debug build at
http://osmscoutserver-test.e24files.com/harbour-osmscout-server-0.8.0-1.i486.rpm
Would you mind to test if this one works? I don't have a tablet and, obviously, all works in the emulator, as far as I can tell.
carmenluci
2017-03-29, 18:07
Yes, its kind of a limbo with the change in the map versions. You could use 0.7.5 without geocoder-nlp, but let's try to fix it for 0.8.0.
I don't know what could be the cause for Illegal instruction (broken build, wrong SFOS version, bug). So, I just recompiled the server and uploaded a debug build at
http://osmscoutserver-test.e24files.com/harbour-osmscout-server-0.8.0-1.i486.rpm
Would you mind to test if this one works? I don't have a tablet and, obviously, all works in the emulator, as far as I can tell.
Thanks again!! I've installed the new rpm file, but this bug is not solved.
If I try open the app via Terminal I get the same message:
[nemo@Sailfish Downloads]$ harbour-osmscout-server
[D] unknown:0 - QML debugging is enabled. Only use this in a safe environment.
Instrucción ilegal
Thanks again!! I've installed the new rpm file, but this bug is not solved.
If I try open the app via Terminal I get the same message:
[nemo@Sailfish Downloads]$ harbour-osmscout-server
[D] unknown:0 - QML debugging is enabled. Only use this in a safe environment.
Instrucción ilegal
If you know how to debug, you could use
gdb /usr/bin/harbour-osmscout-server
issue "run" in gdb and see what happens. It will probably give lots of messages regarding missing symbols, but maybe we can pull something out of it. When it crashes, see what "backtrace" command would give.
Since the server passed Harbour checks, I presume that this bug does not show up on clean installation. You could also try to move away configuration file (please move, don't delete, so we could examine it later!):
mv .config/harbour-osmscout-server/harbour-osmscout-server.conf ~/harbour-osmscout-server-send-for-repair.conf
then move away Maps.OSM (if the folder was as proposed by default) and try to start server again. If that doesn't help, try to reboot (maybe something is wrong with the tablet state?).
Would be good to know if there are other tablet users around and if they see the same error...
carmenluci
2017-03-29, 19:48
If you know how to debug, you could use
gdb /usr/bin/harbour-osmscout-server
issue "run" in gdb and see what happens. It will probably give lots of messages regarding missing symbols, but maybe we can pull something out of it. When it crashes, see what "backtrace" command would give.
Since the server passed Harbour checks, I presume that this bug does not show up on clean installation. You could also try to move away configuration file (please move, don't delete, so we could examine it later!):
mv .config/harbour-osmscout-server/harbour-osmscout-server.conf ~/harbour-osmscout-server-send-for-repair.conf
then move away Maps.OSM (if the folder was as proposed by default) and try to start server again. If that doesn't help, try to reboot (maybe something is wrong with the tablet state?).
Would be good to know if there are other tablet users around and if they see the same error...
I've made a clean install, but it doesn't work, I reboot my tablet and doesn't work.
I've tried to use 'gdb' in my tablet, but I get 'command not found', If I've tried to install it I get error (file doesn't found). :(
I use SailfishOS 2.1.0.10, could it be the problem?
Well, maybe someone here who has a tablet can help us with this bug.
Thank you so much for your attention :)
I've made a clean install, but it doesn't work, I reboot my tablet and doesn't work.
I've tried to use 'gdb' in my tablet, but I get 'command not found', If I've tried to install it I get error (file doesn't found). :(
I use SailfishOS 2.1.0.10, could it be the problem?
Well, maybe someone here who has a tablet can help us with this bug.
Thank you so much for your attention :)
@carmenluci, sorry that I don't know exactly how to help you.
I presume we have two differences already: use of i486 and 2.1.0.10. I am using only ported devices (OnePlus X and Nexus 4), with the both of them ARM and 2.0.5.6. Since these are ported devices, we'll move t 2.1.x series only after EA period is finished and there is a proper release. While there might be some builds for these devices around, or its possible to build myself, I would prefer not to jump over yet until its released.
So, would be good to know from other users of i486 (any SFOS version) or 2.1.0.10 (any CPU) how things are working for them. It would be helpful to know with the respect to this bug.
@carmenluci, when you reinstalled the server, did you move away the configuration file as well? Uninstall does not remove them, you have to move them away yourself.
Probably that wouldn't be of great help, but maybe you could run :
harbour-osmscout-server --help
It should just give you a list of command line options and stop the program. I wonder whether your crash occurs before or after the command line options processing.
The crash with 0.8 seems to be x86/i486 wide (or at least Tablet wide) - it also happens on my tablet. This is the output from Journal when I click on the OSM scout Server icon (not very informative):
Mar 30 00:40:17 Sailfish erver[14491]: [D] unknown:0 - QML debugging is enabled. Only use this in a safe environment.
Mar 30 00:40:17 Sailfish mapplauncherd[880]: Boosted process (pid=14491) was terminated due to signal 4
Mar 30 00:40:17 Sailfish kernel: Double click to light up the screen!
Mar 30 00:40:17 Sailfish kernel: traps: harbour-osmscou[14491] trap invalid opcode ip:ef91952c sp:ffe75210 error:0
Mar 30 00:40:17 Sailfish kernel: in harbour-osmscout-server[ef87a000+635000]
Mar 30 00:40:18 Sailfish booster-silica-qt5[15334]: [D] unknown:0 - Using Wayland-EGL
Mar 30 00:40:19 Sailfish mapplauncherd[15334]: SilicaBooster: Initiate asynchronous preload.
Mar 30 00:40:32 Sailfish lipstick[749]: [W] unknown:0 - Queued images being saved: 6
Running the binary directly is similarly uninformative:
$ /usr/bin/harbour-osmscout-server
[D] unknown:0 - QML debugging is enabled. Only use this in a safe environment.
Illegal instruction
BTW, gdb can be installed by (as root):
pkcon install gdb
Running the binary with gdb:
[nemo@Sailfish ~]$ /usr/bin/harbour-osmscout-server
[D] unknown:0 - QML debugging is enabled. Only use this in a safe environment.
Illegal instruction
[nemo@Sailfish ~]$ gdb /usr/bin/harbour-osmscout-server
GNU gdb (GDB) Mer (7.6.2+git2)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-meego-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/harbour-osmscout-server...Missing separate debuginfo for /usr/bin/harbour-osmscout-server
Try: zypper install -C "debuginfo(build-id)=f7ebd8791602e8215bb0e635727ebe3b190401ae"
(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/bin/harbour-osmscout-server
Missing separate debuginfo for /lib/ld-linux.so.2
Try: zypper install -C "debuginfo(build-id)=e5ca5fea8505c81c292f076bc390c65270d07e89"
Missing separate debuginfo for /usr/lib/libz.so.1
Try: zypper install -C "debuginfo(build-id)=af6a51baa1f07eaa8c9bd73fb017241d3ffb5173"
Missing separate debuginfo for /lib/libdl.so.2
Try: zypper install -C "debuginfo(build-id)=9775c1ca2d3c07fa4cb9f2b59e1fd9f85a08c260"
Missing separate debuginfo for /usr/lib/libsailfishapp.so.1
Try: zypper install -C "debuginfo(build-id)=59aad9e537769043ccf7fa8a0fe0d6499f487c74"
Missing separate debuginfo for /usr/lib/libmdeclarativecache5.so.0
Try: zypper install -C "debuginfo(build-id)=99bba885a24a98da3e94a613be79b20a1c342c02"
Missing separate debuginfo for /usr/lib/libQt5Quick.so.5
Try: zypper install -C "debuginfo(build-id)=8364094fc05636916fe6ff9282e11aa150d0fa5d"
Missing separate debuginfo for /usr/lib/libQt5Gui.so.5
Try: zypper install -C "debuginfo(build-id)=db6991c180e7e5f2d5c4978f60eb0fe30fa4352f"
Missing separate debuginfo for /usr/lib/libQt5Qml.so.5
Try: zypper install -C "debuginfo(build-id)=f84345868f1fa7daacf4515c5c47ebe4e0ccea85"
Missing separate debuginfo for /usr/lib/libQt5Network.so.5
Try: zypper install -C "debuginfo(build-id)=0f06ef7212148f40fd36151deb3fa80b07db4c9f"
Missing separate debuginfo for /usr/lib/libQt5Sql.so.5
Try: zypper install -C "debuginfo(build-id)=bd81360ce77a2ea5219a9f5d3ecb040623a2944c"
Missing separate debuginfo for /usr/lib/libQt5Core.so.5
Try: zypper install -C "debuginfo(build-id)=30e2b9010e918aaf632daa4bd6062a3dd74e6204"
Missing separate debuginfo for /usr/lib/libGLESv2.so.2
Try: zypper install -C "debuginfo(build-id)=206956d019f68d899185b6b96978cb7739e808c2"
Missing separate debuginfo for /lib/libpthread.so.0
Try: zypper install -C "debuginfo(build-id)=804779ddd4720ef9414c8d55cb0337b0ca7a3276"
warning: File "/lib/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
add-auto-load-safe-path /lib/libthread_db-1.0.so
line to your configuration file "/home/nemo/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/home/nemo/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Missing separate debuginfo for /usr/lib/libstdc++.so.6
Try: zypper install -C "debuginfo(build-id)=20af1c910af264fd002286a3dcede24883ca3461"
Missing separate debuginfo for /lib/libm.so.6
Try: zypper install -C "debuginfo(build-id)=d7e2c0a8d5d62aaa787a760e3923de7265b521c9"
Missing separate debuginfo for /lib/libgcc_s.so.1
Try: zypper install -C "debuginfo(build-id)=796db85b4c1b956aaba10424526efcba4f2fe7bf"
Missing separate debuginfo for /lib/libc.so.6
Try: zypper install -C "debuginfo(build-id)=470a95eae97feabde51f8756ebec221f06f6ff1c"
Missing separate debuginfo for /usr/lib/libmlite5.so.0
Try: zypper install -C "debuginfo(build-id)=ff0e97f1d45e51946256451033664745aa6387bd"
Missing separate debuginfo for /usr/lib/libQt5DBus.so.5
Try: zypper install -C "debuginfo(build-id)=224668e47ce9721f9e4953bc204ea14aa68d32d7"
Missing separate debuginfo for /usr/lib/libfontconfig.so.1
Try: zypper install -C "debuginfo(build-id)=8e5a192901cb8404fa802b780b4c65cc5d965955"
Missing separate debuginfo for /usr/lib/libpng16.so.16
Try: zypper install -C "debuginfo(build-id)=55cb5e8531078f7faae0a2fccfe8cdaf0b22f3d7"
Missing separate debuginfo for /lib/librt.so.1
Try: zypper install -C "debuginfo(build-id)=d7ecff8acc2f101c820ae3181da9482d7da7055d"
Missing separate debuginfo for /usr/lib/libicui18n.so.52
Try: zypper install -C "debuginfo(build-id)=c2605a12dc5d5982bcf98321d0af1876f58c6460"
Missing separate debuginfo for /usr/lib/libicuuc.so.52
Try: zypper install -C "debuginfo(build-id)=b8657a0e42eac864f494976a9eb790a51d5e49c3"
Missing separate debuginfo for /usr/lib/libpcre16.so.0
Try: zypper install -C "debuginfo(build-id)=f852001cbaed44bb4979dae66033df457aa1c15d"
Missing separate debuginfo for /usr/lib/libgthread-2.0.so.0
Try: zypper install -C "debuginfo(build-id)=95763db723fcaf3e11cb6b85aa455e885c7a823a"
Missing separate debuginfo for /usr/lib/libglib-2.0.so.0
Try: zypper install -C "debuginfo(build-id)=d9774fdedef98b9695a963c306ec13f0d8528b33"
Missing separate debuginfo for /usr/lib/libsystemd-journal.so.0
Try: zypper install -C "debuginfo(build-id)=c0920ac4c6c7b44121597f6b706cd1cb04174f17"
Missing separate debuginfo for /usr/lib/libsystemd-id128.so.0
Try: zypper install -C "debuginfo(build-id)=23c9454e0d6cc6f3405698e3b09c75ccdb74595c"
Missing separate debuginfo for /usr/lib/libhybris-common.so.1
Try: zypper install -C "debuginfo(build-id)=4ffe924f33c060e11933b1a796dc5cf090b7436d"
Missing separate debuginfo for /usr/lib/libandroid-properties.so.1
Try: zypper install -C "debuginfo(build-id)=35821f6a3a6c5600575bb5d6e3f94c98ee3a05ae"
Missing separate debuginfo for /usr/lib/libdconf.so.1
Try: zypper install -C "debuginfo(build-id)=75654a5bce5432214731346e94a80cce6eea0227"
Missing separate debuginfo for /usr/lib/libgio-2.0.so.0
Try: zypper install -C "debuginfo(build-id)=5a364998350fff6e2db9d14288dad4442a52f218"
Missing separate debuginfo for /usr/lib/libgobject-2.0.so.0
Try: zypper install -C "debuginfo(build-id)=a552acbaf20b128bebedf5cdca07f4a11d78de73"
Missing separate debuginfo for /usr/lib/libdbus-1.so.3
Try: zypper install -C "debuginfo(build-id)=71174827d73ce8bcbb48764bc1dfa1fce261d86f"
Missing separate debuginfo for /usr/lib/libfreetype.so.6
Try: zypper install -C "debuginfo(build-id)=831db487e57a707cfe987211fef0cd1fe86ea63c"
Missing separate debuginfo for /usr/lib/libexpat.so.1
Try: zypper install -C "debuginfo(build-id)=0761e9ef1803296e0c9330f145b427282d21431b"
Missing separate debuginfo for /usr/lib/libpcre.so.1
Try: zypper install -C "debuginfo(build-id)=861bae228ba27dd8739f0f8151872a34d814128f"
Missing separate debuginfo for /usr/lib/liblzma.so.5
Try: zypper install -C "debuginfo(build-id)=b988754a7e2658cae8a6d7317b6d1bdb412de5d2"
Missing separate debuginfo for /usr/lib/libgcrypt.so.11
Try: zypper install -C "debuginfo(build-id)=deca1ceee3c1b9f924c4190cb80b3190ed25645f"
Missing separate debuginfo for /usr/lib/libgpg-error.so.0
Try: zypper install -C "debuginfo(build-id)=75eb21064733ea12f78a2c97f11f05c89187eb1b"
Missing separate debuginfo for /usr/lib/libffi.so.5
Try: zypper install -C "debuginfo(build-id)=e969b8dfecd7369ca329f41c969d4c35e4baf7fc"
Missing separate debuginfo for /usr/lib/libgmodule-2.0.so.0
Try: zypper install -C "debuginfo(build-id)=954c5d94fef1d85eec71cd2ef9451ed21adf40c7"
Missing separate debuginfo for /lib/libresolv.so.2
Try: zypper install -C "debuginfo(build-id)=86a89364d3cd6826fcf9325f9d379d132061452a"
Missing separate debuginfo for /usr/lib/libsystemd-login.so.0
Try: zypper install -C "debuginfo(build-id)=23af8b68d045cbe6ecfe625321f781b6759d773a"
Missing separate debuginfo for /usr/lib/libsystemd-daemon.so.0
Try: zypper install -C "debuginfo(build-id)=e034f754d11b02a959fd00a965924026f5f91c21"
[D] unknown:0 - QML debugging is enabled. Only use this in a safe environment.
Program received signal SIGILL, Illegal instruction.
0x565f452c in ?? ()
(gdb) backtrace
#0 0x565f452c in ?? ()
#1 0x56a49c27 in __libc_csu_init ()
#2 0xf695cd3a in __libc_start_main () from /lib/libc.so.6
#3 0x565f4aa6 in _start ()
(gdb)
Let me know if there is something else I should run to get you mere information. :)
Also IIRC the Sailfish OS emulator which should be part of the SDK is also i486 & the SDK runs i486 binaries in it (citing the docs (https://sailfishos.org/develop/sdk-overview/develop-firstapp-article/)):
Click on the emulator_icon icon to launch the emulator. NOTE: This icon is only available if the MerSDK-SailfishOS-i486 kit is active. You can activate the MerSDK-SailfishOS-i486 kit from menu Build → Open Build and Run Kit Selector….
So it might be worth trying if the crash could be reproduced in the emulator, as that should make debugging it quite a bit easier. :)
The crash with 0.8 seems to be x86/i486 wide (or at least Tablet wide) - it also happens on my tablet. This is the output from Journal when I click on the OSM scout Server icon (not very informative):
Mar 30 00:40:17 Sailfish erver[14491]: [D] unknown:0 - QML debugging is enabled. Only use this in a safe environment.
Mar 30 00:40:17 Sailfish mapplauncherd[880]: Boosted process (pid=14491) was terminated due to signal 4
Mar 30 00:40:17 Sailfish kernel: Double click to light up the screen!
Mar 30 00:40:17 Sailfish kernel: traps: harbour-osmscou[14491] trap invalid opcode ip:ef91952c sp:ffe75210 error:0
Mar 30 00:40:17 Sailfish kernel: in harbour-osmscout-server[ef87a000+635000]
Mar 30 00:40:18 Sailfish booster-silica-qt5[15334]: [D] unknown:0 - Using Wayland-EGL
Mar 30 00:40:19 Sailfish mapplauncherd[15334]: SilicaBooster: Initiate asynchronous preload.
Mar 30 00:40:32 Sailfish lipstick[749]: [W] unknown:0 - Queued images being saved: 6
Running the binary directly is similarly uninformative:
$ /usr/bin/harbour-osmscout-server
[D] unknown:0 - QML debugging is enabled. Only use this in a safe environment.
Illegal instruction
BTW, gdb can be installed by (as root):
pkcon install gdb
Running the binary with gdb:
[nemo@Sailfish ~]$ /usr/bin/harbour-osmscout-server
[D] unknown:0 - QML debugging is enabled. Only use this in a safe environment.
Illegal instruction
[nemo@Sailfish ~]$ gdb /usr/bin/harbour-osmscout-server
GNU gdb (GDB) Mer (7.6.2+git2)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-meego-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/harbour-osmscout-server...Missing separate debuginfo for /usr/bin/harbour-osmscout-server
Try: zypper install -C "debuginfo(build-id)=f7ebd8791602e8215bb0e635727ebe3b190401ae"
(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/bin/harbour-osmscout-server
Missing separate debuginfo for /lib/ld-linux.so.2
Try: zypper install -C "debuginfo(build-id)=e5ca5fea8505c81c292f076bc390c65270d07e89"
Missing separate debuginfo for /usr/lib/libz.so.1
Try: zypper install -C "debuginfo(build-id)=af6a51baa1f07eaa8c9bd73fb017241d3ffb5173"
Missing separate debuginfo for /lib/libdl.so.2
Try: zypper install -C "debuginfo(build-id)=9775c1ca2d3c07fa4cb9f2b59e1fd9f85a08c260"
Missing separate debuginfo for /usr/lib/libsailfishapp.so.1
Try: zypper install -C "debuginfo(build-id)=59aad9e537769043ccf7fa8a0fe0d6499f487c74"
Missing separate debuginfo for /usr/lib/libmdeclarativecache5.so.0
Try: zypper install -C "debuginfo(build-id)=99bba885a24a98da3e94a613be79b20a1c342c02"
Missing separate debuginfo for /usr/lib/libQt5Quick.so.5
Try: zypper install -C "debuginfo(build-id)=8364094fc05636916fe6ff9282e11aa150d0fa5d"
Missing separate debuginfo for /usr/lib/libQt5Gui.so.5
Try: zypper install -C "debuginfo(build-id)=db6991c180e7e5f2d5c4978f60eb0fe30fa4352f"
Missing separate debuginfo for /usr/lib/libQt5Qml.so.5
Try: zypper install -C "debuginfo(build-id)=f84345868f1fa7daacf4515c5c47ebe4e0ccea85"
Missing separate debuginfo for /usr/lib/libQt5Network.so.5
Try: zypper install -C "debuginfo(build-id)=0f06ef7212148f40fd36151deb3fa80b07db4c9f"
Missing separate debuginfo for /usr/lib/libQt5Sql.so.5
Try: zypper install -C "debuginfo(build-id)=bd81360ce77a2ea5219a9f5d3ecb040623a2944c"
Missing separate debuginfo for /usr/lib/libQt5Core.so.5
Try: zypper install -C "debuginfo(build-id)=30e2b9010e918aaf632daa4bd6062a3dd74e6204"
Missing separate debuginfo for /usr/lib/libGLESv2.so.2
Try: zypper install -C "debuginfo(build-id)=206956d019f68d899185b6b96978cb7739e808c2"
Missing separate debuginfo for /lib/libpthread.so.0
Try: zypper install -C "debuginfo(build-id)=804779ddd4720ef9414c8d55cb0337b0ca7a3276"
warning: File "/lib/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
add-auto-load-safe-path /lib/libthread_db-1.0.so
line to your configuration file "/home/nemo/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/home/nemo/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Missing separate debuginfo for /usr/lib/libstdc++.so.6
Try: zypper install -C "debuginfo(build-id)=20af1c910af264fd002286a3dcede24883ca3461"
Missing separate debuginfo for /lib/libm.so.6
Try: zypper install -C "debuginfo(build-id)=d7e2c0a8d5d62aaa787a760e3923de7265b521c9"
Missing separate debuginfo for /lib/libgcc_s.so.1
Try: zypper install -C "debuginfo(build-id)=796db85b4c1b956aaba10424526efcba4f2fe7bf"
Missing separate debuginfo for /lib/libc.so.6
Try: zypper install -C "debuginfo(build-id)=470a95eae97feabde51f8756ebec221f06f6ff1c"
Missing separate debuginfo for /usr/lib/libmlite5.so.0
Try: zypper install -C "debuginfo(build-id)=ff0e97f1d45e51946256451033664745aa6387bd"
Missing separate debuginfo for /usr/lib/libQt5DBus.so.5
Try: zypper install -C "debuginfo(build-id)=224668e47ce9721f9e4953bc204ea14aa68d32d7"
Missing separate debuginfo for /usr/lib/libfontconfig.so.1
Try: zypper install -C "debuginfo(build-id)=8e5a192901cb8404fa802b780b4c65cc5d965955"
Missing separate debuginfo for /usr/lib/libpng16.so.16
Try: zypper install -C "debuginfo(build-id)=55cb5e8531078f7faae0a2fccfe8cdaf0b22f3d7"
Missing separate debuginfo for /lib/librt.so.1
Try: zypper install -C "debuginfo(build-id)=d7ecff8acc2f101c820ae3181da9482d7da7055d"
Missing separate debuginfo for /usr/lib/libicui18n.so.52
Try: zypper install -C "debuginfo(build-id)=c2605a12dc5d5982bcf98321d0af1876f58c6460"
Missing separate debuginfo for /usr/lib/libicuuc.so.52
Try: zypper install -C "debuginfo(build-id)=b8657a0e42eac864f494976a9eb790a51d5e49c3"
Missing separate debuginfo for /usr/lib/libpcre16.so.0
Try: zypper install -C "debuginfo(build-id)=f852001cbaed44bb4979dae66033df457aa1c15d"
Missing separate debuginfo for /usr/lib/libgthread-2.0.so.0
Try: zypper install -C "debuginfo(build-id)=95763db723fcaf3e11cb6b85aa455e885c7a823a"
Missing separate debuginfo for /usr/lib/libglib-2.0.so.0
Try: zypper install -C "debuginfo(build-id)=d9774fdedef98b9695a963c306ec13f0d8528b33"
Missing separate debuginfo for /usr/lib/libsystemd-journal.so.0
Try: zypper install -C "debuginfo(build-id)=c0920ac4c6c7b44121597f6b706cd1cb04174f17"
Missing separate debuginfo for /usr/lib/libsystemd-id128.so.0
Try: zypper install -C "debuginfo(build-id)=23c9454e0d6cc6f3405698e3b09c75ccdb74595c"
Missing separate debuginfo for /usr/lib/libhybris-common.so.1
Try: zypper install -C "debuginfo(build-id)=4ffe924f33c060e11933b1a796dc5cf090b7436d"
Missing separate debuginfo for /usr/lib/libandroid-properties.so.1
Try: zypper install -C "debuginfo(build-id)=35821f6a3a6c5600575bb5d6e3f94c98ee3a05ae"
Missing separate debuginfo for /usr/lib/libdconf.so.1
Try: zypper install -C "debuginfo(build-id)=75654a5bce5432214731346e94a80cce6eea0227"
Missing separate debuginfo for /usr/lib/libgio-2.0.so.0
Try: zypper install -C "debuginfo(build-id)=5a364998350fff6e2db9d14288dad4442a52f218"
Missing separate debuginfo for /usr/lib/libgobject-2.0.so.0
Try: zypper install -C "debuginfo(build-id)=a552acbaf20b128bebedf5cdca07f4a11d78de73"
Missing separate debuginfo for /usr/lib/libdbus-1.so.3
Try: zypper install -C "debuginfo(build-id)=71174827d73ce8bcbb48764bc1dfa1fce261d86f"
Missing separate debuginfo for /usr/lib/libfreetype.so.6
Try: zypper install -C "debuginfo(build-id)=831db487e57a707cfe987211fef0cd1fe86ea63c"
Missing separate debuginfo for /usr/lib/libexpat.so.1
Try: zypper install -C "debuginfo(build-id)=0761e9ef1803296e0c9330f145b427282d21431b"
Missing separate debuginfo for /usr/lib/libpcre.so.1
Try: zypper install -C "debuginfo(build-id)=861bae228ba27dd8739f0f8151872a34d814128f"
Missing separate debuginfo for /usr/lib/liblzma.so.5
Try: zypper install -C "debuginfo(build-id)=b988754a7e2658cae8a6d7317b6d1bdb412de5d2"
Missing separate debuginfo for /usr/lib/libgcrypt.so.11
Try: zypper install -C "debuginfo(build-id)=deca1ceee3c1b9f924c4190cb80b3190ed25645f"
Missing separate debuginfo for /usr/lib/libgpg-error.so.0
Try: zypper install -C "debuginfo(build-id)=75eb21064733ea12f78a2c97f11f05c89187eb1b"
Missing separate debuginfo for /usr/lib/libffi.so.5
Try: zypper install -C "debuginfo(build-id)=e969b8dfecd7369ca329f41c969d4c35e4baf7fc"
Missing separate debuginfo for /usr/lib/libgmodule-2.0.so.0
Try: zypper install -C "debuginfo(build-id)=954c5d94fef1d85eec71cd2ef9451ed21adf40c7"
Missing separate debuginfo for /lib/libresolv.so.2
Try: zypper install -C "debuginfo(build-id)=86a89364d3cd6826fcf9325f9d379d132061452a"
Missing separate debuginfo for /usr/lib/libsystemd-login.so.0
Try: zypper install -C "debuginfo(build-id)=23af8b68d045cbe6ecfe625321f781b6759d773a"
Missing separate debuginfo for /usr/lib/libsystemd-daemon.so.0
Try: zypper install -C "debuginfo(build-id)=e034f754d11b02a959fd00a965924026f5f91c21"
[D] unknown:0 - QML debugging is enabled. Only use this in a safe environment.
Program received signal SIGILL, Illegal instruction.
0x565f452c in ?? ()
(gdb) backtrace
#0 0x565f452c in ?? ()
#1 0x56a49c27 in __libc_csu_init ()
#2 0xf695cd3a in __libc_start_main () from /lib/libc.so.6
#3 0x565f4aa6 in _start ()
(gdb)
Let me know if there is something else I should run to get you mere information. :)
Also IIRC the Sailfish OS emulator which should be part of the SDK is also i486 & the SDK runs i486 binaries in it (citing the docs (https://sailfishos.org/develop/sdk-overview/develop-firstapp-article/)):
Click on the emulator_icon icon to launch the emulator. NOTE: This icon is only available if the MerSDK-SailfishOS-i486 kit is active. You can activate the MerSDK-SailfishOS-i486 kit from menu Build → Open Build and Run Kit Selector….
So it might be worth trying if the crash could be reproduced in the emulator, as that should make debugging it quite a bit easier. :)
Martin, thanks for analysis and feedback! @a-dekker has also opened a bug on GitHub, so we could move technical discussion over there: https://github.com/rinigus/osmscout-server/issues/107 (thank you @a-dekker!). I'll be following the both channels, but would try to reply to the questions via the same channel as the question.
SFOS emulator runs 0.8.0 fine for me, but I use stable SDK, not EA version. Since I develop for the stable SFOS, I would prefer to use that. Unfortunately, just compiling "for fun" by others is rather complicated - you have to install a large amount of support libraries that provide "brains" of the server (libosmscout, marisa, libpostal, ...)
To isolate the issue, tablet users out there, which version of SFOS do you run? Martin, do you have 2.1.x on the tablet or 2.0.x?
Do we have anyone running 2.0.x on the tablet that could test if OSM Scout Server starts?
For those who are stuck with this situation: you could use 0.7.5 and in Map Manager and Settings disable geocoder-nlp. That should allow you to use new maps (its only that component that changed), but would limit you to searches via libosmscout.
I did not work on making these transitions between map formats smoother. At present, I am trying to get the whole functionality of the served maps polished and hope that these bumps would be smaller in future.
Update: I tested OSM Scout Server 0.8.0 on 2.1.x SDK emulator - that worked fine. Note that if there is some illegal instruction for i486, it could be still fine for emulator if the instruction is propagated through VM to the host machine.
Re: Crash on i486 tablet
I have posted new RPM (broken functionality) for testing the crash, see https://github.com/rinigus/osmscout-server/issues/107 .
It looks like the issue with the crash on i486 is fixed, or at least I hope so. I have just replaced i486 0.8.0 RPM in OpenRepos with the new version. I presume that it should be OK now. Please let me know how it works. If all is fine, I'll upload a new version to the Harbour as well.
carmenluci
2017-03-31, 05:30
It looks like the issue with the crash on i486 is fixed, or at least I hope so. I have just replaced i486 0.8.0 RPM in OpenRepos with the new version. I presume that it should be OK now. Please let me know how it works. If all is fine, I'll upload a new version to the Harbour as well.
It works perfect!!! Thank you so much!!
Just few words about the latest developments. I have been looking into improving the rendering of the maps. As a part for ensuring quality rendering, I am planning to add Mapnik backend for rendering.
The compilation of Mapnik for SFOS and its packaging is done, at first approximation, for ARM. For i486, I have been held back with what seems to be a bug in SDK and have asked for advice on how to proceed in developers list. When rendering some tiles with a test application on phone, the performance was good as well as the memory consumption. But its early days and would depend on the stylesheet / database layout.
Lots of work is ahead. I have to incorporate Mapnik into the server, design stylesheet together with the database layout. The plan is to use OSM Bright as a starting point (imposm import) and SQLite/Spatialite for data storage. If anyone around is a Mapnik guru, help would be appreciated. At present, I could render Estonia, World map outlines with SQLite database and shapefiles describing the World.
In addition, I am considering to publish OSM Scout Server with Mapnik only via OpenRepos and avoid publishing it in Harbour. That would allow me to distribute following sane Linux model with separate packages for Mapnik and the server. Otherwise I would have to start configuring Mapnik to look for input plugins / fonts in some obscure OSM Scout Server directories. In Harbour, I can publish then the version with libosmscout only, for example. What do you think about such approach?
Feathers McGraw
2017-04-04, 06:37
In addition, I am considering to publish OSM Scout Server with Mapnik only via OpenRepos and avoid publishing it in Harbour...What do you think about such approach?
Sounds sensible to me, the harbour restrictions can be a real pain sometimes. Seems like if they wanted tight control over what can be installed for each app to avoid dependency issues etc, then they should have used a container format... but that's OT.
Can you explain what you think the benefits to the end user of using Mapnik will be? I've had a quick look at the OSM wiki (https://wiki.openstreetmap.org/wiki/Mapnik) but it's still unclear to me - is it speed of rendering, or quality, or both?
Can you explain what you think the benefits to the end user of using Mapnik will be? I've had a quick look at the OSM wiki (https://wiki.openstreetmap.org/wiki/Mapnik) but it's still unclear to me - is it speed of rendering, or quality, or both?
Quality of the rendering and, in particular, position of labels should be top notch. There has been lots of work done on rendering in this library with lots of attention to details. When compared to libosmscout, Mapnik had much more developer-hours poured into it with the focus on one aspect of the function - rendering.
The stylesheet I was thinking to start with is https://github.com/mapbox/osm-bright . Maybe there are better styles out there that should be considered as a starting point. If so, let me know (ASAP).
When I manage to get stylesheet fully ported (there are several steps involved), it would be possible to engage end-users into design of the map styles using tools developed for Mapnik. Again, eco-system of Mapnik is much bigger allowing us to reap benefits from it.
As for speed of rendering, this is hard to judge right now. With a crippled style I was using it was very fast, but let's see it later.
There will probably be drawbacks too. In particular, right now, the data files (SQLite database) for feeding Mapnik is 2-3 times larger than libosmscout import. But that datafile includes all traffic lights and probably some other not so important information. For now, it was important to see that its actually possible. Let's see how well would it work out.
otsaloma
2017-04-04, 19:17
Sounds interesting! Two questions come to mind: What stylesheet language do you plan on using and are there graphical editors available for it? Does Mapnik and your stylesheet format make it easy to do different ppi tiles?
Sounds interesting! Two questions come to mind: What stylesheet language do you plan on using and are there graphical editors available for it?
Its in CartoCSS which would be converted into XML that Mapnik uses to define maps and styles. Since we have multiple maps (different regions/countries) on the device, I am planning to define styles using normal mapnik approach (Carto -> XML) and then populate it by the server with the maps installed on device. That should allow to render all installed maps at once without the need to select one.
I am planning to take OSM Bright (it seems to be a rather popular style and the base for several others) and adapt it to SQLite import. I am not planning to edit style as such, there are probably better artists out there :)
As for GUI, there are probably at least 2, https://tilemill-project.github.io/tilemill/ is one of them.
Does Mapnik and your stylesheet format make it easy to do different ppi tiles?
Yes, there seem to be no problem with high PPI. As far as I can see, user would be able to define scaling factor that (s)he likes and its gonna be rendered on-fly with the same stylesheet but correspondingly wider/larger elements (lines, fonts, ...). So, it all fits the bill, as far as I can see.
I have just published a minor update by releasing 0.8.1.
Main job was done by translators who updated the strings, thank you. The update was published to get i486 fix submitted to Harbour as well. There is no new functionality added nor anything else refined.
I hope nothing got broken with i486 now.
Release: 0.9.0
I am happy to release 0.9.0 that brings Mapnik as an alternative rendering backend. This is probably the biggest release so far in terms of size - with the bundled libraries we are reaching 15MB for RPM package. And that's without mentioning huge map files (more on that a bit later). The release is not perfect, but I can probably tinker on it for months if I want to get it out very polished. So, I decided to release a bit early.
Mapnik brings an excellent rendering of maps including world coastlines and rendering of all available maps on device. But all this comes at cost - large maps and possibly slower rendering. So, its up to you to decide which backend to use. I have a preference to use Mapnik :)
With the introduction of new backend, there is a problem with the map updates. However, since I am not planning to add too many backends, I think its better to work around it on your (as a user) side than to start investing time into programming fixing the issue. So, a general remark: When a new backend is added, such as with the release of 0.9.0, please ensure that you have subscription information updated. For that, in Map Manager, click on "Check for updates", decline updates, and unsubscribe/subscribe each territory that you have subscribed. After that, proceed with the downloads. This is not needed if you don't want to use the added backend(s).
Finally, regarding this release, there are some issues with the maps distribution server. Right now, I switched to old CDN model. As soon as data.modrana.org is back online, I will stop distribution via CDN (or when the credit runs out, whatever comes first). So, before downloads, just click on "Check for updates". This should reroute you to the current distribution server. Sorry for inconvenience. If anyone has an access to the server that can be used for maps distribution and wants to contribute, please let me know, we could always use them as a backup, for example.
I will look into how to reduce Mapnik datasets. There is an idea on switching to different geometry BLOB format, but it requires some testing. I will not be able to probably get as low as libosmscout does, but would give a shot on reducing it. Right now, serving Germany/France as a single database is not that simple on mobile with Mapnik, but I haven't tried.
In addition to the reduction of the map database size, there is some work to be done on styling it as well. I have adopted a style used as a basis for many others and modified it to suit SQLite database. Style is available at https://github.com/rinigus/osm-bright-sqlite . It does require further review on which features should be visible at which zoom level. Try to use it and give the feedback on when should we should house numbers, POIs, and such. Some braver users could try to re-style it, make a style for night driving, ...
Enjoy using it!
712+(241+223) MB downloaded in 25 minutes.
otsaloma
2017-04-23, 15:21
Tested the Mapnik rendering on the tablet, it seemed to work fine. Rendering speed was reasonably good. The map style though is as they state a bit of a "starting point".
Tested the Mapnik rendering on the tablet, it seemed to work fine. Rendering speed was reasonably good. The map style though is as they state a bit of a "starting point".
I am planning to mainly work on the performance and shrinking the databases. Some changes may require special patches for mapnik, but I'll submit them upstream. For example, I think I found the way to reduce maps significantly, about to the half of their current form. To my surprise it did not effect the performance (initial testing on onyx), so we should probably switch to that. However, that means I would need to keep two versions for the style - one for mobile with all patches applied and the second for PC and style design.
For best performance, I tune the database layout to match the style. This means that if we want to make major changes in loading sequence, we would be better off to tune the database too. Which is not a major issue, as long as we remember it. In particular, you would want SQL SELECT statements in https://github.com/rinigus/osm-bright-sqlite/blob/master/osm-bright/osm-bright.spatialite.mml to match SQL database as arranged at https://github.com/rinigus/osm-bright-sqlite/blob/master/importer/reorganize.sql .
I have not looked into GUIs on style editing. Someone who wants to adjust styles, should probably look into https://github.com/kosmtik/kosmtik or https://tilemill-project.github.io/tilemill/ . TileMill seems to come in a package that installs mapnik as well. So, its not good for me (I have to use a custom one), but could be great for a designer. I don't know if they come with SQLite plugin enabled though. kosmtik I tried, but it didn't seem to work on my Linux PC immediately, probably some tinkering is required.
Anyway, to contribute to style development: we could collect the ideas either here or in issues of the style project; try to setup GUI and see if it works. I'll be happy to help if you run into problems with the setup of GUI, as much as I understand how it works.
If there is someone interested in working on a style, let us know and see how we can help.
otsaloma
2017-04-23, 17:25
I remember making a couple map styles when TileMill was a Mapbox product and connected to their APIs. It's now a bit more work set up data for TileMill, but I remember basic styling, such as changing colors and changing the zoom levels at which elements are visible, being fairly easy. Things like symbols for POIs and road number shields are more complicated, but you can get quite far without those.
I'm not quite a designer, but definitely interested in map styling. However, I'm more likely to try to get my own online tileserver running and style that with the Mapbox GL JSON style format.
I remember making a couple map styles when TileMill was a Mapbox product and connected to their APIs. It's now a bit more work set up data for TileMill, but I remember basic styling, such as changing colors and changing the zoom levels at which elements are visible, being fairly easy. Things like symbols for POIs and road number shields are more complicated, but you can get quite far without those.
I'm not quite a designer, but definitely interested in map styling. However, I'm more likely to try to get my own online tileserver running and style that with the Mapbox GL JSON style format.
POIs were mainly taken from official openstreetmap style. There are probably several other sets that can be used. But you are right, its relatively simple task. Just needs someone to work on it with an eye on design. Let's see if someone will step up.
I am preparing for the next release that would alter Mapnik SQLite database schema. As a result, the new database will be incompatible with the current OSM Scout Server. So, if you need the maps now, please download them.
I expect to finish map imports and upload them to data.modrana.org in the next 24-48 hours. Within this timeframe the corresponding builds will be made and uploaded to OpenRepos/Harbour. For Harbour users: expect the new version appearing with a delay due to review by Jolla.
This timeline will hold assuming that there will be no surprises during the roll-out of the maps nor something will come out during the final testing.
As for Mapnik style development, I have reworked the current one including POI appearance and other details. The base is now OSM Bright version TM2.
Turned out that many map styles cannot be used as a base due to the copyright, including Mapbox Streets, for example. However, if someone wants to design a new style, especially color schemes, we could make it as follows. You could use Mapbox Studio or some other GUI to design it (please read fine print regarding the style that you want to use as a base, it should allow us to use the style without Mapbox servers; acknowledgment is fine - can always be added into About). When ready, let me know and share your Mapbox GL JSON style (including license of your work). I will then try to translate it into Mapnik stylesheet. While not optimal, it allows us to use simple GUI tools. Easiest would be to adjust just colors. Fonts and appearance of the roads, POIs at different levels are possible to change, but not that simple since it may require changes in map database.
The new release is out - 0.9.5, followed by 0.9.6.
In addition to updated translations (thank you for all involved), this release changes were related to polishing Mapnik backend. Via a simple patch, I have added support for TWKB geometry encoding in Mapnik/SQLite. This allowed to reduce the size of the Mapnik databases by roughly 2x times. Note that it maybe at expense of CPU usage during reading, but I did not notice any slowdown due to TWKB packing.
Second part of enhancements are related to the style of Mapnik rendering. I profiled rendering and adjusted stylesheet and underlying database layout to speed it up. However, on slower hardware and using small tiles, as with modRana, there might be some problems (see https://github.com/M4rtinK/modrana/issues/201 , possible fix waiting in review).
While Mapnik databases are large, further reduction of them would require investigation of alternative formats and possibly writing an additional input plugin for Mapnik. Taking into account that the current database file size seems to be smaller than "the-next-big-thing" vector tiles, reduction of the file size would be non-trivial project.
As for the map style, I am looking for someone who would like to work on them. As mentioned earlier, I'll be happy to adapt the styles from other formats, if available.
With the respect of further development, I was planning to extend search support to all available countries next. Also would have to update underlying libosmscout library to catch up with the development.
Would be great to know what are the further expectations of the users and what should be done in future with the respect of the server. See https://github.com/rinigus/osmscout-server/issues for current TODO list.
PS: 0.9.6 fixed a typo in Mapnik style that slowed down at z<9
Looks like something went terribly wrong while importing the planet. At least, Latvia got roads and nodes from all over Europe. I suspect that planet.pbf was updated while I was downloading it leading to a very strange import for at least that country. Please let me know if you see errors in the territory that of interest for you. If its just few countries, I can update country-by-country faster. Latvia is uploaded now and will be probably in the data.modrana.org server in few hours.
I will redownload the planet and make a new import in the next few days.
Edit: Guatemala and Latvia are fixed and available on the maps distribution server.
@rinigus. Thanks. It's good for my two regions.
karlos devel
2017-05-04, 18:58
Edit: Latvia is fixed and available on the maps distribution server. Guatemala fixed, being uploaded
Many thank @rinigus
ferlanero
2017-05-05, 00:27
Thank you very much for this piece of gold for Sailfish offline navigation. I have no words to be grateful for your work! Thank you very much @rinigus !
@ferlanero: thank you for your kind words. Note that large work is made by everyone working on the client part of the solution. Without excellent GUI clients, there would be no offline solution!
The new databases for countries and territories are uploaded to data.modrana.org. So, all of you who had problems with the previous upload, please check if its resolved. For countries that I tested with, it all seemed to be fine. In addition, planet download was checked against MD5 in the scripts now, which should improve robustness of the import. For those, who did not have any problems, please wait few days to avoid majorly stressing the data.modrana.org over the weekend :) . Please let me know if there are still bugs with the import.
Now I wonder if anything has to be changed in Mapnik style? Comments/suggestions?
karlos devel
2017-05-07, 22:07
You are right @rinigus. Big thank, for all those who have made navigation possible in SFOS too @MartinK @otsaloma
Thank you very much for this piece of gold for Sailfish offline navigation. I have no words to be grateful for your work! Thank you very much @rinigus !
+1, the work you (including every people involved - server/clients) are doing is unbelievable, you are basically almost bringing to Sailfish a native solution for the most voted TJC feature! :cool: (almost because no voice, but I personally don't need it). It can really help promoting Sailfish.
One thing I've noticed while trying it is that the search results are totally not relevant when libpostal is not enabled. Wouldn't it make sense to enable it by default and invite the users to chose their search language(s) on first start?
+1, the work you (including every people involved - server/clients) are doing is unbelievable, you are basically almost bringing to Sailfish a native solution for the most voted TJC feature! :cool: (almost because no voice, but I personally don't need it). It can really help promoting Sailfish.
There are still few things missing on server side - search and routing between all maps, for example - but we'll get there. The voice navigation (for me its actually rerouting that's most interesting) would surely come. As far as I remember, these features are in Poor Maps and modRana issues (aka TODO) lists.
One thing I've noticed while trying it is that the search results are totally not relevant when libpostal is not enabled. Wouldn't it make sense to enable it by default and invite the users to chose their search language(s) on first start?
That's a good suggestion. In general, I probably would need to add a wizard of some sorts that would help to setup the server. I was reluctant to enable libpostal by default (same with Mapnik) as it requires about 700 MB extra storage. Now the new version of libpostal may reduce that requirement, but I would have to move to the new version and test it properly first.
I'll add your suggestion into the issues of the server and will take a look into it a touch later. Its a great idea and should be implemented when I polish the interface. Right now, I am trying to get all the required features covered first, if I can do it in reasonable amount of time.
jdrescher
2017-05-12, 12:27
Hello guys,
I am currently developing a sport tracker app for SFOS.
I would like to use OSM Scout Server for providing map informations but unfortunately it does not work.
This is what I tried:
Map {
id: map
plugin: Plugin
{
name: "osm"
PluginParameter
{
name: "useragent"
value: "Laufhelden/0.0.1 (Sailfish)"
}
PluginParameter { name: "osm.mapping.host"; value: "http://localhost:8553/v1/tile/" }
}
}
...
The map from the server seems not to be found and instead is trying to use WIFI connection to download the map.
The server is running nicely and works with modRana and Poor Maps.
You can find the whole code here: https://github.com/jdrescher2006/Laufhelden/blob/master/qml/pages/DetailedViewPage.qml
Hello guys,
I am currently developing a sport tracker app for SFOS.
I would like to use OSM Scout Server for providing map informations but unfortunately it does not work.
This is what I tried:
Map {
id: map
plugin: Plugin
{
name: "osm"
PluginParameter
{
name: "useragent"
value: "Laufhelden/0.0.1 (Sailfish)"
}
PluginParameter { name: "osm.mapping.host"; value: "http://localhost:8553/v1/tile/" }
}
}
...
The map from the server seems not to be found and instead is trying to use WIFI connection to download the map.
The server is running nicely and works with modRana and Poor Maps.
Nice to see it used in map-related applications and I can only hope that it would spread :) .
I presume that you have OSM Scout Server running on the background while using the tracker app, right?
From the look of it, you are using QML Map type. I don't have experience with it, unfortunately. Maybe @otsaloma can help, or reading his Poor Maps code which seems to be using the same plugin. From reading Poor Maps MapPlugin, it looks like @otsaloma is using own server that probably re-routes requests further. Notice that URL syntax seems to be different. But again, I never used it in QML myself, others are much more qualified to help you out.
On the server side, you could check out whether it is accessed. For that, in server settings, enable "Log info messages". Then, while accessing the map in your application, check whether requests arrive to the server (they should be shown in the main window under Events.
If it turns out that Map prefers some other URL scheme than the one used in the server currently, we can add some simplified URL scheme under
http://localhost:8553/v1/tile_simple/
or something similar. That's very easy.
otsaloma
2017-05-12, 17:55
I would like to use OSM Scout Server for providing map informations but unfortunately it does not work.
SFOS still has QtLocation 5.2. That "osm.mapping.host" parameter was added (https://bugreports.qt.io/browse/QTBUG-32937) in 5.6. Documentation for QtLocation 5.2 is not officially available, but maybe Ubuntu 15.04 (https://developer.ubuntu.com/api/apps/qml/sdk-15.04/) might be close. But note that Jolla has backported individual commits and updated individual things like the HERE plugin, so it's understandably quite difficult to figure out.
What you're trying to do is (to my knowledge) simply not possible with the SFOS version of QtLocation. My advice would be to wait for QtLocation 5.6 (maybe we need a new community IRC pressure meeting?) or if you can't wait, ask Martin about his pure QML component. Poor Maps uses QtLocation's Map component, but has custom tile loading to work around stuff like this, but that's not designed to be reusable.
SFOS still has QtLocation 5.2. That "osm.mapping.host" parameter was added (https://bugreports.qt.io/browse/QTBUG-32937) in 5.6. Documentation for QtLocation 5.2 is not officially available, but maybe Ubuntu 15.04 (https://developer.ubuntu.com/api/apps/qml/sdk-15.04/) might be close. But note that Jolla has backported individual commits and updated individual things like the HERE plugin, so it's understandably quite difficult to figure out.
What you're trying to do is (to my knowledge) simply not possible with the SFOS version of QtLocation. My advice would be to wait for QtLocation 5.6 (maybe we need a new community IRC pressure meeting?) or if you can't wait, ask Martin about his pure QML component. Poor Maps uses QtLocation's Map component, but has custom tile loading to work around stuff like this, but that's not designed to be reusable.
I must say that situation with QtLocation on SFOS is frustrating. Its probably a naive question but I am going to ask it anyway: now, when we have Qt5.6, is it possible to compile missing QtLocation 5.6 and add it ourself via OpenRepos or other distribution mechanism? @otsaloma, you have been following for a while, do you know whether it is possible?
To keep you in the loop and give an overview of the server development:
On the server side, I have been working on making it simple to use it with multiple maps. Rendering of the maps has been resolved already with Mapnik. Now I have added support for search using the data from all available maps. This is done for geocoder-nlp backend.
Finally, I think I know how to get routing through multiple maps supported as well. Namely, I am planning to incorporate Valhalla (routing engine behind Mapzen routing solution). After few days of porting, I managed to run the first routing tests using valhalla's program on SFOS device. Calculation of a route from Stockholm (Sweden) to Aarhus (Denmark) took just 2 seconds on Nexus 4. That's compared to ~45 seconds on the same device on the same route when using libosmscout. Not sure whether valhalla is always faster, but the start is promising. In addition to speed, valhalla routing engine allows us to "glue" the countries/territories together solving the long-standing issue of routing between countries.
There are several technical issues to be resolved with the adaptation of Valhalla, but it should be possible. Also we would have to discuss with @otsaloma and @MartinK few technical choices on how to do that when I know a bit more about Valhalla requirements.
I might make one more release before adding Valhalla to push out support for search in all maps and catch up with the advances in libosmscout.
otsaloma
2017-05-13, 14:53
I must say that situation with QtLocation on SFOS is frustrating. Its probably a naive question but I am going to ask it anyway: now, when we have Qt5.6, is it possible to compile missing QtLocation 5.6 and add it ourself via OpenRepos or other distribution mechanism? @otsaloma, you have been following for a while, do you know whether it is possible?
I don't actually know if it's possible, but looking the commit activity (https://git.merproject.org/mer-core/qtlocation/activity), there's a lot of fixes related to e.g. geoclue positioning, and just using upstream 5.6 might not work well on all devices and trying to port those patches to 5.6 might be difficult. Additionally, there are API changes (http://doc.qt.io/qt-5/qtlocation-changes.html) between 5.2 and 5.6, which means upgrading would break many apps written for 5.2. The API changes are small and it's easy to adapt to them, but it might be difficult for app authors to support both versions.
Also note, that this problem of using using a custom tile source, e.g. OSM Scout Server, is only partially served by the OSM plugin in QtLocation 5.6. It allows you to set the host, e.g. "http://tile.openstreetmap.org/" in "http://tile.openstreetmap.org/{z}/{x}/{y}.png", but the requested URL will always end in "{z}/{x}/{y}.png". What we'd really need is a plugin that supports a template URL string, so that you can have X, Y and Z as URL parameters instead of path components or additional parameters like day/night or API keys etc. And also, a plugin parameter for tile size so that the requested tiles can be shown correctly. A custom QtLocation plugin would be easy to distribute separately, e.g. at OpenRepos, but probably difficult to use in Harbour apps.
I don't actually know if it's possible, but looking the commit activity (https://git.merproject.org/mer-core/qtlocation/activity), there's a lot of fixes related to e.g. geoclue positioning, and just using upstream 5.6 might not work well on all devices and trying to port those patches to 5.6 might be difficult. Additionally, there are API changes (http://doc.qt.io/qt-5/qtlocation-changes.html) between 5.2 and 5.6, which means upgrading would break many apps written for 5.2. The API changes are small and it's easy to adapt to them, but it might be difficult for app authors to support both versions.
Also note, that this problem of using using a custom tile source, e.g. OSM Scout Server, is only partially served by the OSM plugin in QtLocation 5.6. It allows you to set the host, e.g. "http://tile.openstreetmap.org/" in "http://tile.openstreetmap.org/{z}/{x}/{y}.png", but the requested URL will always end in "{z}/{x}/{y}.png". What we'd really need is a plugin that supports a template URL string, so that you can have X, Y and Z as URL parameters instead of path components or additional parameters like day/night or API keys etc. And also, a plugin parameter for tile size so that the requested tiles can be shown correctly. A custom QtLocation plugin would be easy to distribute separately, e.g. at OpenRepos, but probably difficult to use in Harbour apps.
@otsaloma, thank you very much for this explanation! I don't know much about it, but would try to read up on QtLocation in future.
What's telling is that there has been no activity visible with QtLocation 5.6. From this maybe we can conclude that we should not expect situation change in a short term.
otsaloma
2017-05-14, 01:44
What's telling is that there has been no activity visible with QtLocation 5.6. From this maybe we can conclude that we should not expect situation change in a short term.
Yes, that's likely true. I get the impression that Jolla has been busy with Xperia and other adaptations, leaving that QtLocation upgrade for later.
jdrescher
2017-05-18, 12:15
Thanks for the infos.
I have a lot of features to implement so I can wait a little longer.
All in all for developers it's still a sad situation after so many years with SFOS. You can't even release a bluetooth app into the harbour.
I think Jolla should make it as easy as possible for developers. Instead they are doing it the other way around :mad::eek:
vBulletin® v3.8.8, Copyright ©2000-2025, vBulletin Solutions, Inc.