[SailfishOS] [Announce] Native offline maps: OSM Scout Server [Archive] - Page 2 - maemo.org

MartinK

2017-06-04, 18:04

My advice would be to wait for QtLocation 5.6 (maybe we need a new community IRC pressure meeting?) or if you can't wait, ask Martin about his pure QML component. Poor Maps uses QtLocation's Map component, but has custom tile loading to work around stuff like this, but that's not designed to be reusable.
There have been two main reasons for using a custom QML map component:
- lack of functionality & unconvincing performance of QtLocation
- QtLocation not being Jolla Store compatible and would thus be preventing modRana from being distributed via the Jolla store

Number one likely is less of an issue nowadays due to the ongoing work on QtLocation but it's quite a shame number two still stands after all those years and complicates application development if any mapping/navigation functionality is required.

In any case this is what modRana is using:
https://github.com/M4rtinK/modrana/blob/master/modules/gui_modules/gui_qt5/qml/PinchMap.qml
It's a QML/Javascript component for efficient display of tiled maps with overlay support. Feel free to use it if you want (it's GPLv3) but I have to note it's not fully standalone and you would likely have to also take the related modRana subsystems as well. For example map tile download and caching is done by a Python module, etc. I have tentative plans to make QML map component more standalone and more easy to reuse in other application by I haven't really got to that.

For that reason it might make sense to also take a look at the QML map component used by Foursail:
https://github.com/Maledictus/foursail/blob/master/qml/PinchMap.qml

It should be quite similar to the modRana PinchMap (both modRana and Frousail projects started with the old PoinchMap.qml from AGTL), a bit less advanced and possibly quite a bit more standalone. :)

rinigus

2017-06-04, 18:17

Thanks. It would be better to install the osmscout-server dependencies without icons rinigus?

I presume you are talking about modules, right?

It would be better if I could use normal linux-style dependencies, but there are limitations when we want to have apps in the Harbour. Another limitation is imposed by RPM (or more specifically, the way we install them).

1. If I want to publish via Harbour, I cannot have multiple executables in one package. I can add as many libraries as I want, as much data, but not executables. I maybe able to smuggle one as a zipped file and unpack on a start, but that becomes iffy from the security point of view.

Valhalla is provided as its own executable that is run by the server if needed (if user wants to choose it). In the end, this will allow me to keep Valhalla up-to-date with relatively small time demand. So, I needed to have a mechanism for delivering multiple executables.

2. If we forget about Harbour and go only through OpenRepos then I'll be able to install the modules as dependencies. However, while highly unlikely and extremely hypothetical, imagine, if someone wants to temporarily uninstall such a brilliant software as OSM Scout Server is. In this case, due to the lack of "apt-get autoremove" (or similar alternative in almost all other Linux package managers), users would be stuck with these dependencies on their devices. On my device, OSM Scout Server and its modules, take the top spot in /usr/share, then place #3 and #7. These are big packages and users should have an ability to simply remove them.

As for extra icons, I would suggest to put modules into some folder far away and not touch them unless you need to uninstall them for one reason or another.

rinigus

2017-06-05, 20:36

I have just released a new version: 0.10.0

In short, this release is one of the major ones. This release has rather large amount of new features, so I'll try to address them one by one:

Search covering all databases

If you use Geocoder-NLP for search, its performed over all countries or territories that you have on device. Search is started from the map selected in GUI and later the data in other databases is considered. If you want to limit search to only one database, see Geocoder-NLP settings.

New backend: Valhalla

This is a major new functionality. I have ported Valhalla to SFOS. Valhalla is a routing engine behind Mapzen routing service. Now you could use it offline on your devices. Its fast, allows you to route between countries, and there are many other interesting features that are or could be easily made available to client programs. Main problem has been on how to distribute the datasets for Valhalla. For that, packaging the datasets and their download support had to be written as a part of the new release. Hence it took a while to implement it.

Since Valhalla is a full-blown server by itself, the communication between Valhalla and the client uses Valhalla's own protocol. Clients have to make adjustments to support it. Right now, before these changes are done in the clients, use libosmscout for routing.

Poor Maps and modRana do not support Valhalla backend yet. For Poor Maps, support is "simple" by making a copy of Mapzen Turn-by-Turn plugin and adjusting its URL + used geocoder. For modRana its probably a bit more complicated, unfortunately. Please wait for corresponding release and the new routing options in the clients.

Since its a new backend, use unsubscribe/subscribe for all the countries and territories that you update. As you did with Mapnik.

Libosmscout

The backend has been updated and new version of the maps has been uploaded to the data.modrana.org for distribution. This version should play better with OSM Scout by @Karry. But note that full compatibility is never guaranteed.

Storage of libosmscout datasets is also made optional. If yo don't use it, you can now opt out.

Modules

This release is the first one that uses modules. There are two modules: Fonts and Route. Fonts are packaged separately to avoid huge downloads on every update.

Fonts module contain Noto fonts by Google and should be able to display all possible languages, to my knowledge. It should be surely more than native SFOS fonts are supporting (rejoice people of Sri-Lanka).

Route module contains Valhalla routing engine that had to be distributed as a separate executable to allow separate updates of Valhalla and the server and to comply with Harbour rules.

Please install the modules with this release. If you don't have the module installed, a warning is displayed in the main view of the server.

Translations

There are two targets for translations now - OSM Scout Server and Valhalla. Valhalla's routing instructions come from Valhalla itself. So, the client programs have to request the corresponding language using Valhalla API. Since we are using Sailfish, ability of selecting en-US-x-pirate locale would be great to get authentic Sailfish navigation experience. I hope that Poor Maps and modRana could implement such selection :) .

As for translators, Valhalla's routing instructions are translated via Valhalla project. This is not done via transifex, but some other arrangement. See translations README for updated pointers (https://github.com/rinigus/osmscout-server/blob/master/translations/README.md)

For translators: I'll be happy to release as many point releases as needed. So, we could expect several 0.10.x releases with the updated translations.

Current state and upcoming new default

To put things into perspective: As with libpostal, to my knowledge, SFOS is the first mobile platform supporting Valhalla in offline mode. These are major libraries in the field and I do wonder, when the clients adopt Valhalla, what is the real life performance of SFOS maps vs others (Android, iPhone)...

As a part of the discussion with @otsaloma and @MartinK, it was suggested to select a sensible default backend(s) and mainly support that. I am mainly using Mapnik / Geocoder-NLP / Valhalla myself and I plan to develop and support this combination of backends. As soon as the adaptation of Valhalla by clients will be ready, its planned to make Mapnik / Geocoder-NLP / Valhalla combination as a default. Note that this combination allows you to render, search, and route between all the maps on device (if the maps are for the territories next to each other, though).

Libosmscout backend is there to stay. If you wish you could still use libosmscout, but please note that its testing would be limited. Libosmscout has a smaller datasets, which is currently its main advantage.

What's next

Before getting 1.0 out, I would like to mainly address the following.

* Helping clients to work with Valhalla backend. Poor Maps is easy on paper since it has support for the protocol already. I just don't know how to make it so that there would be minimal code duplication between OSM Scout and Mapzen routers. modRana is trickier for me - I am not familiar with its implementation of routing.

* Bugfixes. Please report bugs, I'll try to fix them as soon as I can

* Polishing. I am planning to mainly polish the experience. It should be easy for users to start using offline maps. So, if you have ideas on how to make it simple, wish to write documentation, howto guides, or just record all steps that you need to make it work on a fresh install. I am planning to make a small wizard on the first start to help with the selection of backends and settings sensible defaults.

* Night mode for Mapnik. Mainly design of the style. Default style adjustments / comments are always welcome.

* Geocoder-NLP does not support "Nearby" search requests. I am planning to implement that as well.

PS: Hopefully I haven't forgotten anything.

otsaloma

2017-06-05, 22:14

* Helping clients to work with Valhalla backend. Poor Maps is easy on paper since it has support for the protocol already. I just don't know how to make it so that there would be minimal code duplication between OSM Scout and Mapzen routers.

It's such a small change to Mapzen's online Valhalla, that I can do the Poor Maps implementation myself. And I'll probably just duplicate the code, it's fine. I'll ping you on GitHub so you can check the work. But, I'll want to do some testing too -- I'll probably have the time to do it over the weekend.

eson

2017-06-06, 07:14

Translation and some testing done...

If not a Valhalla issue, there is a missing translation string. Maybe also a bug? Can't really test it.
.

rinigus

2017-06-06, 07:26

Translation and some testing done...

If not a Valhalla issue, there is a missing translation string. Maybe also a bug? Can't really test it.
.

Thank you very much!

That's Valhalla's output, unfortunately. So, we have to translate that through Valhalla.

What the server does is starting Valhalla as a separate process. Then it catches the output and forwards it to INFO stream. Unfortunately, Valhalla is a rather chatty beast and would be telling a lot of details when calculating the route. Maybe we should just ignore these messages and show on GUI only if requested by the user? Let's see what is the reaction of the users when it all gets supported by the clients.

PS: This WARN message is harmless since we don't use extracts in such form

eson

2017-06-06, 07:32

That's Valhalla's output, unfortunately. So, we have to translate that through Valhalla.

Ok, thanks! I will take a look at Valhalla translation, but as Swedish translation isn't even started, it will take some time. Lots of complicated strings to translate.
Think I'll need client support to get a grip of it all.

rinigus

2017-06-06, 07:41

Ok, thanks! I will take a look at Valhalla translation, but as Swedish translation isn't even started, it will take some time. Lots of complicated strings to translate.
Think I'll need client support to get a grip of it all.

Yes, and as mentioned earlier, client support is coming :) .

Before getting into translating Valhalla, its probably good to add github issue in their project (https://github.com/valhalla/valhalla). Valhalla's developers are quite fast to respond, but they do it during workdays - its their day job! As I could see in the issue tracker, Valhalla is considering moving translations into a separate platform as well. It would be good to tell your plan to translate via the issue and, I am sure, they will help you with the process.

eson

2017-06-06, 07:59

Yes, and as mentioned earlier, client support is coming

Do you know a nice desktop client (linux) I can use to play with Valhalla?

rinigus

2017-06-06, 08:45

Do you know a nice desktop client (linux) I can use to play with Valhalla?

No, not really. modRana is not supporting it yet. I played a bit with the leaflet, but its not ready for mobile release. Maybe I'll look into it soon again.

On Linux (or any PC), you could subscribe to Mapzen developer id (should be free as long as you don't exceed their limits) and make a small web-page with leaflet. See https://github.com/valhalla/valhalla-docs/blob/master/add-routing-to-a-map.md for tutorial. Note that you don't need any web-server, just point the browser to a corresponding HTML file.

rinigus

2017-06-06, 20:07

For a fun of it, I benchmarked the routers on the same hardware. I used Nexus 4 and calculated a route from Stockholm to Aarhus (or nearby it). Rote itself is a bit more than 900 km.

Test results:

SFOS, Valhalla: on cold start (boot of the phone), it took 12 seconds from starting calculations in Poor Maps till getting the result. This includes geocoder search result for target and origin, communication between Poor Maps and the server (3 requests, 2x geocoder + 1 route). Out of these, Valhalla used only 5 seconds according to its logs.

SFOS, Valhalla: reroute calculation (Poor Maps navigation cleared, closed, and started again) took 5-6 seconds (2 were used by Valhalla itself). In this case, server was still running after it calculated the route. So, few things were in FS and application caches.

Android, MAPS.ME: Same route, without geocoder since it searches for origin and target separately. For route calculation, it took ... 50 seconds . Recalculation of the route by using a target nearby: 50 seconds again.

Android OsmAnd+: Again, geocoder is out of equation since GPS coordinates are used when calculating route. Here it took about 2 minutes(+-5 seconds).

So, we are getting a rather fast router. It would take some time to get it into clients, but not too long (NB: 'soon' not used). Libosmscout is quite fast as well (I leave its benchmarking for users when Valhalla can be compared against it).

@eson and other translators: Valhalla developers would love to get your help and extra languages on their server.

MartinK

2017-06-07, 02:02

No, not really. modRana is not supporting it yet.
Just wanted to note I am aware of this and it's one of the items on top of my priority list. :)

I plan to do some traveling in the summer, so support for a robust and fast offline router in modRana will certainly be handy. :)

rinigus

2017-06-08, 18:42

0.10.1 is out with the updated translations. Translators, thank you very much for this great job!

ferlanero

2017-06-08, 23:13

rinigus

2017-06-09, 03:55

Excelent work. But I have a problem, when I activate Valhalla as route engine in OSM Scout Server, PoorMaps can't calculate any route. It always reports "No results". What is what I'm doing wrong?

If I disable Valhalla in OSM Scout Server, It works perfect, as expected.

I have SFOS 2.1.0.11 installed.

Thank you.

You don't do anything wrong. Poor Maps and modRana don't support Valhalla in OSM Scout Server yet. Please wait for a new release of Poor Maps and modRana to use it. See
http://talk.maemo.org/showpost.php?p=1528857&postcount=260 http://talk.maemo.org/showpost.php?p=1528916&postcount=268 for details.

otsaloma

2017-06-15, 19:57

Poor Maps 0.29 is out now with Valhalla support.

carmenluci

2017-06-15, 20:46

Poor Maps 0.29 is out now with Valhalla support.

Thanks for the update, routing with Valhalla works perfect!!
Is possible to get routing instruccions in other languages?

otsaloma

2017-06-15, 22:35

Is possible to get routing instruccions in other languages?

Poor Maps should (hopefully) detect your OS language and request navigation instructions in that language. If Valhalla doesn't have translations for your language, see the previous two pages about where the translation is done. If you want to use a different language for navigation and rest of the UI, that's not currently possible. Should it be?

carmenluci

2017-06-16, 04:38

Poor Maps should (hopefully) detect your OS language and request navigation instructions in that language. If Valhalla doesn't have translations for your language, see the previous two pages about where the translation is done. If you want to use a different language for navigation and rest of the UI, that's not currently possible. Should it be?

Valhalla is translated to Spanish language: https://github.com/valhalla/valhalla/tree/master/locales

But I get routing instructions in English.

rinigus

2017-06-16, 05:32

Valhalla is translated to Spanish language: https://github.com/valhalla/valhalla/tree/master/locales

But I get routing instructions in English.

OK, looks like its a bug in Valhalla or Valhalla/Poor Maps interaction. I will look into it.

I am assuming that you are using Spanish in your GUI.

Meanwhile, would you mind checking whether Mapzen Turn-by-Turn fetches the instructions in Spanish for you?

PS: @otsaloma, thank you for the update!

carmenluci

2017-06-16, 06:23

OK, looks like its a bug in Valhalla or Valhalla/Poor Maps interaction. I will look into it.

I am assuming that you are using Spanish in your GUI.

Meanwhile, would you mind checking whether Mapzen Turn-by-Turn fetches the instructions in Spanish for you?

PS: @otsaloma, thank you for the update!

Yes, I use Spanish language in my phone.

If I select 'Mapzen Turn-by-Turn' in Navigation, the instructions are in English too.

rinigus

2017-06-16, 06:30

Re translations: we were using, what looks like, an outdated API. I have fixed the call and submitted corresponding pull request to Poor Maps ( https://github.com/otsaloma/poor-maps/pull/40 ). This seemed to fix routing instructions for OSM Scout Server Valhalla and Mapzen Turn-by-Turn. At least when I called Poor Maps with LANG=es_ES (or de_DE), I've got instructions in what appears to be Spanish (German).

Re separate option for specifying language: Maybe it would be great to have that option which would allow users to choose the second language. I would use it to get "Pirate routing version", but its surely not a high-priority task :) . So, would be nice to have as an option, but not essential with other aspects of navigation probably much more important.

karlos devel

2017-06-16, 16:45

I can not use osm scount server from -v 0.10.0. not load the maps ... i updated the maps in -v 0.10.0 but same. including poormaps-modrana

rinigus

2017-06-16, 17:00

I can not use osm scount server from -v 0.10.0. not load the maps ... i updated the maps in -v 0.10.0 but same. including poormaps-modrana

To help, I need to get more details:

* I assume that you want to view maps, right?

* Which backend do you use, Mapnik or libosmscout?

* If Mapnik, did you install fonts module?

* When viewing with Poor Maps or modRana, do you see jobs submitted to OSM Scout Server?

* What do the logs of the server say?

* Does Map Manager show that your maps are fine (nothing listed as incompatible/needs download)?

karlos devel

2017-06-16, 18:21

* Which backend do you use, Mapnik or libosmscout?
i use mapnik, in libscount look same.

* If Mapnik, did you install fonts module?
yes, 2 modules

* When viewing with Poor Maps or modRana, do you see jobs submitted to OSM Scout Server?
starting sending, but look like not complete a request

* What do the logs of the server say?
the unique error logs is: Tile extract could be not loaded

* Does Map Manager show that your maps are fine (nothing listed as incompatible/needs download)?
yes

rinigus

2017-06-16, 20:13

Let's focus on Mapnik first. I assume you use Guatemala, right?

* When viewing with Poor Maps or modRana, do you see jobs submitted to OSM Scout Server?
starting sending, but look like not complete a request

So, do you see requests as INFO messages in the server's Events log?

* What do the logs of the server say?
the unique error logs is: Tile extract could be not loaded

This is Valhalla's message and its nothing to worry about (I am planning to cleanup messages a bit later).

OK, if there are no "strange" messages, then maybe I could take a look on full log. For that, go to Settings/in the end of Settings, select "Log messages into session log file" and select "Log info messages" above it/Accept. Close the server, open it up again. Go to Poor Maps, clear cache, close Poor Maps, open Poor Maps and try to move around on the map.

While you move, the server should have displayed number of jobs and that there are requests coming in as Events on the main window. If you still don't have any map, close the server and send me the file

.cache/harbour-osmscout-server/harbour-osmscout-server/session.log

from your phone. Probably best is to use pastebin (or alike, https://ptpb.pw/f ) with some reasonable expiry date (few days should be fine). Then I can check the logs and see if something comes up.

* Does Map Manager show that your maps are fine (nothing listed as incompatible/needs download)?
yes

There is a bug which I should fix and may have something to do with your problem. Do you have world coastlines and language parser downloaded?

Maybe the error is somewhere else. I have just tested Guatemala on my device and it shows the map fine. So, its probably not an importer bug (if you use that country map).

karlos devel

2017-06-16, 20:31

rinigus I traveled here for a few days, check the map ("georgia US" it is not work) guatemala map work nicely.

thanks

rinigus

2017-06-16, 20:46

rinigus I traveled here for a few days, check the map ("georgia united states" it is not work) guatemala map work nicely.

thanks

Georgia (US) has very suspiciously small Mapnik database (3MB) when compared to OSM Scout (165MB). So, something has happened with its import. And, indeed, on testing, map is rather "sad" over there. I'll take a look into it (will file an issue in github).

rinigus

2017-06-17, 07:32

For those facing issues with RAM consumption in Jolla One, see https://github.com/rinigus/osmscout-server/issues/153 ( @a-dekker thank you for reporting and feedback). Try to reduce cache size for Valhalla and see if it helps

rinigus

2017-06-17, 10:12

rinigus I traveled here for a few days, check the map ("georgia US" it is not work) guatemala map work nicely.

thanks

Its possible that it was "an operator" error, i.e. mine, during the import. I might have stopped Georgia import accidentally. I'll write a script to check for such strange imports and maybe I can catch these cases before publishing them.

Anyway, when I re-imported Georgia, all seemed to be fine. Any other territories that are problematic?

If no-one needs Georgia/US right now, I would update it together with the full update later, probably in 2-3 weeks. So, if there are some places which are imported wrongly and you need them before, let me know.

rinigus

2017-06-17, 19:59

From the analysis of the import sizes, it looks like the following territories had issues with Mapnik import:

europe/france/provence-alpes-cote-d-azur
north-america/canada/new-brunswick
north-america/canada/nunavut
north-america/canada/ontario
north-america/canada/quebec
north-america/canada/saskatchewan
north-america/mexico
north-america/us/georgia
north-america/us/idaho
north-america/us/indiana
north-america/us/kentucky
north-america/us/maine
north-america/us/montana
north-america/us/north-carolina
north-america/us/ohio
north-america/us/oklahoma
north-america/us/region/midwest
north-america/us/wyoming

On the next planet import, I'll check for Mapnik import errors by comparing import sizes. If I have missed anything, please let me know. If you need these maps within 2 weeks, please tell me.

karlos devel

2017-06-17, 23:20

Yes i need georgia US fixed almost right know ... if do , Is appreciated

rinigus

2017-06-18, 07:40

Yes i need georgia US fixed almost right know ... if do , Is appreciated

Should have guessed that myself. I am uploading and, hopefully, timezone differences would work for the benefit this time. It would take few hours, maybe 6-8, until all is in data.modrana.org due to the distribution model that we use. I'll update this post when its all up and ready.

Edit Uploads done and Georgia (US) should be fine. Go to Map Manager, press "Check for updates", you should get the message with Georgia listed as a candidate for updates; accept and wait till it finishes the download. Hopefully, not too late.

karlos devel

2017-06-18, 15:37

rinigus

2017-06-18, 15:59

\0/ Thank you: everything work perfectly. I will also say that with the integration of Mapnik and Valhalla, it looks same that "Here Map" or others.

I also want to add that some is missing in PoorMap ,when i search a POI PoorMap has give first result according to the location a gps, will usefull.

Well, intension is to make it as good as any, if not better.

Re POI search: I added POI search support to the geocoder-nlp and it will be available with the next release update (0.11.x).

karlos devel

2017-06-18, 16:36

Re POI search: I added POI search support to the geocoder-nlp and it will be available with the next release update (0.11.x).

Tnk , but add POI search support to the geocoder-nlp only resolve POI the case sensitive right? then i am talk about ,that if i search something(POI) the first results of the search would have to be according to the GPS location in PoorMap. Modrana is already Implemented this features. I think this feature corresponds to PoorMap project because is responsible for handling the GPS, not the server i guess. Note: I'm talking about PooMaps and ModRana because somehow are related to this thread indirectly . . .

rinigus

2017-06-18, 16:51

Tnk , but add POI search support to the geocoder-nlp only resolve POI the case sensitive right? then i am talk about ,that if i search something(POI) the first results of the search would have to be according to the GPS location in PoorMap. Modrana is already Implemented this features. I think this feature corresponds to PoorMap project because is responsible for handling the GPS, not the server i guess. Note: I'm talking about PooMaps and ModRana because somehow are related to this thread indirectly . . .

Poor Maps does support it. You go to main menu, select "Nearby" and then search next to your location. Note that while you are searching for "type", its actually looking for names as well in online backends, if I understand it correctly. Starting from 0.11.x, the same will be for OSM Scout Server.

rinigus

2017-06-24, 07:26

I am planning to push soon for a next update. Since it would change the database format for the geocoder, the service could be disturbed for 1-2 days after the maps hit the data.modrana.org server. If you need the maps working within this period, please ensure that you have them on device.

I am planning to start pushing new update later today, unless anything else comes out during final testing.

Edit: Please do not download maps right now, until 0.11.x release. I have to re-upload Valhalla's tiles due to a mistake in import and that may take some time. Right now, due to Valhalla's packages inconsistency, the downloading could result in a forever loop.

rinigus

2017-06-25, 18:19

New release - 0.11.1 - is out together with the corresponding maps update.

The main changes are "under the hood". Geocoder-NLP has now support for reverse geocoding (rather primitive-but-works approach for simple searches). This required database format adjustments, so only compatible database can be used (available on data.modrana.org). As a result, Guide/Nearby search should work with this geocoder and you don't have to rely on libosmscout for that. Note that while you are looking for "type", it searches also "name" record in the database with all libpostal goodness.

An important change was adaptation of Valhalla's router request API to get routes by libosmscout. As a result, the map clients can have just one routing provider corresponding to OSM Scout Server and send requests / process replies via single code. For users, it means that they just need to specify routing engine in the server and the client will figure it out. Upcoming Poor Maps (0.30.x) would have support for that. With modRana, we would have to implement Valhalla's API first and then join the current OSM Scout Server processing with the new one.

Few changes were introduced to help users to avoid situations that could lead to excessive RAM use. I would like to thank here @a-dekker for help!

Valhalla's limits for maximal distance between target and origin were reduced from defaults for walking and cycling. These limits can be adjusted under Settings/Valhalla. Note that as a result of these changes, Router module 2.2.8 requires the server 0.11.x or later.

In addition, if you have not selected languages for parsing by Geocoder-NLP, you will be warned about it on the main screen.

The start of the server has been changed with the delayed Mapnik processing. It should start faster, but the required processing will delay your first rendered tile. This change is required for supporting day/night styles as well as other styles suggested by @Thra11 (issue https://github.com/rinigus/osmscout-server/issues/150).

Translators have been really fast and I would like to thank them all! Some strings may have not made it into this release, but we can make a new one as soon as more translations will be finished.

Special thanks and attached screenshot for our own Valhalla's translator - @eson! With the upcoming 0.30 Poor Maps localized Valhalla routes have been requested and you should be able to use your translation as well.

For those who cannot wait and have Linux desktop - Poor Maps (and probably modRana) are really easy to package yourself. Just clone the source and run "make rpm". You may require rpm, but it should be available on non-rpm distros as well (running on Gentoo without problems).

eson

2017-06-26, 06:52

@rinigus I made a fresh start with last release and found the text shown below, untranslated.

Btw... Swedish translation's looking good on your screen. Looking forward to try it out.
.

rinigus

2017-06-26, 07:15

@rinigus I made a fresh start with last release and found the text shown below, untranslated.

Btw... Swedish translation's looking good on your screen. Looking forward to try it out.
.

It looks like this string is untranslated (just English version copied over to the Swedish one). It was introduced at commit https://github.com/rinigus/osmscout-server/blob/227302fc56b6593f918df7e1bbbc17b5dee9f84f/translations/harbour-osmscout-server-sv.ts and I can see the same for String 344 in transifex.

Don't know if its some bug that I introduced or it came from transifex...

eson

2017-06-26, 07:37

OK, fixed it at Transifex, along with some minor adjustments.
It might even be my own mistake, although it is not very likely. :rolleyes:

otsaloma

2017-06-26, 23:50

An important change was adaptation of Valhalla's router request API to get routes by libosmscout. As a result, the map clients can have just one routing provider corresponding to OSM Scout Server and send requests / process replies via single code. For users, it means that they just need to specify routing engine in the server and the client will figure it out. Upcoming Poor Maps (0.30.x) would have support for that.

Poor Maps 0.30 is out now.

eson

2017-06-27, 04:48

Poor Maps 0.30 is out now.

Works fine with OSM Scout Server/Valhalla, so far. Thanks!

rinigus

2017-07-01, 07:33

New release is out - 0.12.0

This release introduces Mapnik styles (@Thra11, thank you for suggestion) and night mode as well. The available styles are "default" and "car" . The styles are almost the same with the gas stations dropped in pecking order for the default and shown at lower zooms for car. However, the base is there and, if someone is interested, its possible to adjust available styles to show pedestrian/bicycle/transport layers more prominently (or coffee shops, for that matter).

For styles to show up in the clients, one has to add them into the list of basemaps. Which brings me to the question: Poor Maps users - do you use @1 OSM Scout Server layer or should we just remove it and simplify the matter by keeping @2 only? Note that larger tiles lead to more efficient rendering.

From this release, Mapnik supports day/night styles as well. It should be all configured and supported by modRana and Poor Maps via different basemaps already.

This release fixes a bug that slowed down Mapnik rendering by over sizing buffer around each tile. It should increase rendering speed for all clients. The increase would be more significant for clients requesting small tiles, such as modRana.

rinigus

2017-07-05, 20:16

The new release is out - 0.13.0, code name Beta 1

This release is mainly about polishing GUI and simplification of configuration settings.

This release introduces profiles that allow you to simply select the backends that you wish and configures storage and settings accordingly. Default profile is set to Mapnik/Geocoder-NLP/Valhalla. So, here is a shift of the default settings as suggested earlier in this thread. You can still select each backend individually if you wish by setting profile to Custom.

On the first start (and should be only for one time), you are greeted with the wizard that guides you through selection of profiles and languages for address parsing. If the wizard comes back on other program starts as the small-town leader of French resistance (only for those who get the joke), please file it as a bug.

I have implemented all the functionality that I was planning for 1.0 release. The remaining bugs/issues that I am aware of are at https://github.com/rinigus/osmscout-server/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.0 . In addition, I may work on https://github.com/rinigus/osmscout-server/issues/163 before making 1.0 release. If there is something more that should be adjusted before that release, please let me know.

I hope that you enjoy using it!

Edit: 0.13.1 has updated translations and a bugfix related to POI search (thank you @MartinK for reporting it).

rigow

2017-07-10, 18:09

Hi all,

I get a corrupted file when downloading "europe-france" which is a pity after the 9Gb download. In fact the mapnik.sqlite is broken, said the log file.

rinigus

2017-07-10, 18:28

Hi all,

I get a corrupted file when downloading "europe-france" which is a pity after the 9Gb download. In fact the mapnik.sqlite is broken, said the log file.

Thank you for reporting, I will look into it ASAP. Until its resolved, I suggest to get regions of France that you are interested in. It may even end up as a faster solution for rendering than getting France as a single package. As it is, map drawing, search, and routing all support databases that are split into sub-regions in the default profile (mapnik/geocoder-nlp/valhalla).

I'll get back here when I get France downloaded and tested.

rinigus

2017-07-10, 19:50

Hi all,

I get a corrupted file when downloading "europe-france" which is a pity after the 9Gb download. In fact the mapnik.sqlite is broken, said the log file.

I just managed to download France on PC and mapnik seems to be fine with France nicely drawn. So, the file on the server shouldn't be corrupted.

What's exactly in the corrupted message? Does it state that the territory is subscribed or that it still needs to download it?

If you still need to download it, put the device on charger to prevent it going to sleep while downloading and use as reliable connection as you can (usually WiFi).

Zeta

2017-07-14, 13:25

Hello rinigus,

Sorry for the delay, but I finally managed to update the French translations. It should be ok to include it in your next release.

Thanks for you work again !

rinigus

2017-07-14, 14:13

Hello rinigus,

Sorry for the delay, but I finally managed to update the French translations. It should be ok to include it in your next release.

Thanks for you work again !

Thank you very much!. The next release would need few more strings to be translated since there is a bit of new functionality. But those strings should be easy. I still have to finish few things and then will be ready for the next cycle.

rinigus

2017-07-17, 11:06

I am planning to update the server in 1-2 days. As a part of the update, the database formats for Mapnik, Geocoder-NLP, and libosmscout will change and become incompatible with the earlier versions. So, if you critically depend on the server, please grab the maps that you would need this week to be sure that you would not be hit with any bug unprepared.

rinigus

2017-07-18, 18:09

New release and new maps are out: 0.14.0, 0.14.1 (Beta 2).

This release adds ability to search and draw maps in English, instead of the local language. Thus, for you who are in Asia and are struggling with the local script, you can fetch new maps and new version and would see names of the places in English, if available. For that, you would have to set "Preferred language" to English in Settings. I am expecting this to be the last new feature before 1.0.

Since Valhalla adds large number of files, I have rewrote cleanup code to use separate thread for deletion of files. It should not lock GUI while deleting.

As usual, translators have been working hard - thank you very much! I pulled translations this morning and will make new point release when more translations will be updated.

Edit: 0.14.1 increased the number of allowed queries done internally by geocoder-nlp in accordance with the number of languages. Without this, I was missing few search results that were working earlier.

rinigus

2017-07-20, 10:40

nthn

2017-07-20, 15:49

Would be great if you could make those pages translatable, too.

rinigus

2017-07-20, 16:07

Would be great if you could make those pages translatable, too.

I actually don't know how to do that. All I can suggest, at present, to fork https://github.com/rinigus/osmscout-server , make a new subfolder under docs, like

docs/fr

and insert translated index.md together with html files and screenshots under that directory (docs/fr). Styles and used JS can be all linked from there.

I guess we need the first brave translator to do that and all others would follow. Not as simple as for the main app, but should be doable.

Or is there a better way?

PamNor

2017-07-20, 16:49

After updating to OSM scout server to 0.14.1-1, search and navigating returns "no results" in Poor maps. i'm using default profile and Norwegian map. Is there a bug in the latest build?

rinigus

2017-07-20, 17:47

After updating to OSM scout server to 0.14.1-1, search and navigating returns "no results" in Poor maps. i'm using default profile and Norwegian map. Is there a bug in the latest build?

@PamNor, looks like its a bug during import. I am looking into it now. I'll let you know via this thread when its fixed. Sorry for it.

Zeta

2017-07-20, 18:41

User's manual coveeing setup of the server and clients (Poor Maps and modRana) is out and available at https://rinigus.github.io/osmscout-server/ .

Hopefully, I covered the most important aspects. Let me know if I missed something. Look by CSS/HTML experts is welcome to get the fonts shown a bit bigger on Jolla browser in tutorials.
That's a huge work you have done here ! :eek: You rock !:cool:

I guess we need the first brave translator to do that and all others would follow. Not as simple as for the main app, but should be doableI'll take a try tonight at translating the main markdown part, but I'm not sure to have the courage right now to start taking all the screenshots in French...
Have you done those screenshots on the SDK emulator, or did you take those on the device with lbt's screenshot app ?

Seeing how the app starts to be user friendly, if I can get that tutorial running in French, I could give the link to the French Jolla community (JollaFr.org) to try to enhance its visibility. What do you think of this ? Is it too soon ? Does the map server can handle some more users right now ?

Zeta

2017-07-20, 22:58

Rinigus,

I started translation of the english user manual to French.

So far, I have only translated the index.md (https://github.com/Sagittarii/osmscout-server/commit/16d347b7cb1d443419e150ef25a355bb5440881b#diff-e3e2a9bfd88566b05001b02a3f51d286).

I also tried to reorder the docs directory to make it easier to navigate when there will be several translations, along the original english files :
* move the english pages to docs/en (to locate easily what needs to be translated, and because even if it is the default one, it is still a language among all)
* move the css file to docs/assets/ (that is the name I usually find in web stuff, but I am opened to suggestion)
* added the html files for french (not yet translated), but keeping the same filenames as in english, to easily check if some files are missing/new when doing translations, and so that only the /en/, /fr/,... part has to change in the url to make it easy to point to other languages without having to translate the filenames.

This can be seen in the following commit: https://github.com/Sagittarii/osmscout-server/commit/b9f031cdeae7e89db8b18344fce5193d26089c1a#diff-e3e2a9bfd88566b05001b02a3f51d286
which gives the tree seen in : https://github.com/Sagittarii/osmscout-server/tree/master/docs

I updated the path to js and css stuff in the new /en/ directory, so that the html files there should run correctly after these moves.

I kept the fullPage.js dir at the docs root, but we could put it in assets/ too ?

Also, there is a missing index at the docs root to select the language now.

What do you think of this ? If you like it, I can put a pull request for you to integrate this changed structure before other translators start working on it.

Web is not my specialty, so if someone knows better than me, please help us !:)

rinigus

2017-07-21, 06:54

@PamNor, I think that the bug is in libosmscout location index code and I have submitted the issue to the library. It seems that it's not very common bug and I haven't been influenced by it when testing with several other countries. I know that Norway used to work just recently since I have been in Oslo and used it. So, it looks like its a rather specific case.

In case of geocoder-nlp import, I am using libosmscout to import data from OSM-based PBF and, after that, transverse location index to feed geocoder-nlp importer. So, even if we don't use libosmscout in the default profile, I am using it for the import.

One option is to to downgrade to libosmscout version that was used earlier. If it works and I or libosmscout developers cannot fix the issue fast, I'll make new map import for libosmscout and geocoder-nlp with the older version.

Let me look into it a bit longer, may take few days.

If anyone has seen degradation in the recent version, please let me know.

@Zeta: thank you! The screenshots were made using lbt's screenshot app on device. That way it can be done with the morning coffee.

It would be great to get tutorials and manual translated - that should help the users. Whether its too soon to advertise in the wider audience, I don't know. I presume many know already and I guess the server should hold it, there have been no feedback from @MartinK.

However, before advertising, I must fix the import bug as seen for Norway. (see more specific replies below)

Rinigus,

I started translation of the english user manual to French.

So far, I have only translated the index.md (https://github.com/Sagittarii/osmscout-server/commit/16d347b7cb1d443419e150ef25a355bb5440881b#diff-e3e2a9bfd88566b05001b02a3f51d286).

I also tried to reorder the docs directory to make it easier to navigate when there will be several translations, along the original english files :
* move the english pages to docs/en (to locate easily what needs to be translated, and because even if it is the default one, it is still a language among all)
* move the css file to docs/assets/ (that is the name I usually find in web stuff, but I am opened to suggestion)
* added the html files for french (not yet translated), but keeping the same filenames as in english, to easily check if some files are missing/new when doing translations, and so that only the /en/, /fr/,... part has to change in the url to make it easy to point to other languages without having to translate the filenames.

This can be seen in the following commit: https://github.com/Sagittarii/osmscout-server/commit/b9f031cdeae7e89db8b18344fce5193d26089c1a#diff-e3e2a9bfd88566b05001b02a3f51d286
which gives the tree seen in : https://github.com/Sagittarii/osmscout-server/tree/master/docs

I updated the path to js and css stuff in the new /en/ directory, so that the html files there should run correctly after these moves.

Excellent, lets move these.

I kept the fullPage.js dir at the docs root, but we could put it in assets/ too ?

Would be good to move it too, just note that its submodule.

Also, there is a missing index at the docs root to select the language now.

Yes, that's needed too. It could be also redirect to English with en/index.md listing all languages. But I am open to any option.

What do you think of this ? If you like it, I can put a pull request for you to integrate this changed structure before other translators start working on it.

Web is not my specialty, so if someone knows better than me, please help us !:)

I think its a great way and let's optimize the directory tree to make it simple for others.

rinigus

2017-07-21, 11:13

@PamNor, I am going to revert to older libosmscout version that used to work. When testing with Norway, all seemed fine.

So, I will be reimporting planet for libosmscout and geocoder-nlp. Then I will have to push it all to @MartinK's server and release new version of OSM Scout Server (0.15). Should be all done in 1-2 days if nothing unexpected will come out during testing. [While typing, Faroe Islands are imported already]

Looks like the Netherlands were hit as well: https://github.com/rinigus/osmscout-server/issues/174

If anyone has observed similar degradation in search performance (or anything else), please report.

PamNor

2017-07-21, 12:29

Thanks for your hard work and premium support. it's then time for me to speed up the Norwegian translation. :)

rinigus

2017-07-21, 19:36

Thanks for your hard work and premium support. it's then time for me to speed up the Norwegian translation. :)

0.15.0 is packaged already and I am uploading maps right now. The version will be released on OpenRepos tomorrow morning, assuming that maps will be transferred fully during a night.

Feathers McGraw

2017-07-21, 20:33

0.15.0 is packaged already

You're a machine! :)

Zeta

2017-07-21, 21:10

@Zeta: thank you! The screenshots were made using lbt's screenshot app on device. That way it can be done with the morning coffee.Ok. I will get to it maybe this weekend. I have to remove it first, to get the full install (with the first time wizard, and directory creation).

It would be great to get tutorials and manual translated - that should help the users. Whether its too soon to advertise in the wider audience, I don't know. I presume many know already and I guess the server should hold it, there have been no feedback from @MartinK.
However, before advertising, I must fix the import bug as seen for Norway. (see more specific replies below)Ok. I will first try to finish the translations, and I'll see.

So about the docs, I also moved the js part in the assets subdir (and changed path in html files accordingly).

Yes, that's needed too. It could be also redirect to English with en/index.md listing all languages. But I am open to any option.I have not much idea how to do the redirect, so I only added a simple index.md file with simple links to all languages. It can do the placeholder before getting something better, yet allowing to navigate easily.

I think its a great way and let's optimize the directory tree to make it simple for others.Thanks.
I put those last changes to github, and sent you a pull request :
https://github.com/rinigus/osmscout-server/pull/177
Please let me if I need to change anything to make it easy to pull (I hope it is fine, but I had to git push -force to get it there...).

rinigus

2017-07-22, 08:05

New release is out: 0.15.0, Beta 3

Its mainly bugfix release and is accompanied with the new maps. Libosmscout has been downgraded to its end of May version and the planet has been reimported for libosmscout and geocoder-nlp. The Netherlands issue has been confirmed to be fixed already, Norway I haven't check yet.

If anyone else had some problems with the search and, in particular, for countries smaller (in terms of the database size) than Norway, please tell so. Smaller imports would help to fix the issue we stumbled upon in libosmscout.

In 0.15.0, I also added a link in About to the user's guide.

Ok. I will get to it maybe this weekend. I have to remove it first, to get the full install (with the first time wizard, and directory creation).

It's sufficient to remove configuration file in .config/harbour-osmscout-server and move away Maps folder if its in the default location

So about the docs, I also moved the js part in the assets subdir (and changed path in html files accordingly).

I have not much idea how to do the redirect, so I only added a simple index.md file with simple links to all languages. It can do the placeholder before getting something better, yet allowing to navigate easily.

I put those last changes to github, and sent you a pull request :
https://github.com/rinigus/osmscout-server/pull/177
Please let me if I need to change anything to make it easy to pull (I hope it is fine, but I had to git push -force to get it there...).

Pull request merged and I have added the redirect via simple HTML. In English version we now have the link to the French one and I hope that's OK. Should be possible to hack the header to include all available languages though, have to look into it.

Re screenshots: large chunk of brain-dead work was actually renaming them.

@Feathers McGraw: just can't stand when something that was supposed to work is not working anymore :)

PS: Is Jolla Store on strike? I was probably moved in the queue every time I pushed new update, but as a result, Jolla Store version would probably jump from 0.13 to 0.15 directly.

XOleg

2017-07-22, 10:46

@rinigus. Yes, I had a problems with search with v.0.14. I requested XX, Street_name City but I had only Street_name City and pointer in random place of street.
It's OK with v.0.15.

PamNor

2017-07-22, 10:47

@rinigus. I first updated the Norwegian map data. Search and navigate from Poor maps no returns valid data from OSM scout server 0.14.1-1. The geocoder NLP data increases from 10MB to 113MB.
I then upgrade to OSM Scout server 0.15.0-1. Same result. Search and navigating from Poor maps OK. Thank you rinigus

bomo

2017-07-22, 13:02

I have issues with downloading some maps, please see https://m.imgur.com/IQQF77P

rinigus

2017-07-22, 13:11

I have issues with downloading some maps, please see https://m.imgur.com/IQQF77P

Please update the list of available maps. You seem to have older lists in your installation.

For updating , go to Map Manager, click 'Check for updates' and proceed as instructed.

rinigus

2017-07-29, 09:27

Looks like there are no additional bug reports for the last version (0.15.0). @mal started making Finnish translation and is getting close to finishing it. As soon as its ready, I can release an update.

As for User's Guide, I think the current structure is ready for translators. So, if anyone is wishing to translate that, please fork OSM Scout Server repo and add translation directly under docs/langcode and follow the lead under "fr" or "en". Its a lot of work though, taking into account large amount of screenshots needed for it.

rinigus

2017-07-30, 18:18

I have uploaded all OSM Scout Server dependencies and its own packages to OBS (16 packages in total), under https://build.merproject.org/project/show/home:rinigus:maps .

Hacking the server has never been so easy - clone the main repository in github and add repository for your architecture to resolve all dependencies - and you are all set!

rinigus

2017-08-09, 18:35

I have just released 1.0.0 version.

Compared to the previous version, the work is done by translators:

@carmenfdezb has translated Users Guide to Spanish (see https://rinigus.github.io/osmscout-server/es/), based on the ground work by @Sagittarii alllowing us to have translated Users Guide
@mal started Finnish translation

The functionality provided by 1.0 version certainly exceeds my original plan for it. I hope that, at this stage after a longer feature-freeze, its user-friendly, stable, and accessible to general SFOS users.

I am going to focus right now on the client side of things which may take some time (a month or maybe longer). After that I will continue the development according to suggestions and some ideas that I have. Right now, it has been suggested to allow using the server as a proper daemon. I may look into possible integration with QtLocation 5.6 (if that would arrive), exposing map matching, and few other things.

Enjoy using it!

ferlanero

2017-08-10, 00:30

Really impressed in how this offline map application works in sfos. Without doubt, one of the best solution in mobile offline navigation out there. Thanks!

karlos devel

2017-08-10, 00:51

A tons of Thanks to @rinigus of this. :)

rinigus

2017-08-15, 20:21

I have released a new version with the changes by @mal: fixed typos found by him as well as updated his Finnish translation. Thank you @mal!

I wonder if there is someone around interested in linguistics or able to spell out pronunciation of some words. Namely, Valhalla supports US English (Pirate version) for its instructions. While a joke, its a nice one and, I think, very appropriate for Sailfish. Unfortunately, TTS software does not know how to pronounce "Arrr" and few other words. What TTS software can do, is to pronounce according to given phoneme description.

So, I would like to ask for help and maybe someone here can propose their versions for words listed at https://github.com/valhalla/valhalla/issues/891 .

In Sailfish, install mimic from OpenRepos ( https://openrepos.net/content/rinigus/mimic ) and use

mimic -voice awb -ssml -t 'Some <phoneme ph="k ae p n">Capn</phoneme> wheel' -o /tmp/out.wav && gst-launch-1.0 -q filesrc location=/tmp/out.wav ! wavparse ! pulsesink

to test the phonemes.

DrYak

2017-08-15, 20:36

After that I will continue the development according to suggestions and some ideas that I have. Right now, it has been suggested to allow using the server as a proper daemon.

The best way to be compatible with the official Jolla shop ("no daemon allowed" rule) and end-users desire (background daemon), while avoiding to have 2 completely different codepaths, would be :

- always run the server in a separate process
- (which also means : design some inter-process communication. DBUS ? simple webservice ? protocole buffers ?)
- give the ability to start only that process ("OSMScout --daemon" or something)

- for the version available on openrepos.net :
pack a systemd ".service" file that only start the daemon in background

- for the version avaiable on the shop :
don't pack that file.

When UI starts, it checks (see the IPC mentionned above) if there's a daemon process answering.
If not, it launches the daemon as a child process / as just another thread of the main UI task.

On the official version it means that the service will only be available while the UI is running (even if it uses a weird IPC instead of directly calling functions).

On the openrepos version, people have the possibility to launch the daemon in background at boot ("systemctl enable OSMscout.service") and only launch the UI to monitor daemon activity.

rinigus

2017-08-16, 11:08

The best way to be compatible with the official Jolla shop ("no daemon allowed" rule) and end-users desire (background daemon), while avoiding to have 2 completely different codepaths, would be :

- always run the server in a separate process
- (which also means : design some inter-process communication. DBUS ? simple webservice ? protocole buffers ?)
- give the ability to start only that process ("OSMScout --daemon" or something)

- for the version available on openrepos.net :
pack a systemd ".service" file that only start the daemon in background

- for the version avaiable on the shop :
don't pack that file.

When UI starts, it checks (see the IPC mentionned above) if there's a daemon process answering.
If not, it launches the daemon as a child process / as just another thread of the main UI task.

On the official version it means that the service will only be available while the UI is running (even if it uses a weird IPC instead of directly calling functions).

On the openrepos version, people have the possibility to launch the daemon in background at boot ("systemctl enable OSMscout.service") and only launch the UI to monitor daemon activity.

Indeed, that would be a proper way of doing it. Would have to move all communication between GUI and the server to RPC then. Not too easy since it requires some redesign, but possible. I have added your suggestions to the corresponding github issue and let's see when I will get to that.

Alternative to RPC would be to add a command line switch and make some lock file. When user starts GUI, I can kill the daemon and run as GUI app. When GUI app is closed, a new daemon can be started as well. I would have to think how to ensure that its all working

rinigus

2017-08-17, 17:46

I just released a bugfix release that handles better tile requests coming with the out-of-bounds indexes. In addition, all maps have been updated and uploaded to the data.modrana.org server. Please let me know if there are any issues with the maps or otherwise.

seiichiro0185

2017-08-21, 08:17

rinigus

2017-08-21, 08:48

Another idea for running the server in the background would be to use Systemd Socket Activation, this way the Server would autostart if it is needed (=if a Request comes in), and could stop after a certain time of idle.
This way it would not be neccessary to waste resources if nothing is using the server, but activation would be transparent to any client app.
This would of corse imply that some proper IPC Mechanism is in place, because I don't think the "stop/start when GUI is launched" would work too well with that

I have never used it, have to read up on socket activation. However, from the brief look into intro blogs on it, it seems that the server code will have to be modified to accept the socket activated call. There are also questions on how to mix it with gui-started and socket-started options. Ideally, I don't want to modify the code too much and add additional split between worker and gui server threads connected via some RPC. But I'll read about this socket activation option and maybe its a solution. Seems that libmicrohttpd used in the server has some options to support it too, which is a good start.

rinigus

2017-08-22, 09:47

@seiichiro0185, I looked into it and probably, that's a proper way to do it. Such socket activation would need a bit of code, but, at least on the paper, it doesn't seem to be anything excessive. I might be able to work around shared port by using start/stop - this would need some testing though. I am finishing up the things that I started on client side and would look into daemon mode as soon as I be ready for it (probably would start working on it by the end of this week, if all goes to the plan).

rinigus

2017-08-31, 19:53

I have just uploaded 1.1.0 release.

This release adds automatic activation by systemd sockets. Thank you all for suggesting daemon mode and, in particular, @seiichiro0185 for the way to implement it. In this case, systemd's socket activation works very well and allows us to have minimal resources allocated for the server when we don't use it.

The server supports GUI and socket-activated modes. When you open the GUI, it stops the service and socket activation, waits till the server's ports get free, and works as usual. On closing GUI, socket activation is started again. If the server is started via socket activation, it runs in the background (no GUI) and would stop itself after some period of time when noone was accessing it (idle period can be set in settings).

Notice that if you access the server while switching between GUI and socket activation modes, you may get some requests ignored/dropped. This is a side effect of the implementation allowing me to avoid larger rewrite of the code. In practice, I have not seen such effect on my phone (OPX).

Enjoy using it and let me know if there are issues. The update is available via OBS, OpenRepos, and was submitted to the Harbour.

Feathers McGraw

2017-08-31, 20:55

So impressed with how quickly you acted on that suggestion!

Really appreciate being able to follow the development process on here too. Proper open source development in action :)

seiichiro0185

2017-09-01, 04:15

Thanks for the really fast implementation, and I can report the autoactivation works nicely on my Jolla C :)

rinigus

2017-09-01, 12:12

Thank you very much! As for speed, its all relative. It took more than a month to get from proposal at OpenRepos, discussion over here, and implementation. Meanwhile, a launcher has been made by ferlanero clearly demonstrating the need for better user experience. I could have done it probably faster, but was busy with the client side of things. Part of my work on clients is not merged yet, but all is submitted as PRs and, hopefully, would be soon reviewed/polished/rewritten/published. In meanwhile, I can work on some other aspects.

Looks like the whole approach was also acceptable for QA of the Harbour. So, the server is available via store with the full functionality. I would like to thank Jolla's QA team for approving the server.

Fellfrosch

2017-09-01, 13:25

Don't make yourself smaller than you are. :eek:
Your work and your working speed is phenomenal!
Tankx for all your efforts!!!

bomo

2017-09-01, 14:19

Don't make yourself smaller than you are. :eek:
Your work and your working speed is phenomenal!

exactly, @rinigus! And most importantly: where is actually the donate button?

rinigus

2017-09-01, 18:22

exactly, @rinigus! And most importantly: where is actually the donate button?

not implemented :)

Folks, but did you realize that Jolla Harbour approved an application that can act as a service, if requested by user? In particular, the socket activation required systemd service that is installed on users request in its home and enabled using user's permissions.

Note that, if you read Harbour FAQ, such approach seems to be in accordance with them. No special files are installed in the system, there are not scripts in RPM spec file, and so on.

Maybe its a way to provide such applications in future as well?

DrYak

2017-09-02, 15:40

Big thanks for the daemon mode !

MartinK

2017-09-03, 12:12

not implemented :)

Folks, but did you realize that Jolla Harbour approved an application that can act as a service, if requested by user? In particular, the socket activation required systemd service that is installed on users request in its home and enabled using user's permissions.

Yeah - I was also kinda surprised they accepted it as they are generally kinda edgy on less radical stuff. So certainly a big achievement! :)

Note that, if you read Harbour FAQ, such approach seems to be in accordance with them. No special files are installed in the system, there are not scripts in RPM spec file, and so on.

Yeah - as nothing should be started until connections start coming in, it really should be fine. This is also heavily used in "normal" Linux distros such as Fedora, so the mechanism is pretty well tested and robust by now.

Maybe its a way to provide such applications in future as well?
Yeah - I can imagine quite a few other cases where a network accessible daemon could be used to provide functionality to various client applications. Maybe for example a IM daemon the various IM apps could use as a backend as long as it's still not possible (IIRC) to really contribute improvements to the built-in messaging system (which is quite nice, but missing fat to many features and seem to be long term neglected). Or a socket activated speech to text/text to speech service apps can use, etc.

Sthocs

2017-09-08, 09:11

Hi, I haven't followed a lot the progress lately, but I see that everything is a LOT more user friendly! Very nice!
Unfortunately, I am not able to download maps anymore since a few releases (at least beginning of August). Here is the error I get:

203 - Error transferring http://data.modrana.org/osm_scout_server/geocoder-nlp-9/geocoder-nlp/europe-france-ile-de-france/geonlp-primary.sqlite.bz2 - server replied: Not Found
Similar message for other countries (I tried Mexico). I'm with the default profile, version 1.1.0
I've seen that the correct path is geocoder-nlp-10, but do you know why I tries to reach this URL? If you don't I will just reinstall, I'm just reporting this in case there is something you'd want to fix. Thanks!

rinigus

2017-09-08, 09:51

Hi, I haven't followed a lot the progress lately, but I see that everything is a LOT more user friendly! Very nice!
Unfortunately, I am not able to download maps anymore since a few releases (at least beginning of August). Here is the error I get:

Similar message for other countries (I tried Mexico). I'm with the default profile, version 1.1.0
I've seen that the correct path is geocoder-nlp-10, but do you know why I tries to reach this URL? If you don't I will just reinstall, I'm just reporting this in case there is something you'd want to fix. Thanks!

Yes, URL is changing when I upload newer maps. So, all you have to do when you get such message, is to click on

"Check for updates"

in Map Manager. Then your list of URLs will be updated, the countries that you have would be proposed for an update, and you would be able to get new ones as well.

I should make this solution easily reachable, opened an issue with it. Thank you for reporting!

Sthocs

2017-09-08, 11:31

Oh right! I wrongly assumed that unsubscribing and subscribing again would automatically try to fetch the latest resources. Thanks for the info.
But yeah, adding an info message when this error occurs, inviting the user to click on that button would be perfect :)
Thanks again!

rinigus

2017-09-09, 19:59

I have just published a release 1.1.1 with

* the updated translations

* expanded error message that should guide users to update the maps if the file is missing in the server (as reported by @Sthocs)

I would like to thank all the translators for the great work!

I have generated the packages a bit early today and couldn't anymore incorporate the latest changes in Russian and Swedish (really fast response by the translators). Namely, I am moving my SDK to Qt 5.6 level and decided to release the current package with the older SDK. That should allow the ports that have not yet updated to 2.1 to use the server. So, I am sorry that the latest translations were not incorporated, I'll add them the next time.

XOleg

2017-09-10, 07:05

I have just published a release 1.1.1

I would like to thank all the translators for the great work!

I have generated the packages a bit early today and couldn't anymore incorporate the latest changes in Russian and Swedish (really fast response by the translators). So, I am sorry that the latest translations were not incorporated, I'll add them the next time.

@rinigus, it's accidentally for me :-) It's more complicated to write good help. You work very fast(development of server), not me.

Zeta

2017-09-10, 14:55

Hello Rinigus,

I have finally made most of the translation/screeshots for the documentation : https://github.com/Sagittarii/osmscout-server/commit/277ef29f2fb75d7712bfe05eb9139c0c12f12392

I still have the modRana and poor maps pages to translate and take screenshots. I'm not sure when I will take the time to finish it, so I can already make a pull request for this part right now if you want.

I have also seen that two fellow French guys are now in the transifex effort, so thanks to Jordi and Sthocs to have made the updates when I was a bit away. :)
If you want to proof-read the documentation (I made my best, but it is far from perfect), feel free to do so, and send me your remarks so that I can fix them (or you can clone the project in Github and send me directly a pull-request).

rinigus

2017-09-11, 07:58

eson

2017-09-12, 20:08

Sure, please send the current state as PR. Then we can integrate it into the current manual and it will be available to the users.
Does all the screen shots have to be JPG and 1080x1920px? I'm almost done, except for the images.

rinigus

2017-09-12, 20:15

Does all the screen shots have to be JPG and 1080x1920px? I'm almost done, except for the images.

No, I think smaller resolution should work as well. Its just I happen to have device with such screen

eson

2017-09-12, 20:30

Great! Then I won't have to Gimp. :)

Zeta

2017-09-13, 19:32

Sure, please send the current state as PR. Then we can integrate it into the current manual and it will be available to the users.Done : https://github.com/rinigus/osmscout-server/pull/201

For the pictures I took for french, they are in the Jolla 1 phone resolution, so also a bit less than those taken by rinigus.

JoOppen

2017-09-14, 09:20

rinigus

2017-09-14, 10:17

@rinigus: first off all: THANK YOU FOR THE GREAT EFFORT!

atm I am about to try OSM scout server with Poor Maps and - while offline search works perfectly - I cannot see the map tiles. I get the following error messages (off cuorse, maps are saved on SDcard):

INFO: 09:51:18 Request: /v1/tile?daylight=1&shift=1&scale=4&z=14&x=8802&y=5372
INFO: 09:51:18 Request: /v1/tile?daylight=1&shift=1&scale=4&z=14&x=8802&y=5373
INFO: 09:51:18 Request: /v1/tile?daylight=1&shift=1&scale=4&z=14&x=8801&y=5373
INFO: 09:51:18 Request: /v1/tile?daylight=1&shift=1&scale=4&z=14&x=8801&y=5372
INFO: 09:51:19 Mapnik: Hinzufügen von /media/sdcard/5792-2A74/Maps.OSM/mapnik/countries/europe-germany
FEHLER:09:51:19 Mapnik exception: Sqlite Plugin: 'database disk image is malformed' (/media/sdcard/5792-2A74/Maps.OSM/mapnik/countries/europe-germany/mapnik.sqlite)
Full sql was: 'PRAGMA cache_size = -20480;' encountered during parsing of layer 'landuse_gen0' in Layer of '/home/nemo/.cache/harbour-osmscout-server/harbour-osmscout-server/mapnik/default/day/mapnik.xml'
WARNUNG: 09:51:19 Mapnik nicht verfügbar
FEHLER:09:51:19 Mapnik exception: Sqlite Plugin: 'database disk image is malformed' (/media/sdcard/5792-2A74/Maps.OSM/mapnik/countries/europe-germany/mapnik.sqlite)
Full sql was: 'PRAGMA cache_size = -20480;' encountered during parsing of layer 'landuse_gen0' in Layer of '/home/nemo/.cache/harbour-osmscout-server/harbour-osmscout-server/mapnik/default/day/mapnik.xml'

Any idea?

I suspect that the database is broken, hopefully only your copy and not on the server.

Note that the databases used for search, routing, and map tiles are all different. So, if one works, it could still be that the other is broken, as you have.

Please try to download smaller region and test on that. Germany is big and maybe something went wrong during download. To avoid overlapping regions (no idea how mapnik will handle it), unsubscribe from Germany when testing with the smaller region. If you keep the files, you could later subscribe again and the download should not be triggered.

Iff it turns out that the database is broken, try to download it again. For that, stop the server, go to the maps directory, follow common sense and find the mapnik database for Germany. Delete it and start the server again. Map Manager should suggest to redownload the database for Mapnik/Germany

Let me know how it went.

JoOppen

2017-09-14, 10:35

Thank you, rinigus, I am just about to try a further download and will report on the resullt!

Fellfrosch

2017-09-14, 12:59

Germany runs well on my device. So probably it's only JoOppen's database. But I have to admit, that I downloaded it a few days ago.

JoOppen

2017-09-14, 16:47

I have downloaded the entire Germany map files on a second device - same problem (I started the time-consuming download prior to reporting the problem and therefore did not Interrupt the second download).

I also downloaded the Netherland map files and initially encoutered the some problem as above. However, after unsubscribing Germany, everything was fine (Germany left as a white spot, however)..

Therefore, there might be a problem with the Germany map files on the server.

Veraendert

2017-09-14, 17:13

I don't think so. I downloaded the map of Germany some time ago and it works like a charm.

JoOppen

2017-09-14, 17:49

I don't think so. I downloaded the map of Germany some time ago and it works like a charm.

O.k., I will give it a third try. I wish, Germany was smaller

Fellfrosch

2017-09-14, 18:01

Well we downloaded it some days ago. So maybe a file got damaged on the server. improbable but not impossible.

rinigus

2017-09-14, 18:22

O.k., I will give it a third try. I wish, Germany was smaller

Let's check whether the file size matches. It should be

6993805315 distribution/mapnik/countries/europe-germany

for Mapnik database. I wonder if at such size the filesystem used by SD card starts playing a role.

Note that you could also download Germany by parts, just take one hierarchy down and it should work as well. All routing, maps search, and tile rendering works over the subscribed databases.

Zeta

2017-09-16, 15:15

Rinigus,
I finally took the time to complete the french documentation.
The new pull request is here: https://github.com/rinigus/osmscout-server/pull/203

Doing so, I noticed that the docs for poor-maps and modRana end with :
"You are all set! Don't forget that you need to have OSM Scout Server and Poor Maps running <i>simultaneously</i> for offline operation."

I assume we could change that a little now that the server can start automatically, like giving a link to the documentation page describing how to activate the auto start.
I have only translated as is for now, to have all the documentation at the same level.
Maybe it would be better to wait for the auto-start option being set as default before updating the documentation ?

rinigus

2017-09-16, 16:59

I assume we could change that a little now that the server can start automatically, like giving a link to the documentation page describing how to activate the auto start.
I have only translated as is for now, to have all the documentation at the same level.
Maybe it would be better to wait for the auto-start option being set as default before updating the documentation ?

Thank you!

Since I am now trying to get Mapbox GL working ASAP, I'll probably not release new version too soon. Looks like we can set the auto-activation by default, since there have been no complains regarding it. I'll make the corresponding commit. As for docs, let's keep them for now. We can probably do the changes later...

Zeta

2017-09-16, 17:12

Thank you!You're welcome, but please remember YOU made the hard work, I simply translated some strings :D

Since I am now trying to get Mapbox GL working ASAP, I'll probably not release new version too soon. There is no urge.

As for docs, let's keep them for now. We can probably do the changes later...I agree.

rinigus

2017-09-28, 07:31

Maps are updated on data.modrana.org. If you get failure during download, don't forget to "Check for updates". Enjoy!

bomo

2017-10-03, 08:22

@rinigus

After the last (map) update I can't find any locations anymore.
It looks like the server crashes. I hope this log helps:

Oct 03 10:14:17 jolla-bomo2 [30874]: [D] unknown:0 - MLS response: "{\"location\": {\"lat\": 47.2648138, \"lng\": 10.2200436}, \"accuracy\": 13.3110427}"
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8652,5729,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8653,5732,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8651,5729,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8649,5729,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8653,5728,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8650,5730,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8652,5730,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8651,5730,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8649,5730,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8653,5729,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8650,5731,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8652,5731,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8651,5731,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8649,5731,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8653,5730,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8650,5732,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8652,5732,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8651,5732,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8650,5728,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8649,5732,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8653,5731,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8652,5728,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8651,5728,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8649,5728,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:23 jolla-bomo2 ur-poor-maps[31655]: [W] unknown:0 - QGeoTileRequestManager: Failed to fetch tile (8650,5729,14) 5 times, giving up. Last error message was: 'Connection refused'
Oct 03 10:14:27 jolla-bomo2 [30874]: [D] unknown:0 - MLS response: "{\"location\": {\"lat\": 47.2648135, \"lng\": 10.2200442}, \"accuracy\": 13.2932322}"
Oct 03 10:14:28 jolla-bomo2 connmand[14009]: Error with UDP server 10.99.0.12
Oct 03 10:14:28 jolla-bomo2 [29442]: [D] unknown:0 - QQuickGridLayoutBase::geometryChanged QRectF(48,278.5 1824x244) QRectF(48,278.5 984x244)
Oct 03 10:14:28 jolla-bomo2 [29442]: [D] unknown:0 - "" QQuickGridLayoutBase::rearrange() QSizeF(1824, 244)
Oct 03 10:14:28 jolla-bomo2 connmand[14009]: Error with UDP server 10.99.0.12

In the end, poor maps always responds with "no results"

Another question: Is it possible to add other maps to the server? I see (in poor maps) that there are two types "normal" and "car". For me, an outdoor type e.g. for hiking is missing. For this I so far used the "Mapbox Outdoor GL" from poor maps with are very nice but of course they are not available without internet access.

rinigus

2017-10-03, 09:09

@rinigus

After the last (map) update I can't find any locations anymore.
It looks like the server crashes. I hope this log helps:

In the end, poor maps always responds with "no results"

Another question: Is it possible to add other maps to the server? I see (in poor maps) that there are two types "normal" and "car". For me, an outdoor type e.g. for hiking is missing. For this I so far used the "Mapbox Outdoor GL" from poor maps with are very nice but of course they are not available without internet access.

To diagnose the issue, would you mind:

* Start OSM Scout Server as an app. This will allow you to see the log messages of the server

* Start Poor Maps

* Search for location

After you get "No results", please tell me:

* Did OSM Scout Server app crashed?

* If it did not, what are the log messages? They are displayed under "Events" on the main screen.

* If the app crashed, please start it from the terminal as "harbour-osmscout-server" and repeat. After crash, please let me know what's an error message.

If something is still wrong, please tell me

* Which map do you use?

* Which search fails?

Re other maps: As we move over to GL-rendered maps, I am planning to add support for them in the server as well. It maybe wise to wait a bit until vector maps are available and then design styles around it.

As for "outdoor" version, note that contour lines could take significant amount of space. You could already make yourself familiar with the available styling tools in meantime and see how to make styles + read licensing conditions on map data usage.

Edit PS: QGeoTileRequestManager are, if I am not mistaken, resulting from the hack used by Poor Maps to use the old qtlocation shipped with SFOS. Should have nothing to do with OSM Scout Server...

otsaloma

2017-10-03, 17:15

Edit PS: QGeoTileRequestManager are, if I am not mistaken, resulting from the hack used by Poor Maps to use the old qtlocation shipped with SFOS. Should have nothing to do with OSM Scout Server...

Yep, that's normal.

paoletto

2017-10-19, 18:15

@rinigus: have you considered packaging this also as a QtLocation plugin? :)

rinigus

2017-10-20, 04:57

@rinigus: have you considered packaging this also as a QtLocation plugin? :)

I have looked into it earlier, but not keen on it right now for the following reasons:

We are still on QtLocation 5.2. There is no future in developing plugins for that
With MapboxGL QML plugin, we have QtLocation 5.9 level functionality. Since it renders maps on device screen directly, you have many advantages when compared to raster-tile based approach. I would expect that most of the map-requiring apps would start using it. However, this QML plugin is non-standard - hence no additional benefit to package the server as plugin for map rendering
When(if) QtLocation 5.6 will land to SFOS, I may look into it. But note that it has to bring something significant for me as a user as well. If there is no significant gain that we, as users, can see in terms of better apps developed, there is very low motivation in repackaging the working solution

I am currently working on bringing MapboxGL vector tiles to OSM Scout Server. It would take some time to figure out how to convert the maps, but I am making some progress. One of the available solutions uses docker with a rather lengthy planet import into the postgis db, but it looks like I maybe able to avoid it. I'll try to get to the scheme that's compatible with the server setup by @otsaloma

rinigus

2017-12-10, 20:41

I have just released 1.2.0.

This is the first release after additional translations have been done for the User's Guide, so its a great opportunity to thank the translators for their work.

We also have a new backup server for maps distribution provided by @bomo, thank you very much! It has been needed few times already. I'll integrate multiple servers support in future releases.

As for new functionality, the server now supports Mapbox GL vector tiles. While server itself doesn't have to do much work to provide them, there has been a lot done to make import of map data possible and importing the maps. In addition, the server provides framework for styles, supplies fonts and icons as well. So, it should be all ready and in place for map application developers. The first such application has been released today as well.

The tiles for Mapbox GL have been packaged by splitting the planet by 2x2 degrees boxes (zoom levels 7 and larger) and the overview map (zoom levels 0-6). As a result, 2x2 is the smallest region you can download. So, when you choose the country of interest, you will get all boxes within that country as well as the boxes on the border. In case of Estonia, I happen to get part of Helsinki and St Petersburg. This, as expected, makes the map footprint larger. However, if I will be downloading Finland, for example, then only missing tiles will be downloaded. As a result of such distribution, its easy to find corresponding database on the device and serve the tiles. So, I plan to stick with this model.

Similar scheme is used for Valhalla, with 1x1 degrees boxes that will probably also move to 2x2 in future. Right now we have lots of files when we upload them to the servers for maps distribution and the situation is not ideal for Valhalla.

With addition of Mapbox GL support, new profiles have been added as well for either using support for the both type of tiles or vector only. Default is Mapnik-provided raster tiles at the moment, this has not been changed.

I have been postponing updates of the maps due to the work on Mapbox GL. I would still have to work on Mapbox GL styles (checking out work of @otsaloma on it) and will probably adjust import scripts to suit the style. After that, the map updates would resume as usual.

Enjoy the vector tiles if you can (not on Jolla 1 due to crashes of Mapbox GL widget, sorry).

rinigus

2017-12-11, 13:37

Just in case if anyone has missed it. Mapbox GL is a new backend (and I hope that the last one).

If you want to use it, please configure the server:

enable Mapbox GL in profiles (select either just vector or vector and raster tiles) or, if you select Custom profile, select Mapbox GL in the Map Manager (no need to select it Settings since its a data only).

update subscription info in Map Manager. For that, press "Check for updates"

unsubscribe/subscribe each territory that you have.

the server should indicate that you need to download new datasets. Please review and download them

When downloads are done, the server should be able to provide the tiles, fonts and other Mapbox GL-related data to the clients.

seiichiro0185

2017-12-11, 14:04

rinigus

2017-12-11, 14:10

I can report that the new version 1.2.0 with poor maps GL works beautifully on my Xperia X / Sailfish X. Rendering speed of the maps is blazingly fast and looks quite nice.

So again thanks for the great work on this essential piece of native SFOS software.

Thank you very much for reporting! Is it against OSM Scout Server or online maps by Mapbox?

PS: Ahh, now I can see that its against the server :) .

PamNor

2017-12-11, 18:12

rinigus

2017-12-11, 18:39

Poor Maps GL with OSM scout server and vector tiles only work when online. Wifi or mobile data. Is this the intention behaviour. Great work! Is it possible to have 2 fingers panning in Poor Maps for vector tiles?

Same reported at OpenRepos few minutes ago - https://openrepos.net/comment/22975#comment-22975 . Not intended by me, I will look into it ASAP.

As for 2 fingers panning - not sure what you mean.

PamNor

2017-12-11, 18:49

Same reported at OpenRepos few minutes ago - https://openrepos.net/comment/22975#comment-22975 . Not intended by me, I will look into it ASAP.

As for 2 fingers panning - not sure what you mean.

Sorry. English is not my native language. I meant. 2 fingers gesture to change pitch in vector tiles. I tried demo mapbox-sfos and was very impressed.

rinigus

2017-12-11, 18:55

Sorry. English is not my native language. I meant. 2 fingers gesture to change pitch in vector tiles. I tried demo mapbox-sfos and was very impressed.

In demo I had a pitch slider. I am sure we'll get that too, we could link the angle to the speed, for example. There are lots of things possible, its early days and I would expect lots of polish later. Don't forget, we'll get map client developers on board in addition (Poor Maps proper and modRana and maybe more will join). Right now I have to get the basics right, to fix strange bugs as we have with the network access.

MartinK

2017-12-12, 00:36

Don't forget, we'll get map client developers on board in addition (Poor Maps proper and modRana and maybe more will join).
Mapbox GL integration is currently my top priority in modRana, together with better voice output.

It will likely not be easy to support everything modRana does (routing/navigation/POI/on map markers, logging trace, overlays) right away, but I would like to have Mapbox GL support available in some form soon, so that users can provide feedback. :)

rinigus

2017-12-12, 07:41

Network access issue seems to be Qt bug. Full description via "official" channel, see https://together.jolla.com/question/175944

Please vote for it to get attention and higher probability to get the fix

rinigus

2017-12-12, 09:12

There maybe a workaround by switching Mapbox GL over to curl. I will try tonight and see whether it can be easily done (there is a hope that it is).

Still, in long term, we will need a proper fix and not having locally-patched-somewhat-different-QMapboxGL. So, the votes are needed for Jolla to take the notice :)

jdrescher

2017-12-12, 13:35

For that reason it might make sense to also take a look at the QML map component used by Foursail:
https://github.com/Maledictus/foursail/blob/master/qml/PinchMap.qml

It should be quite similar to the modRana PinchMap (both modRana and Frousail projects started with the old PoinchMap.qml from AGTL), a bit less advanced and possibly quite a bit more standalone. :)
That was a really good tip from you. I recently had a little bit of spare time and looked into PinchMap used by FourSail. With some slight modifications it is now doing most of the things I want.

However there is one thing I don't even know how to start with. It's track lines. The only idea I have right now, is to draw a qml image with a small dot on every track coordinate. But that's thousand of images, I don't know if that makes sense. May I ask how you did that in modRana?

MartinK

2017-12-12, 14:47

That was a really good tip from you. I recently had a little bit of spare time and looked into PinchMap used by FourSail. With some slight modifications it is now doing most of the things I want.

However there is one thing I don't even know how to start with. It's track lines. The only idea I have right now, is to draw a qml image with a small dot on every track coordinate. But that's thousand of images, I don't know if that makes sense. May I ask how you did that in modRana?

I'm using a Canvas (http://doc.qt.io/qt-5/qml-qtquick-canvas.html) element on top of the map to draw routes, tracklogs & the track recording trace:

- recording trace paint method (https://github.com/M4rtinK/modrana/blob/master/modules/gui_modules/gui_qt5/qml/MapPage.qml#L549)
- tracklog paint method (https://github.com/M4rtinK/modrana/blob/master/modules/gui_modules/gui_qt5/qml/MapPage.qml#L520)
- route paint method (https://github.com/M4rtinK/modrana/blob/master/modules/gui_modules/gui_qt5/qml/MapPage.qml#L261)

The canvas is moved around when the map is panned and redrawn when needed (zoom in/out, the polyline changed, clear the polyline, etc.).

rinigus

2017-12-12, 14:56

rinigus

2017-12-12, 20:52

libcurl to the rescue! I have just published an updated version of the plugin that allows (at least on my device) to use Mapbox GL against OSM Scout Server in fully offline mode (all network interfaces are disabled).

Get new plugin here: https://openrepos.net/content/rinigus/mapbox-gl-native-bindings-qt-qml

or by running update against my OpenRepos repo.

We've got lucky, since Mapbox GL uses libcurl for other platforms. So, I just had to make small changes in build file to use the code that loads data using libcurl. Seems to work as intended.

Downside is that I have a version of QMapboxGL which is somewhat different from the upstream. At least until the issue in Qt will not be resolved properly.

MartinK

2017-12-12, 21:22

Downside is that I have a version of QMapboxGL which is somewhat different from the upstream. At least until the issue in Qt will not be resolved properly.
Is the Qt 5 source code & packaging Jolla uses available somewhere in the open (eq. on git.merproject.org (http://talk.maemo.org/showthread.php?p=1539322#post1539322), etc.) ?

That could possibly enable tracking the issue down independently on Jolla & sending a patch, which could make thing happen quicker, as they would just have to review, merge and QA the the fix, but not also develop it.

PamNor

2017-12-12, 21:37

I have tried the new Mapbox GL update. super fast map update. Fully functional offline. You are genius .

rinigus

2017-12-13, 07:01

Is the Qt 5 source code & packaging Jolla uses available somewhere in the open (eq. on git.merproject.org (http://talk.maemo.org/showthread.php?p=1539322#post1539322), etc.) ?

That could possibly enable tracking the issue down independently on Jolla & sending a patch, which could make thing happen quicker, as they would just have to review, merge and QA the the fix, but not also develop it.

I expect so, but I don't know where.

jdrescher

2017-12-13, 11:51

@jdrescher, unless you want to run it on J1, why not to switch to Mapbox GL? It has all you need and even more: pan/zoom, you can easily insert the routes or POIs into the map as geo-objects and the widget will draw it for you.

Hm, that sounds interesting. Is there an example project with a little bit of information/explanation?

rinigus

2017-12-13, 12:25

Hm, that sounds interesting. Is there an example project with a little bit of information/explanation?

Sure!

Example codes:

Example at https://talk.maemo.org/showpost.php?p=1536584&postcount=1
More advanced showing how to add objects: https://github.com/rinigus/mapbox-gl-qml/blob/master/app/qml/main.qml
Poor Maps port to this widget https://github.com/rinigus/poor-maps/tree/gl with map QML at https://github.com/rinigus/poor-maps/blob/gl/qml/Map.qml

Vector maps are different from raster in the interactivity with them, you can insert/remove/modify your objects. So, API has to be richer than before.

To learn, I would suggest to
Make an account at Mapbox, go to Mapbox Studio and try to edit one of their styles. Just play a little bit with it (10-15 min) to get the feeling of interaction between sources/layers/properties
Take a brief look into https://www.mapbox.com/mapbox-gl-js/style-spec (10-15 min); later you will need to read sections on the need basis
API for the widget is documented at https://github.com/rinigus/mapbox-gl-qml/blob/master/api.md . Look through, you don't need every property/signal, just get a feeling
Start hacking on the basis of examples
Don't hesitate to ask for help.

jdrescher

2017-12-13, 14:34

To learn, I would suggest to
Make an account at Mapbox, go to Mapbox Studio and try to edit one of their styles. Just play a little bit with it (10-15 min) to get the feeling of interaction between sources/layers/properties
Take a brief look into https://www.mapbox.com/mapbox-gl-js/style-spec (10-15 min); later you will need to read sections on the need basis
API for the widget is documented at https://github.com/rinigus/mapbox-gl-qml/blob/master/api.md . Look through, you don't need every property/signal, just get a feeling
Start hacking on the basis of examples
Don't hesitate to ask for help.

Damn, PinchMap is almost working and now you come along with this one:rolleyes:;)
Before I start digging into it I have a few questions:

is it stable enough to work with it already?
is it possible to have map buttons?
OSM Scout Server connection is working?

Thanks!

rinigus

2017-12-13, 14:42

Damn, PinchMap is almost working and now you come along with this one:rolleyes:;)
Before I start digging into it I have a few questions:

is it stable enough to work with it already?
is it possible to have map buttons?
OSM Scout Server connection is working?

Thanks!

Yes/Yes/Yes.

To test, get Poor Maps GL from OpenRepos and try it out. That will tell you about stability. I don't recall it crashing for a while, if ever. As for API stability, hard to tell. I don't have any plans to change it, but it maybe adjusted for better as long as we get more developers using it. Its in our hands this time, not Qt/Jolla/...

You can get widgets on top of the map by long press, as in Poor Maps proper. When you move, rotate and change the pitch, all widgets would move accordingly.

OSM Scout Server connection is working for 20 hours already :)

rinigus

2017-12-16, 07:51

MartinK

2017-12-16, 12:20

data.modrana.org is down and there is an error in OSM Scout Server download manager code which prevents it to use backup server provided by @bomo. That part of the code was tested against a single server since I didn't have a luxury of map distribution via multiple servers. As a result, right now we cannot download the maps, sorry for inconvenience.

I'll re-write parts of Map Manager to support multiple servers, so we could be a bit more resistant to data.modrana.org downtime.
I've sent them an email, but it's indeed unlikely the will fix it during the weekend.

In any case, it's pretty strange the outages happen so often and especially before/during the weekend. One would thing they have found and fixed all the causes long ago during the previous outages.

rinigus

2017-12-16, 12:47

I've sent them an email, but it's indeed unlikely the will fix it during the weekend.

In any case, it's pretty strange the outages happen so often and especially before/during the weekend. One would thing they have found and fixed all the causes long ago during the previous outages.

Good thing about it is that it gives me initiative to write new code to make the server more robust and use multiple providers. I am on the case and, hopefully, all will go smoothly and I can test against failed data.modrana.org :)

rinigus

2017-12-17, 20:33

New release is out: 1.3.0

It adds support for multiple map data download servers. I managed to test it while data.modrana.org was down, but now you would have to wait till you can test it too (modrana server is up). Download servers are ordered and, on each OSM Scout Server start, the downloads start from data.modrana.org first and, on failure, move to @bomo's server. If anyone would like to join the club, please let me know. The implementation should support as many servers as we could have, I think.

Next, Valhalla's delayed start has been fixed that, when OSM Scout Server was started by systemd in response to routing request, led to an empty first reply (No route found). Now, routing and everything else should work immediately.

Translations have been pulled from Transifex and all the updated texts for Mapbox GL support should be there now. Thank you, translators.

Finally, I have added donations link as well. The donations are appreciated and are for my coding and time I spend on this and other SFOS projects. The donations platform is Liberapay which has rather transparent approach allowing us all to see the current sum of donations and their distribution in time. I think it maybe interesting for other developers and users to see such stats.

Enjoy using 1.3.0. I am planning to work through Mapbox GL import and styles, if nothing else comes up in between.

Amboss

2017-12-29, 14:36

rinigus

2017-12-29, 14:45

I have followed this development for some time. Last week I thought, I give it a try.

First I had problems with downloading all information, once with the timeout, the others with spome unzip error. I think the 4th or 5th try got me alle information including map of germany. Is there a way to maybe download less information at a time?

Now I get errors when trying to use the server with both Modrana and Poor Maps. See attached image

Looks like the SQLite database used by Mapnik is damaged. Germany is rather difficult case, its huge. I would suggest to get just a part of Germany you are interested in. Start small and progress from that. Note that Mapnik/Valhalla/Geocoder-NLP as well as MapboxGL backends glue all databases on fly. So, you may as well get parts of Germany.

To proceed, I suggest to delete Germany and then get new dataset for smaller part. Please let me know whether it helped.

Note that unzipping errors also suggest that something went wrong.

Finally, during this week we had malfunctioning of data.modrana.org and that led to waiting for downloads, I imagine.

ralooyar

2017-12-30, 20:36

rinigus

2017-12-30, 21:08

I just started with PoorMaps and OSM Scout Server and I like it! :D

In order to analyse my POI problem (https://talk.maemo.org/showpost.php?p=1539758&postcount=117), I encountered the following issue:

The GitHub Readme (https://github.com/rinigus/osmscout-server/blob/1.3.0/README.md) says:

But if I enter this URL in Firefox on my Sailfish X while OSM Scout Server is running, all I get is: "Error while listing available POI types"
In OSM Scout Server Event-Log, I see: "Database is not open, cannot list POI types"

Test-environment:

PoorMap can search for POI in OSM Scout Server
I have downloaded the map: "Europe / Germany / Bayern" (with the "Default" profile)
Languages is set to: "de"
I use OSM Scout Server 1.3.0-1 from Jolla Store

Its a bug. I think you are the first user trying to access it, and this is after just a bit of more than a year of development. It has been broken for anyone not using the original libosmscout backend. To get the list for the default profile, you would have to open SQLite database that is in Maps folder, under geocoder-nlp/europe-germany-bayerngeonlp-primary.sqlite and run

open database:

sqlite3 geocoder-nlp/europe-germany-bayerngeonlp-primary.sqlite

in sqlite3 shell:

select name from type order by name;

The names of the types are searched for using substring search. Before that, your query is normalized by libpostal. So, when you search for "ATM", its first normalized to "atm" and later would hit "amenity_atm".

I am working now on Mapbox GL styles/import, when its finished I was planning to look into geocoding and update libpostal to newer version. But that may take some time - I may have to fix some yet unknown bugs of Mapbox GL widget before.

ralooyar

2017-12-30, 21:52

rinigus

2017-12-31, 06:39

Thanks for the workaround.

No need to hurry. I do not urgently need this functionality. I just stumbled upon this issue while playing with the POI data.

----

Other question on the topic POI:

Do you do some kind of tag merging, when creating the OSM Scout Server database? In OsmAnd I can search for POI with tag "amenity _library" or "amenity_police". In OSM Scout Server database libraries and police stations in my hometown can be found, but only by their name. And the type is listed by PoorMaps as "Building".

In my opinion, it would be helpful, to be able to explicitly search for "libraries" or "police stations" by tag, since if I am new in town, I don't know the name of the library. What do you think about this? Should I file a github issue?

Yes, please file the issue. Since its geocoder component, please file it at https://github.com/rinigus/geocoder-nlp

As far as I remember, types are merged in the import and assigned as only one for POI. Hence the problem.

Keep digging into POIs and other components. When designing geocoder, I was mainly focused on getting results by name since that was really poor. Now it's time to look into search by type as in nearby search and make it work well. I suspect that the database will have to be redesigned, but let's see how.

ralooyar

2018-01-01, 09:22

Yes, please file the issue. Since its geocoder component, please file it at https://github.com/rinigus/geocoder-nlp

Done: https://github.com/rinigus/geocoder-nlp/issues/36

ralooyar

2018-01-01, 19:50

I recently had another problem, finding "gas station" POIs in PoorMap, using OSM Scout Server as data source.

The problem is, that in OSM data "gas stations" have the tag name "amenity : fuel" so the search text "gas station" is not matching.

More details at: https://github.com/otsaloma/poor-maps/issues/59

@rinigus:
Do you think, this is a problem which could be and should be fixed by OSM Scout Server (or Geocoder-NLP)? If yes, I would create a separate Github issue.

rinigus

2018-01-01, 20:11

rinigus

2018-01-09, 21:05

Just a short update - I am preparing an update to the maps. At present, the last component - Mapbox GL tiles - is generated and when ready I will distribute them. This would bring the data to the level of OSM just before Christmas.

With the addition of Mapbox GL the imports do take significantly longer and I probably would have to optimize it further. Let's see after that whether once a month reimports are feasible or we have to move to once in two months schedule.

I have reworked the styles for Mapbox GL and, if there is interest, could try to look into how to make editing of them easier. That's in case if some of you would like to participate in making the styles.

As for timeline, the map import will probably take few nights, maybe a week.

rinigus

2018-01-13, 08:37

All maps are imported and I am going to upload the maps this weekend. Since Mapbox GL tile schema has changed, I changed the version of the tiles as well. The new Mapbox GL maps will be compatible with the next version of OSM Scout Server (planned to be released this weekend as well), not the current one.

So, there could be some disturbance in the service this weekend due to non-perfect sync with software and data publishing.

rinigus

2018-01-14, 11:16

New version of OSM Scout Server (1.4.0) and updated maps are out.

For those using Jolla Harbour, you can start using new maps only after Jolla Harbour version is updated as well.

Main work was done not on the server side itself, but on optimization of Mapbox GL tiles import, adjustments of the tile schema, and generation of map styles. On server side, I mainly updated documentation by adding sections on running the server and what does automatic activation means in terms of file generations at the home directory (asked during Jolla Harbour review). Thanks to the translators for updated strings, I included the version available this morning.

Now to give you an idea of what's going on with the implementation and import of the tiles, I describe it a bit below. While we use Mapbox GL Native as a library, lots of work done by Mapbox on styling is not available for use in external projects (when you want to use it independently from Mapbox service). Which is fine since they do have to get income stream as well, but requires for our project to make new styles and generate tiles. Fortunately, OpenMapTiles project has been working on tiles import and schema, but there are several problems with them which I have been fixing during several months:

* Very long time required to import - 20+ days (24h) of server time
* Problems with POI data (icons were frequently missing and its hard to debug)
* Several other problems as well, some of them I don't even recall now.

Now the import time was a major issue. Import works by first pushing all data from PBF (Planet.PBF) to PostGIS database and later generation of tiles via Mapnik and tilelive-copy. Pushing data into PostGIS takes about 8 hours of intensive CPU use and later ~12+ hours by some threads to reorganize the database. As such, this part is not a problem and has rather reasonable timeframe. Now, when I used OpenMapTiles scripts, there about 20 days of import were estimated.

By default, as usually used, the tilelive-copy (main import command generating tiles) doesn't use all available CPUs for several reasons. Since for OSM Scout Server we need tiles split into smaller databases covering parts of the World, it was simple to make parallel imports and write Makefiles that pushed the server to full usage.

Next, I had to optimize queries, pre-generate as many of them as possible to avoid repetition induced by the same queries run for tiles at different zoom levels, and PostGIS data storage layout.

All these optimizations resulted in dropping tile generation time (after PostGIS import) to about 4*24h, probably a bit less. I expect to get it somewhat faster after realizing that probably 1-2 days were slowed down significantly by huge ice polygons in Alaska, Canada, and Greenland. One way to reduce the polygons would be to run as many imports as possible leading to the increase in overall ambient temperature in these regions, but a faster way would be to split the polygins into the smaller ones. Splitting of the polygons will be tested the next time. My scripts are available at https://github.com/rinigus/mapbox-gl-importer .

In addition to imports, the styles have been reworked. For Mapbox GL, styles are described using a single JSON file which defines all the layers one-by-one. There are nice GUI utilities to work with the style file, but they all suffer from rather large amount of repetitive work. Namely, many layers use the same colors, similar logic, and so on. As a result, there is a lot of point-and-click or emacs/vi replace if you want to make different colorscheme (light and dark), for example.

I have setup a project for the styles used by OSM Scout Server that tries to address some of the repetitive work in describing the styles and modifying them: https://github.com/rinigus/mapbox-gl-styles . The styles are described by collection of layers (https://github.com/rinigus/mapbox-gl-styles/tree/master/styles/src/layers) that are used by all styles, style-specific variables (example for https://github.com/rinigus/mapbox-gl-styles/blob/master/styles/src/osmbright/variables.less) allowing to use LESS color transformations, and the list of layers used by specific style. Thus, for example, style with English names is made by trivial modification of variables file (but that's easy to do with emacs/vi as well by running replace on JSON) and dark style is made by changing colors only.

In future, when Mapbox GL Native will start supporting expressions better, it will be possible to describe style layers with less repetitions than its done now, but that would take some time till Mapbox GL Native will get it implemented.

If you wish to edit styles for OSM Scout Server, let me know. On significant interest, I will try to address the simplicity of visual feedback and providing tiles. At present, I am using an editor, OSM Scout Server on Linux PC, and Mapbox GL QML widget based viewer on Linux. Its surely possible to replace the viewer with web-based one (Mapbox GL JS) and probably make it easier to get tiles for testing.

At present, I included osmbright and osmbright-en as OSM Scout Server styles, with the dark styles expected in the future releases. Plan is to get to parity with Mapnik styles.

Hmm, that turned out to be rather long description of what has been done. Sorry :)

Enjoy using the server and updated maps.

seiichiro0185

2018-01-14, 16:09

karlos devel

2018-01-14, 16:20

same issues, me too. xperia x 5121

rinigus

2018-01-14, 16:21

At first, thanks again for your continued work on osmscout server and the surrounding infrastructure.

I updated to the latest 1.4.0 and downloaded the new maps for germany on my tablet. Unfortunately I have a Problem using it together with Poor Maps GL. In higher zoom levels all text on the maps (like streetnames and house numbers) and POI icons are somehow missing. with 1.3.0 and fitting maps it was working fine.

Does anyone else see this problem or did I miss something?

Just got reports regarding Spain (https://github.com/rinigus/osmscout-server/issues/233). I'll try to download Spain/Germany and see if its the same for me.

However, try to close Poor Maps GL and delete its cache via

rm -rf ~/.cache/harbour-poor-maps-gl

and try again. I suspect that maybe caching could influence it. Please let me know if it helped.

Edit Just tried with Barcelona and all seems to be fine. Just keep zooming in and you should get street names. Although, most probably cache cleaning is needed for it (I was doing it routinely)

rinigus

2018-01-14, 16:38

For Spain it just got confirmed - clearing cache helped and all works as expected.

So, please remove your application cache when Mapbox GL schema is changed. Right now it applies to Poor Maps GL and Laufhelden

seiichiro0185

2018-01-14, 16:44

Clearing the Poor Maps GL cache helped. It's now working as expected.

rinigus

2018-01-20, 20:24

I have just published a point update release: 1.4.1

In addition to updated translations, this release brings support for multiple Mapbox GL icon packs and is bundled with the dark Mapbox GL style (mc). Mapbox GL styles are under styles/mapboxgl/styles of the server source tree.

I will try to keep parity between light and dark styles allowing map client applications to switch automatically between map styles as for day/night maps. For example, Laufhelden already has day/night theme switch implemented which can be extended to the map as well.

rinigus

2018-02-02, 11:12

I am considering a modification in the import scripts to avoid getting too large databases in Mapnik. In particular, Germany, France, and maybe few more countries that

* are huge in terms of OSM data
* have sub-regions defined

will be made available only through sub-regions. All recommended backends are able to "glue" on fly the downloaded maps allowing users to download all the regions of Germany/France if there is such interest.

From my side, I will probably stop getting questions on why Germany/France maps don't show up with the logs showing broken SQLite database messages and me suspecting that its due to filesystem and/or sqlite driver limitations.

So, if there will be no protests to this plan, I might as well do it. However, if it turns out that many of you use these bigger datasets, we may continue as it is as well.

Note that, at this stage, its a proposal, not decided strategy

monkeyisland

2018-02-02, 12:23

HI Ringius on Jolla1 it is working with parts of the maps, but Poormaps take ages to rendering the maps especially for routing and searching.
But when there is no other possibility for me its okay.

rinigus

2018-02-02, 13:30

HI Ringius on Jolla1 it is working with parts of the maps, but Poormaps take ages to rendering the maps especially for routing and searching.
But when there is no other possibility for me its okay.

rendering, routing, and searching are actually 3 different parts. They could be all 3 slow or one of them in particular. While rendered tiles are cached between Poor Maps sessions, the search and routing is not. So, at least for rendering, it is using cache.

To test which of the components is the slowest, try to mix OSM Scout Server with online providers. For example, take Mapbox for tiles and try to search. That would tell you whether search is slow.

With the routing, try to route between current location and a location that has been found not as a search result but pointed on a map. That will tell you whether routing is slow.

Not sure I can fix it all, but maybe some things I can.

seiichiro0185

2018-02-03, 13:31

rinigus

2018-02-03, 14:35

I currently use the combined german database (although with the MapBox GL backend, but I guess it would be affected too), but it wouldn't be a big problem to just download all the different regions, which should work the same form the clients (PoorMaps GL) point of view if I understand correctly.

If you use Mapbox GL, then that is split already by tiles. Same goes for Valhalla routing. The only difference would be Geocoder-NLP which then searches in separate databases. And that's what it is doing if you keep the default settings.

rinigus

2018-02-21, 16:46

MartinK

2018-02-21, 18:37

I am happy to announce a point update (1.4.3) which includes updated translations and fixed a bug in Mapbox GL style leading to visible rivers in the new version.

What was supposed to be a simple update, turned out to be nothing like it. With the deprecation of libsystemd-daemon library and integration of its functionality to libsystemd, I suddenly had to link to full-blown systemd lib. Turned out, that Harbour wasn't very happy about it. After spending few nights on trying to get around it, I decided to stop distribution of OSM Scout Server via Harbour. This allows me to explicitly request systemd dependency in the package and focus on developing software instead of running around and avoiding Harbour barriers.

I have opened corresponding issue at https://github.com/sailfishos/sdk-harbour-rpmvalidator/issues/102 and the fix doesn't seem to be expected in the near future.

Thus, please use OpenRepos or OBS for getting the last versions.

This is really sad - it's already really hard to write non trivial native applications and apparently not well though out changes like this don't make this any better.

In any case, huge thanks for doing all this, including all the ridiculous workarounds and all the extra work you have to do to get what should be basic platform functionality. I've seen also many of your emails to sailfish-devel pointing to all these issues and I'm note sure I remember even a single case of Jolla actually doing the changes or fixing the issue you pointed out - that really does not look good for them & for Sailfish OS overall...

So thanks a lot, I really mean it! :)

And on a more positive note, we now finally have something like a roadmap (https://blog.jolla.com/sailfish-x-whats-next/) & new update (https://blog.jolla.com/sailfish-os-2-1-4-now-available-early-access/), so let's hope it's a sign Jolla might finally start to work on all the critical things that have been ignored for much too long.

I have been busy lately with other projects and couldn't work much on the server (trying to help @kimmoli with Onyx SFOS port and working on Presage keyboard with @martonmiklos @ljo). As soon as we get the keyboard into a good shape, I'll continue server development and work on resolving the outstanding issues, such as search for POIs without names and others.
I'll hopefully should be now (after a major conference & the obligatory February-plague) able to start in earnest on MapBoxGL integration in modRana and all the other important improvements that really need to happen. :)

rinigus

2018-02-21, 19:56

@MartinK, I don't think it is very sad, just natural evolution. Jolla's store is just one of the channels and I am planning to distribute the work through the remaining ones. Thanks for support!

PS: Actually, they did add SQLite as a possible dependency. So, there was a progress. But, in the end, each store has their policies and we should just distribute via the ones that are compatible.

paoletto

2018-02-22, 17:15

@rinigus, do you have the data generation pipeline documented somewhere? I'm interested in particular (/only) in generating the valhalla routing datasets.
Do you do it from raw OSM pbf?

rinigus

2018-02-22, 17:50

@rinigus, do you have the data generation pipeline documented somewhere? I'm interested in particular (/only) in generating the valhalla routing datasets.
Do you do it from raw OSM pbf?

Sure, see scripts and README at https://github.com/rinigus/osmscout-server/tree/master/scripts/import . Valhalla is quite simple, you'll have to install it on Linux PC and use valhalla tools for import. See my scripts on how to do it.

As for data, you would need OSM PBF as a source. Valhalla will need full region that you want to cover. In my case, it's Planet.pbf and it does need quite some RAM (running on server with 256gb, but probability smaller should do). But smaller regions should be fine with smaller requirements.

I was planning to start new import this weekend, but it may take now longer than 48 hours to cover all backends.

Note that if you want to have integrated with OSM Scout Server , it's bit trickier since data is now obtained from online servers. So, you may have to manage local copies outside of the provided data manager .

Let me know if you run into any problems

rinigus

2018-03-04, 11:24

Just to let you know: The updated maps are out on data.modrana.org. Enjoy!

XOleg

2018-03-04, 12:42

Just to let you know: The updated maps are out on data.modrana.org. Enjoy!

@rinigus, many thanks :-)

pagis

2018-04-01, 13:19

I am using version 1.4.3-10.29.1.jolla of OSMScout (updated from older versions).

https://www.dropbox.com/s/5q2itncznmceygb/Screenshot_20180401_003.png?dl=0

rinigus

2018-04-01, 14:07

I am using version 1.4.3-10.29.1.jolla of OSMScout (updated from older versions).

https://www.dropbox.com/s/5q2itncznmceygb/Screenshot_20180401_003.png?dl=0

This is very strange. I have expanded the error message in a new version. Please,

* uninstall current OSM Scout Server,
* check out that there are no files in /usr/share/harbour-osmscout-server . if there are any, delete the folder
* install server from http://repo.merproject.org/obs/home:/rinigus:/maps/sailfish_latest_armv7hl/armv7hl/harbour-osmscout-server-1.4.3+master.20180401135315.10.g6dbda9a-10.30.1.jolla.armv7hl.rpm . If you need i486, install the same from i486 repo.

Please report back with the error screenshot, if it occurs.

pagis

2018-04-01, 14:32

ok thanks, I am re-installing packages

pagis

2018-04-01, 16:49

,
* check out that there are no files in /usr/share/harbour-osmscout-server . if there are any, delete the folder
* install server from http://repo.merproject.org/obs/home:/rinigus:/maps/sailfish_latest_armv7hl/armv7hl/harbour-osmscout-server-1.4.3+master.20180401135315.10.g6dbda9a-10.30.1.jolla.armv7hl.rpm . If you need i486, install the same from i486 repo.

Please report back with the error screenshot, if it occurs.

I've re-installed OSMScout and new maps, there are no errors or warnings, however the map layer on WhoGo maps is blank.

https://www.dropbox.com/s/ijhf6ksxxfoh2fi/Screenshot_20180401_004.png?dl=0

https://www.dropbox.com/s/xp2gk5qqyke7gdj/Screenshot_20180401_005.png?dl=0

rinigus

2018-04-01, 16:53

I've re-installed OSMScout and new maps, there are no errors or warnings, however the map layer on WhoGo maps is blank.

https://www.dropbox.com/s/ijhf6ksxxfoh2fi/Screenshot_20180401_004.png?dl=0

https://www.dropbox.com/s/xp2gk5qqyke7gdj/Screenshot_20180401_005.png?dl=0

That's better. Do you have a profile with vector tiles enabled? If not, please do so and download maps. I hope this will resolve the issue

pagis

2018-04-01, 17:03

That's better. Do you have a profile with vector tiles enabled? If not, please do so and download maps. I hope this will resolve the issue

I thought the default profile was including vector tiles?

rinigus

2018-04-01, 17:08

I thought the default profile was including vector tiles?

No, not yet. Please change it to the one with vector tiles

peterleinchen

2018-04-01, 18:13

Do you have a more simple way than liberapay (speak paypal) for donations?

rinigus

2018-04-01, 18:46

Do you have a more simple way than liberapay (speak paypal) for donations?

thanks for asking! no, I don't. I like liberapay since its transparent - so all can learn from my experience. As for simplicity, I don't know much about it. I think it should be simple to go around the system by making payment a single time and just stop it. Some have done it before and its appreciated as much as anyone taking time and doing it.

peterleinchen

2018-04-01, 19:04

Yes, sure.
But you/one will need another account on another system (and probably only via credit card)...

rinigus

2018-04-01, 19:08

Yes, sure.
But you/one will need another account on another system (and probably only via credit card)...

No, not really. You could just transfer from there to your bank account directly and there is no need to have anything in between.

olf

2018-04-02, 15:16

@rinigus, AFAIU the OSM Scout Server Module: Fonts is currently only useful when providing Mapnik maps by OSM Scout Server.
As the Font Module is quite large (due to containing the Noto font), please clarify in its description on Openrepos (https://openrepos.net/content/rinigus/osm-scout-server-module-fonts) (if so?):
The Font Module benefits Mapnik users only.
It is not used for displaying vector maps or OSM Scout bitmap maps.
If it is a hard requirement for displaying Mapnik maps (i.e. no labels at all without it) or just enhancing the rendering of labels.

As I never used Mapnik offline maps in OSM Scout Server due to their huge size, I always wondered about aforementioned points.

P.S.: You may also consider altering the messages in OSM Scout Server, stating that the Font Module and / or Valhalla Module are "missing", as both are not needed for OSM Scout Server to work properly: I use libosmscout's maps and routing for a long time and they are working fine.

rinigus

2018-04-02, 16:49

MartinK

2018-04-02, 18:20

@olf, good point. There was a statement in Fonts module description, but I have added now a separate paragraph stressing it.

Fonts module is needed only if you use Mapnik backend.

Mapbox GL has the same fonts, but they are packaged differently, in Mapbox GL specific glyphs. These you download via OSM Scout Server and, in reality, they are larger than TTF fonts used by Mapnik. However, you probably store Mapbox GL data on SDCard and don't care much about it.

As for altering the messages - I'll take a look. I will link them to the selected profile. Opening an issue for it
Thinking about this - if the fonts module only contains the fonts and no executable code would it maybe make sense to turn it into a data pack that gets auto registered when users download the Mapnik rendering data ? IIRC there are already some global geographic datasets that serve a similar purpose.

This would automate one more thing, reduce number of packages users might need to install manually and possibly eliminate some out of space issues on devices with small rootfs volumes.

rinigus

2018-04-02, 18:38

Thinking about this - if the fonts module only contains the fonts and no executable code would it maybe make sense to turn it into a data pack that gets auto registered when users download the Mapnik rendering data ? IIRC there are already some global geographic datasets that serve a similar purpose.

This would automate one more thing, reduce number of packages users might need to install manually and possibly eliminate some out of space issues on devices with small rootfs volumes.

It depends on how we look into it.

These are Noto fonts which maybe attractive option for some other app to display full international characters set. If such interest will happen, we can repackage them accordingly.

Next, any investment of time into Mapnik maybe not so wise in long term. modRana is probably the only client which is using it and I would expect that, at some point, it will also switch to Mapbox GL. Which will leave us with the backend that nobody is using anymore. Use of the server on Linux is non-existent (probably only myself) and I don't see much reason for users to stick with Mapnik if suddenly all devices support Mapbox GL.

Moving over the data package is not trivial - not very difficult, but still would take time. It seems to me that this time can be spent better on other aspects of the map applications.

MartinK

2018-04-02, 23:45

Next, any investment of time into Mapnik maybe not so wise in long term. modRana is probably the only client which is using it and I would expect that, at some point, it will also switch to Mapbox GL. Which will leave us with the backend that nobody is using anymore.

Actually, I plan to add Mapbox GL support in parallel with the current tile based widget. The main aim is to both enable early testing with fallback to the current widget & support for Jolla 1 class devices where Mapbox GL is not supposed to run very well.

But if some of the recent user feedback turns out to be correct (eq. Mapbox GL actually runs fine on Jolla 1 & similar) then this option might not be needed & modRana might be able to switch to Mapbox GL on Sailfish OS exclusively. Then indeed the Mapnik rendering might not longer be really needed.

Use of the server on Linux is non-existent (probably only myself) and I don't see much reason for users to stick with Mapnik if suddenly all devices support Mapbox GL.

Actually, I'm a Fedora user & packager (I maintain for example the Fedora pyotherside & pydbus packages) and have been thinking it might not be a bad idea to package the OSM Scout Server for Fedora to make it available to the broader Linux user community.

The features and APIs OSM Scout Server provides are more or less unrivaled by any other project and I can see many uses even on plain desktops/notebooks, such as offline POI search and map display. While desktop users are mostly online these days there might still be very valid usecases for using offline geodata, such as bad connectivity while traveling or wanting to guard ones privacy from online mapping vendors.

Also a bit subjective reason - I develop modRana primarily on desktop so it would be nice to have the same set of APIs available even during general development. Also modRana uses more and more of the awesome OSM Scout Server functionality, so having modRana on desktop as full featured as on Sailfish OS would be nice. :)

In any cases this is at this point still more or less a nice to have compared to Sailfish OS support, but I'm ready to help with package review & be a package maintainer if needed. :) There is also the possibility to use Copr (https://copr.fedorainfracloud.org/), which is a Fedora service similar to PPAs/OBS to build OSM Scout Server packages rather than target inclusion as a "official" Fedora package from the start. :)

rinigus

2018-04-04, 06:10

1.5.0 is out which should give better feedback to the users regarding missing plugins and data. There is no other new functionality in the server this time.

@MartinK: I'll probably have to port server's GUI to QtQuick then. This should be relatively simple and would allow using the server simply in Linux. Right now, console interface is rather un-intuitive and hard to use. Corresponding issue is opened, let's see when I can make it.

As soon as proper GUI is ready for Linux, would be great to package it for Fedora. Although, it will require packaging of Valhalla, probably.

MartinK

2018-04-05, 00:19

@MartinK: I'll probably have to port server's GUI to QtQuick then. This should be relatively simple and would allow using the server simply in Linux. Right now, console interface is rather un-intuitive and hard to use.

I guess you can use Universal Components (https://github.com/M4rtinK/universal-components) to keep Sailfish OS specific stuff (pull down menus, buttons, fast scroll listview, etc.) while being able to run the same UI codebase with QtQuick Controls 2 on desktop.

Corresponding issue is opened, let's see when I can make it.

Thanks! You can definitely count me in for testing. :)

As soon as proper GUI is ready for Linux, would be great to package it for Fedora. Although, it will require packaging of Valhalla, probably.
I've looked at the Sailfish OS Valhalla packaging (https://github.com/rinigus/pkg-valhalla) and it looks pretty clean and not that Sailfish OS specific (even though I guess we could --enable-static=yes and use "normal" dynamic linking on Fedora). I'll try to build a package in Copr (https://copr.fedorainfracloud.org/) from the spec file and will report back my findings. :)

bomo

2018-04-08, 13:07

@rinigus does Valhalla support any routing options (fastest, shortest route) ?

rinigus

2018-04-08, 13:50

@rinigus does Valhalla support any routing options (fastest, shortest route) ?

@bomo, it does: https://github.com/valhalla/valhalla-docs/blob/master/turn-by-turn/api-reference.md#costing-models

It has to be called by a map client. There are lots of options which maybe of interest, see the README. Since OSM Scout Server just forwards the URL as a part of Valhalla call, there are no adjustments needed on the server part - it should be ready.

Xeno_PL

2018-04-12, 20:23

Hi rinigus,

I'm trying to use OSM Scout Server with WhoGo Maps, tiles work fine, but I hava a little problem with route finding. Do both start and end points have to be on the same map?
Problem is map of Poland has one basic for whole country (selected in map selection) and detailed maps for each area (voivideship). As long as both start and endpoint are in the same area, routing works, if it's in different area (map) WhoGo Maps says no resoults found (although I can see in server mesages Valhalla has found it)

rinigus

2018-04-19, 19:34

@MartinK: just read you have started on Fedora packaging, sounds great. There are probably few dependencies to go through. Note that I am using older libpostal, haven't worked on updating it yet.

Since its easier to test functionality on Desktop, I started porting GUI over to QtQuick 2. I hope to get it ready soon.

karlos devel

2018-04-19, 20:36

@rinigus do you know if valhalla could/will manage "offline traffic"?
thanks
/carlosgonz

rinigus

2018-04-20, 05:31

@rinigus do you know if valhalla could/will manage "offline traffic"?
thanks
/carlosgonz

The main issue is availability of the data. To my understanding, Mapzen started a project to collect such data, but got shut down. I think Mapbox (new home for Valhalla team) has an option to collect data for traffic on ios and android (not qt, if I understand correctly), so they may work on it too. But in Mapbox case, I didn't see any project that was supposed to make this data open. Although I may have missed it.

karlos devel

2018-04-20, 11:19

The main issue is availability of the data. To my understanding, Mapzen started a project to collect such data, but got shut down. I think Mapbox (new home for Valhalla team) has an option to collect data for traffic on ios and android (not qt, if I understand correctly), so they may work on it too. But in Mapbox case, I didn't see any project that was supposed to make this data open. Although I may have missed it.

Well, the unique data needed is "speed-limit" of each street, with that could be knowing the traffic in whatever street; basically is comparing the speed current with speed of gps. As an examp: If I driving in a street where the speed-limit is 45mph but I driving 15mph for whatever reason then valhalla could suspecting a traffic based on data of "speed-limit/speed-gps/time-running-in-15mph" then new route could be generated. But this way perhaps need constant monitoring. : )
But even this feature could be implemented too without any of data, by just applying the "Logistic".
Thanks

rinigus

2018-04-20, 13:05

Well, the unique data needed is "speed-limit" of each street, with that could be knowing the traffic in whatever street; basically is comparing the speed current with speed of gps. As an examp: If I driving in a street where the speed-limit is 45mph but I driving 15mph for whatever reason then valhalla could suspecting a traffic based on data of "speed-limit/speed-gps/time-running-in-15mph" then new route could be generated. But this way perhaps need constant monitoring. : )
But even this feature could be implemented too without any of data, by just applying the "Logistic".
Thanks

Devil is in the details. You are welcome to search for "traffic open data" and read about it. Lots of challenges to make it work. For example, http://opentransportmap.info/ provides "average daily Traffic Volumes for the whole EU", but daytime is only for "Pilot Cities". Not much use of such data for routing in practice.

karlos devel

2018-04-20, 15:20

Devil is in the details. You are welcome to search for "traffic open data" and read about it. Lots of challenges to make it work. For example, http://opentransportmap.info/ provides "average daily Traffic Volumes for the whole EU", but daytime is only for "Pilot Cities". Not much use of such data for routing in practice.

Are you rejecting?
First osmscout-server is opensource which you must accept shared ideas from any sources. There are diferent way to implement "offline traffic" not just following ideas of other devs as a example,I know this task is no easy,at the end I just sharing with you the idea, because I believe in your huge work that you already done, at the same time I am the first on started donating money for you work, of course I will keep this.

otsaloma

2018-04-20, 16:50

Your suggestion is very obscure and hardly generalizable. Traffic is inherently online and trying to hack vague data to an offline graph model would be a lot of trouble for little gain.

karlos devel

2018-04-20, 17:45

Your suggestion is very obscure and hardly generalizable. Traffic is inherently online and trying to hack vague data to an offline graph model would be a lot of trouble for little gain.
Hello otsaloma, thanks so much to come to clarify this.
I know you are professional in this area; Same to @rinigis. The problem is that you still do not understand my trick. I will explain more better I guess...

rinigus

2018-04-20, 19:40

karlos devel

2018-04-20, 19:55

If we even forget about online/offline aspect, I am not aware of any data that is available on traffic conditions and covering large fractions of the world. By just monitoring your current speed and comparing it to the speed limit, I cannot make a choice of the future road since I don't have any data to base it on.

Actually, the best offline selector in this case is you - as a driver. You see traffic ahead and can decide to either stay or turn. If you turn, Valhalla will reroute accordingly. And again, you could alter your path as well.

Yeap rerouting is the solution of this too, you right. I just was thinking on funny feature, where whogo could be detect traffic by using some offline datas then whogo-maps could give an option to the driver if accept go to other way. Really Really Funny.
On my side I like offline datas. So once again thanks @Rinigus/@Otsaloma for supporting offline datas.

rinigus

2018-04-28, 14:53

I have just merged Qt Quick 2 port of the server to the master branch. This allows to use the server on regular Linux with QML GUI. GUI is shaped in similar way to the GUI on SFOS, so, in theory, it should work on mobile Linux platforms as soon as they support QML QtQuick2. An obligatory screenshot is attached.

At present, there is a lot of code duplication in QML interfaces (Silica and QtQuick). Let's see, if we will get users and developer(s) on QtQuick side, I will organize it better by splitting more platform-dependent and independent code parts.

@MartinK, if you are packaging it for Fedora, maybe you could use qtquick version?

MartinK

2018-04-29, 01:41

@MartinK, if you are packaging it for Fedora, maybe you could use qtquick version?
Sure, that's definitely the plan & thanks a lot for working on the QtQuick GUI! :)

At the moment I'm going "up" from the OSM Scout Server dependencies, and I'm currently trying to get prime_server (https://github.com/kevinkreiser/prime_server), which is (as far as I can tell) a Valhalla dependency, to build on Fedora. I've hit some weird errors during compilation, but I'm suspecting it might have been some temporary Rawhide (https://fedoraproject.org/wiki/Releases/Rawhide) issue.

I'll give it another go tomorrow & will try to see if I'll get the same issues when I try the F27 and F28 mock targets.

BTW, once I manage to build the packages locally, I'll start putting them to this COPR repository:

https://fedoraproject.org/wiki/Releases/Rawhide

(COPR is basically the Fedora counterpart of OBS, with a significantly more sane and maintainable codebase than OBS has. :) )

rinigus

2018-04-29, 08:09

@MartinK: if you have issues, check which repository was used at mine OBS project. Some dependencies were built, at least at some point, against the patched versions. It may have been to resolve SFOS case, but maybe something else.

prime_server is ZMQ-based server used in Valhalla and is needed, indeed.

As for OSM Scout Server itself, you may want to look at PRO file and suggest a better way to specify valhalla's and fonts path. Since you have more experience with packaging, I would wait for your advice on it.

rinigus

2018-05-05, 16:20

New maps are out, Planet downloaded at the end of April. The main change in packaging of Valhalla. Now, I grouped the tiles somewhat differently and the planet is split into 1600 packages instead of 16000, as it was before.

The format is the same as before, so there are no incompatibilities with the current/bit older version of the server. All should work, as before.

For the end users, it would mean somewhat larger section of the Planet grabbed when downloading a country. While the neighboring territories will share the data, we end up with a bit more than we wanted to download.

For those who are involved in distribution of the maps, this suddenly makes it much easier to handle uploads/downloads. So, this change was made to make it a bit more sane syncing the servers.

Fellfrosch

2018-05-15, 15:36

DrYak

2018-05-15, 16:58

Regarding the size: remember that Rinigus has modified it so that now it can also serve vector data to application doing openGL (like WhoGo Maps, Laufhelden, etc.) and not only bitmap tiles.
That requires even more data on your sd card (see the new "Mapbox GL country-specific" data for each map).

Fellfrosch

2018-05-15, 17:05

rinigus

2018-05-15, 19:32

Hi rinigus, How do I download larger areas? I had some older maps on my device and decided to download the new maps. But now I'm quite unsure. Before I had a package "Germany" that was nearly 7 GB. But now Germany is spitted up in different federal states and some of them are even splitted up further. All german maps together would be much more than 20 GB. And I would have to choose a lot of areas seperately. Is that by intend?

I have dropped full Germany and France imports. But *DON'T PANIC*, its still possible to download them and it shouldn't take much more space than before.

Germany and France resulted in huge Mapnik SQLite databases (4GB+) that, due to limitations of FAT FS on SDCards, resulted in periodic questions "why my Germany download doesn't work?". An additional bonus was that it reduced a time of imports by few hours, reduced HDD requirements of the servers as well.

For this convenience on my side, users that want full Germany and France have to get them by regions. So, in your case, you would have to select all subregions (except Berlin which is in some other region as well, if I understood correctly). For Bayern, for example, you could just choose it and there is no need to get all its subregions since they should be all included in it.

Now, the sizes displayed in the menus correspond to the case where it is assumed that you will download only that region. This is to prepare you for the worst-case scenario. Fortunately, the regions in Germany overlap and will share the same data - will be downloaded and stored only once. Let's go through main backends, so you'll get the idea:

* Address parser is given for full country. So, in case of Germany, its 200MB is accounted for in each of the regions

* Geocoder-NLP and OSM Scout are the only backends that don't overlap, but are cut by the border. [technically, there is a minimal overlap, but its not important]

* Mapbox GL and Valhalla are using tiles. If the tile is touching a region, its accounted for in it. Fortunately, tiles are downloaded and unpacked only once.

In your case, you would have to unsubscribe from Germany and subscribe to all its regions. For curiosity, would be good interesting to know current (old schema) and the new disk usage. Note that I pulled Valhalla tiles in somewhat bigger tiles, but I would expect it didn't change much Germany since Valhalla tiles over there were big already before.

pichlo

2018-05-15, 19:34

The previously downloaded maps where already vector maps. There must be a different problem:
If you look inside the federal state Baden Württemberg, there you can download the whole federal state, size nearly 2 GB. But you also can download the further splitted areas which are all together 3,5 GB so it has something to do with the splitting. Maybe because the areas are overlapping.

That's exactly it. Overlapping. I had a similar problem. Downloading maps for Germany, I had to do that one state at a time. But then the total space taken on the memory card was much smaller than the grand total of all maps added together. When I say much I really mean MUCH. Like less than a half. I remember adding the last one, which alone was supposed to be about 1.5GB, added only about 200MB on the card.

Fellfrosch

2018-05-16, 06:35

@rinigus thanx for your detailed explanations. I've downloaded all the regions, and am absolutely satisfied. The total space taken doesn't differ to much (less than 500 MB) from the Full Germany download. So everything is fine and I'm totally flashed again by your great work and the ongoing support!!!

pichlo

2018-05-16, 12:46

The total space taken doesn't differ to much (less than 500 MB) from the Full Germany download.

I expect that to be simply because the maps have grown, not because of the fragmenting. I just updated all my maps (15 areas, including Germany already fragmented before the update) and the total space taken has increased by about 1 GB. I now have only 1 GB left free on a 16 GB SD card :)

Whilst on the topic, rinigus, would it be possible to update one map at a time? I did not see such an option, I had to update them all in one large go taking a few hours.

rinigus

2018-05-16, 17:43

Whilst on the topic, rinigus, would it be possible to update one map at a time? I did not see such an option, I had to update them all in one large go taking a few hours.

Not currently and not in my plans. In addition to work that is needed to make it happen, there are issues with the possible database format changes that would interfere with it. I am sure its possible to do, its just there are different priorities for me (geocoder, for example).

I presume it depends on requirements - whether to keep large regions on device or not. Notice, however, that these days we have two servers distributing the maps. So, if one is down, the second takes over. Since switching to two servers, I don't remember any downtime where the both were unavailable. Which makes it reasonable to consider downloading maps just few days before the trip. But that doesn't help if you have to travel constantly, though

pichlo

2018-05-16, 21:19

Thanks, rinigus. It was just a suggestion, definitely not a priority.

One possible workaround for the user is quite simple. Unsubscribe from all but one, update, subscribe again one at a time.

rinigus

2018-05-17, 05:37

One possible workaround for the user is quite simple. Unsubscribe from all but one, update, subscribe again one at a time.

That's not going to work. As soon as you accept update, new map definitions are loaded and the old maps are considered out-of-date. So, when you start plugging them back by re-subscribing, you would be doing so for new map version. And you would have to download them.

While you are presented the maps by regions, some backends are imported in one go. For example, Valhalla needs maps from the same import to calculate the routes. Hence, we cannot mix old and new maps. And thinking of it, response to original request - looks to be impossible to update only selected regions if you want to use Valhalla.

pichlo

2018-05-17, 06:41

That's not going to work. As soon as you accept update, new map definitions are loaded and the old maps are considered out-of-date. So, when you start plugging them back by re-subscribing, you would be doing so for new map version. And you would have to download them.

I know. That was the point. Downloading them one at a time, rather than all 15 in one go.

I should have included one step, purging the cache: Unsubscribe - purge - update - subscribe (and download).

pichlo

2018-05-18, 09:39

Do you have a more simple way than liberapay (speak paypal) for donations?

I second that. I was just trying to send you a donation but you do not make it exactly easy :(

rinigus

2018-05-18, 19:16

I second that. I was just trying to send you a donation but you do not make it exactly easy :(

Thanks for trying! I'll look into it a bit later...

As for OSM Scout Server, I am working on updating libpostal to 1.0 version. This will bring changes in Address parser database format leading to incompatibilities of the future maps with the current server version. I'll try to warn in advance, so you could plan map downloads and server updates.

rinigus

2018-05-25, 08:06

As mentioned above, I am working on updating libpostal to 1.0. While API was adopted fast, there were few issues that I had to hack around. The main is https://github.com/openvenues/libpostal/issues/351 , for which I seem to have workaround (at least the planet was imported).

I'll be testing it over the next few days. The tests over the last week (even before workaround) were promising. If all goes fine, I'll update the software and the maps via corresponding distribution channels. However, note that its rather large change and those who have to rely on the server in the next couple of weeks, please check for the feedback of the ones who updated first. Also, if you wish to delay the update, please get all the maps before the update will hit the servers. Otherwise, you'll be unable to do that later.

rinigus

2018-05-27, 04:06

New version is out: 1.6.0

This is a major upgrade and, due to the upgrade of libpostal API, the address parsing part of the maps is incompatible with the earlier versions. After upgrade, go to Map Manager, click "Check for updates" and proceed as usual. Below, is the description of what's new.

This version is the first one supporting QtQuick Controls GUI. So, it opens full GUI for Linux desktops. It could also simplify porting to Nemo, Ubuntu Touch, Plasma Mobile, if any porting is needed at all.

Upgrade of libpostal brings us a large work done by libpostal author in 2017. I would like to encourage you to test performance of the search and check whether your query was parsed correctly. From disk storage requirements side, the new libpostal doesn't use language parser dataset anymore which should reduce storage requirements by ~500MB on global dataset. Country datasets have changed as well, but that could be larger or smaller than before, depending on the country.

Geocoder now sorts the results somewhat differently to promote Glasgow (city) in front of the pub with the same name. This is done by preferring results with smaller number of admin levels (so Malmö kommun is preferred to Malmö city) and, for the same admin level results, with the shorter name.

Valhalla packaging changes were already in use earlier, but now the scripts are part of the release. I am also not showing country selection when its not needed in the main screen (its used only if geocoder is limited to one country or libosmscout is in use).

The translations have been updated and Dutch (BE) added to the list by @pljmn. Thanks for all the translators!

Current libpostal SFOS adaptation uses one reversed commit. Such adaptation allowed me to import the planet, but it may have some side effects. So, please tell if the server is killed for one reason or another. The corresponding issue has been opened at libpostal repository, but its not properly resolved yet.

Finally, as a simpler version, I added bitcoin donation link. Please don't ask for paypal, I don't want to engage with it at the moment. Bitcoin, as liberapay, gives you the list of transactions. So, all donations are transparent in terms of time and amounts.

New updated libpostal and geocoder-nlp datasets are available, enjoy the release.

MartinK

2018-05-28, 20:37

New version is out: 1.6.0
This version is the first one supporting QtQuick Controls GUI. So, it opens full GUI for Linux desktops. It could also simplify porting to Nemo, Ubuntu Touch, Plasma Mobile, if any porting is needed at all.

Nice! :) On a related note, I finally managed to have some progress in my Fedora packaging efforts:

prime_server
- PR: https://github.com/rinigus/pkg-prime_server/pull/1 (already merged, thanks! :) )
- builds fine in Fedora COPR
- opened an upstream issue to clarify licensing: https://github.com/kevinkreiser/prime_server/issues/70

valhalla
- PR: https://github.com/rinigus/pkg-valhalla/pull/1
- fails during build in COPR due to some weird protocol buffer errors, details are in the pull request

What OSM Scout Server dependencies would you recommend next ? Looking at the OSM Scout Server spec file (https://github.com/rinigus/osmscout-server/blob/master/rpm/harbour-osmscout-server.spec):

already available
- marise (0.2.4) is already packaged
- mapnik (3.0.19) is already packaged
- libmicrohttpd (0.9.59) is already packaged
- tokyocabinet (1.4.48) is already packaged

missing in Fedora
- libosmscout is not yet packaged
- libpostal is not yet packaged

rinigus

2018-05-29, 06:38

Nice! :) On a related note, I finally managed to have some progress in my Fedora packaging efforts:

prime_server
- PR: https://github.com/rinigus/pkg-prime_server/pull/1 (already merged, thanks! :) )
- builds fine in Fedora COPR
- opened an upstream issue to clarify licensing: https://github.com/kevinkreiser/prime_server/issues/70

valhalla
- PR: https://github.com/rinigus/pkg-valhalla/pull/1
- fails during build in COPR due to some weird protocol buffer errors, details are in the pull request

What OSM Scout Server dependencies would you recommend next ? Looking at the OSM Scout Server spec file (https://github.com/rinigus/osmscout-server/blob/master/rpm/harbour-osmscout-server.spec):

already available
- marise (0.2.4) is already packaged
- mapnik (3.0.19) is already packaged
- libmicrohttpd (0.9.59) is already packaged
- tokyocabinet (1.4.48) is already packaged

missing in Fedora
- libosmscout is not yet packaged
- libpostal is not yet packaged

Marisa, libmicrohttpd, tokyocabinet should be OK

Mapnik: Mapnik in 3.0.x line still doesn't include TWKB support for SQLite databases. The corresponding PR has been merged upstream (https://github.com/mapnik/mapnik/pull/3660) and I am currently applying patch while building Mapnik for SFOS (https://github.com/rinigus/pkg-mapnik/blob/master/rpm/mapnik.twkb.patch). Without it, Mapnik maps were 2x larger, if memory serves me right. Hence, all distributed maps use this format and vanilla 3.0.x Mapnik will be unable to render them. Se, we'll need either to convince Fedora packagers to add this patch or package it separately and link statically to the server.

libosmscout: I am using rather old version of it, see https://github.com/rinigus/libosmscout/tree/sailfish . Newer versions have somewhat different API and I decided not to get involved into keeping it up to date. So far, nobody volunteered to keep this part of code in sync with the latest libosmscout releases. We can also skip libosmscout for this packaging round and compile the server without it (there is a disable option in qt pro file). The release that I use is https://github.com/rinigus/libosmscout/releases/tag/0.0.git.20170521-1

libpostal: Current libpostal has a bug that can be triggered by searching for "Хозтовары" (issue https://github.com/openvenues/libpostal/issues/351). Please use https://github.com/rinigus/pkg-libpostal with devel branch (https://build.merproject.org/package/view_file/home:rinigus:maps/libpostal/_service?expand=1); loads https://github.com/rinigus/libpostal/tree/test_expansion_fail_fix branch. Usually, libpostal developer is fast in responding to issues. For some reason, he hasn't been that active lately. But I am sure it will get fixed and we can later switch to the upstream branch.

So, while lots of functionality is provided by packages, there is still quite some work that has to be done behind the scenes to make it all work in practice :)

Thanks for working on it, looking forward to see the server in mainstream Linux.

DrYak

2018-05-29, 09:52

Big Kudos to you guys !
http://www.sympato.ch/smileys/megabounce.gif

MartinK

2018-06-02, 11:39

Marisa, libmicrohttpd, tokyocabinet should be OK

Mapnik: Mapnik in 3.0.x line still doesn't include TWKB support for SQLite databases. The corresponding PR has been merged upstream (https://github.com/mapnik/mapnik/pull/3660) and I am currently applying patch while building Mapnik for SFOS (https://github.com/rinigus/pkg-mapnik/blob/master/rpm/mapnik.twkb.patch). Without it, Mapnik maps were 2x larger, if memory serves me right. Hence, all distributed maps use this format and vanilla 3.0.x Mapnik will be unable to render them. Se, we'll need either to convince Fedora packagers to add this patch or package it separately and link statically to the server.
Oh, so the patch is already upstream, just not yet in a released version of Mapnik? For now I've just built the Fedora Mapnik package with the patch on top in COPR:
https://copr.fedorainfracloud.org/coprs/m4rtink/osm-scout-server/build/761846/

Then once we are far enough to submit OSM Scout Server for package review (so that it can go the the official Fedora repos) we can either ask the Mapnik maintainers to include the patch or possibly a new upstream version including it has been released (and the Mapnik maintainers seem to track upstream releases pretty closely).

libosmscout: I am using rather old version of it, see https://github.com/rinigus/libosmscout/tree/sailfish . Newer versions have somewhat different API and I decided not to get involved into keeping it up to date. So far, nobody volunteered to keep this part of code in sync with the latest libosmscout releases. We can also skip libosmscout for this packaging round and compile the server without it (there is a disable option in qt pro file). The release that I use is https://github.com/rinigus/libosmscout/releases/tag/0.0.git.20170521-1

Is there some functionality libosmscout provides that is not covered by other libraries (Valhalla/libpostal/Mapnik/etc.) ? I think we can indeed skip libosmscout for the initial packaging round, as even if it builds fine in COPR, it would be weird to submit an old version of libosmscout for package review if we know OSM Scout Server will not work with a new version if it ever gets updated.

libpostal: Current libpostal has a bug that can be triggered by searching for "Хозтовары" (issue https://github.com/openvenues/libpostal/issues/351). Please use https://github.com/rinigus/pkg-libpostal with devel branch (https://build.merproject.org/package/view_file/home:rinigus:maps/libpostal/_service?expand=1); loads https://github.com/rinigus/libpostal/tree/test_expansion_fail_fix branch. Usually, libpostal developer is fast in responding to issues. For some reason, he hasn't been that active lately. But I am sure it will get fixed and we can later switch to the upstream branch.

I've tried to build libpostal yesterday and while the packing & the software itself (a C library with few depndencies) looks fairly simple, the build failed with some rather weird errors:

/bin/sh ../libtool --tag=CC --mode=link gcc -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/usr/local/libpostal/data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/usr/local/libpostal/data/libpostal"' -g -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -L/usr/local/lib -o test_libpostal test_libpostal-test.o test_libpostal-test_expand.o test_libpostal-test_parser.o test_libpostal-test_transliterate.o test_libpostal-test_numex.o test_libpostal-test_trie.o test_libpostal-test_string_utils.o test_libpostal-test_crf_context.o ../src/libpostal.la -lm
libtool: link: gcc -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR=\"/usr/local/libpostal/data/libpostal\" -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR=\"/usr/local/libpostal/data/libpostal\" -g -Wl,-z -Wl,relro -Wl,-z -Wl,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o .libs/test_libpostal test_libpostal-test.o test_libpostal-test_expand.o test_libpostal-test_parser.o test_libpostal-test_transliterate.o test_libpostal-test_numex.o test_libpostal-test_trie.o test_libpostal-test_string_utils.o test_libpostal-test_crf_context.o -L/usr/local/lib ../src/.libs/libpostal.so -lstdc++ -lm
/usr/bin/ld: test_libpostal-test.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_expand.o: relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_parser.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_transliterate.o: relocation R_X86_64_32 against symbol `greatest_type_info_string' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_numex.o: relocation R_X86_64_32 against symbol `greatest_type_info_string' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_trie.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_string_utils.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_crf_context.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:575: test_libpostal] Error 1
make[2]: Leaving directory '/builddir/build/BUILD/libpostal-1.0.0/test'
make[1]: *** [Makefile:453: all-recursive] Error 1
make[1]: Leaving directory '/builddir/build/BUILD/libpostal-1.0.0'
make: *** [Makefile:362: all] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.TOmV84 (%build)

Full log: https://copr-be.cloud.fedoraproject.org/results/m4rtink/osm-scout-server/fedora-rawhide-x86_64/00761920-libpostal/builder-live.log

The spec file used: https://github.com/M4rtinK/pkg-libpostal/blob/master-fedora_packaging/rpm-fedora/libpostal.spec

Failed build in COPR: https://copr.fedorainfracloud.org/coprs/m4rtink/osm-scout-server/build/761920/

Googling for "Nonrepresentable section on output" I found some hints that it might be related to PIE/PIC and linking. Any ideas/pointers what to try to fix the build ? Thanks in advance! :)

So, while lots of functionality is provided by packages, there is still quite some work that has to be done behind the scenes to make it all work in practice :)

Thanks for working on it, looking forward to see the server in mainstream Linux.
It's a important project to which I don't know about any alternatives - so it should also be as widely available as possible, even if it requires some work. :)

rinigus

2018-06-02, 12:52

Oh, so the patch is already upstream, just not yet in a released version of Mapnik? For now I've just built the Fedora Mapnik package with the patch on top in COPR:
https://copr.fedorainfracloud.org/coprs/m4rtink/osm-scout-server/build/761846/

Then once we are far enough to submit OSM Scout Server for package review (so that it can go the the official Fedora repos) we can either ask the Mapnik maintainers to include the patch or possibly a new upstream version including it has been released (and the Mapnik maintainers seem to track upstream releases pretty closely).

Its in the master branch for a year, I think. However, that's for upcoming 3.1 series and they don't seem to include it into 3.0 versions.

Is there some functionality libosmscout provides that is not covered by other libraries (Valhalla/libpostal/Mapnik/etc.) ? I think we can indeed skip libosmscout for the initial packaging round, as even if it builds fine in COPR, it would be weird to submit an old version of libosmscout for package review if we know OSM Scout Server will not work with a new version if it ever gets updated.

No, there is none. In this context, it just allows to use smaller databases. And I agree, it will be weird to submit old libosmscout version.

I've tried to build libpostal yesterday and while the packing & the software itself (a C library with few depndencies) looks fairly simple, the build failed with some rather weird errors:

/bin/sh ../libtool --tag=CC --mode=link gcc -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/usr/local/libpostal/data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/usr/local/libpostal/data/libpostal"' -g -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -L/usr/local/lib -o test_libpostal test_libpostal-test.o test_libpostal-test_expand.o test_libpostal-test_parser.o test_libpostal-test_transliterate.o test_libpostal-test_numex.o test_libpostal-test_trie.o test_libpostal-test_string_utils.o test_libpostal-test_crf_context.o ../src/libpostal.la -lm
libtool: link: gcc -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR=\"/usr/local/libpostal/data/libpostal\" -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR=\"/usr/local/libpostal/data/libpostal\" -g -Wl,-z -Wl,relro -Wl,-z -Wl,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o .libs/test_libpostal test_libpostal-test.o test_libpostal-test_expand.o test_libpostal-test_parser.o test_libpostal-test_transliterate.o test_libpostal-test_numex.o test_libpostal-test_trie.o test_libpostal-test_string_utils.o test_libpostal-test_crf_context.o -L/usr/local/lib ../src/.libs/libpostal.so -lstdc++ -lm
/usr/bin/ld: test_libpostal-test.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_expand.o: relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_parser.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_transliterate.o: relocation R_X86_64_32 against symbol `greatest_type_info_string' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_numex.o: relocation R_X86_64_32 against symbol `greatest_type_info_string' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_trie.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_string_utils.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_crf_context.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:575: test_libpostal] Error 1
make[2]: Leaving directory '/builddir/build/BUILD/libpostal-1.0.0/test'
make[1]: *** [Makefile:453: all-recursive] Error 1
make[1]: Leaving directory '/builddir/build/BUILD/libpostal-1.0.0'
make: *** [Makefile:362: all] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.TOmV84 (%build)

Full log: https://copr-be.cloud.fedoraproject.org/results/m4rtink/osm-scout-server/fedora-rawhide-x86_64/00761920-libpostal/builder-live.log

The spec file used: https://github.com/M4rtinK/pkg-libpostal/blob/master-fedora_packaging/rpm-fedora/libpostal.spec

Failed build in COPR: https://copr.fedorainfracloud.org/coprs/m4rtink/osm-scout-server/build/761920/

Googling for "Nonrepresentable section on output" I found some hints that it might be related to PIE/PIC and linking. Any ideas/pointers what to try to fix the build ? Thanks in advance! :)

Try to drop https://github.com/M4rtinK/pkg-libpostal/blob/master-fedora_packaging/rpm-fedora/libpostal.spec#L33

CFLAGS="$CFLAGS -fPIC -lstdc++"
CXXFLAGS="$CXXFLAGS -fPIC"

I had to use it for SFOS packaging, but it shouldn't be needed for you.

It's a important project to which I don't know about any alternatives - so it should also be as widely available as possible, even if it requires some work. :)

Well, there are servers around (valhalla, tile servers, and geocoders) that are surely better in many aspects. Probably, the main advantage is that all map data is packaged and available online. It helps that you "just" need to install only one server as well [which means that the hard part of install is done by packagers].

MartinK

2018-06-02, 23:15

Try to drop https://github.com/M4rtinK/pkg-libpostal/blob/master-fedora_packaging/rpm-fedora/libpostal.spec#L33

CFLAGS="$CFLAGS -fPIC -lstdc++"
CXXFLAGS="$CXXFLAGS -fPIC"

I had to use it for SFOS packaging, but it shouldn't be needed for you.

Tried that, still the same errors (at least AFAICT):
/bin/sh ../libtool --tag=CC --mode=link gcc -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/usr/local/libpostal/data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/usr/local/libpostal/data/libpostal"' -g -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -L/usr/local/lib -o test_libpostal test_libpostal-test.o test_libpostal-test_expand.o test_libpostal-test_parser.o test_libpostal-test_transliterate.o test_libpostal-test_numex.o test_libpostal-test_trie.o test_libpostal-test_string_utils.o test_libpostal-test_crf_context.o ../src/libpostal.la -lm
libtool: link: gcc -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR=\"/usr/local/libpostal/data/libpostal\" -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR=\"/usr/local/libpostal/data/libpostal\" -g -Wl,-z -Wl,relro -Wl,-z -Wl,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o .libs/test_libpostal test_libpostal-test.o test_libpostal-test_expand.o test_libpostal-test_parser.o test_libpostal-test_transliterate.o test_libpostal-test_numex.o test_libpostal-test_trie.o test_libpostal-test_string_utils.o test_libpostal-test_crf_context.o -L/usr/local/lib ../src/.libs/libpostal.so -lm
/usr/bin/ld: test_libpostal-test.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_expand.o: relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_parser.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_transliterate.o: relocation R_X86_64_32 against symbol `greatest_type_info_string' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_numex.o: relocation R_X86_64_32 against symbol `greatest_type_info_string' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_trie.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_string_utils.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_crf_context.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:575: test_libpostal] Error 1
make[2]: Leaving directory '/builddir/build/BUILD/libpostal-1.0.0/test'
make[1]: *** [Makefile:453: all-recursive] Error 1
make[1]: Leaving directory '/builddir/build/BUILD/libpostal-1.0.0'
make: *** [Makefile:362: all] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.NmocgJ (%build)
Full log: https://copr-be.cloud.fedoraproject.org/results/m4rtink/osm-scout-server/fedora-rawhide-x86_64/00762152-libpostal/builder-live.log

Link to the failed build: https://copr-be.cloud.fedoraproject.org/results/m4rtink/osm-scout-server/fedora-rawhide-x86_64/00762152-libpostal/builder-live.log

Spec file diff from previous attempt: http://copr-dist-git.fedorainfracloud.org/cgit/m4rtink/osm-scout-server/libpostal.git/commit/?id=94d518b03917c562bf99d19f5b7c50aef7480688

Googling for "relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC" uncovered some discussion about cases in other programs when trying to link relocatable and non-relocatable code together, so I wonder if we are seeing something like that as well. Looking at the test subfolder where this is happening, it seems to have some custom flags defined here:
https://github.com/openvenues/libpostal/blob/master/test/Makefile.am
So maybe it compiles the testing code wrong or something like that ?
Also I wonder if the tests could just be disabled, but that feels like a hack.

Well, there are servers around (valhalla, tile servers, and geocoders) that are surely better in many aspects. Probably, the main advantage is that all map data is packaged and available online. It helps that you "just" need to install only one server as well [which means that the hard part of install is done by packagers].
Yes, that's what I think OSM Scout Server is best in the world at - making all the power of the existing open geodata & libraries seamlessly available both to application developers (language independent & well documented common API) and users (super easy offline data downloading & updates + data sharing with all apps using the OSM Scout Server API).

I don't think any existing solution can come close to this - otherwise developers would have to handle every lib themselves (compilation/packaging/bundling) & library APIs are often limited (C/C++ only, API that requires a heavy server software to run, etc.), not to mention all the hassles with offline data management that would have to be solved more or less separately for every application.

From the user side, there is super easy & unified offline data management & all the nice features made possibly by the easily accessible APIs. :)

rinigus

2018-06-03, 05:27

MartinK

2018-06-11, 01:09

@MartinK, it builds static and shared versions of the library. I would suggest to add --disable-static to configure options and try again. Since static lib is built after the shared one, it may got confused and linked tests to the static libs.

Does it build on your PC?
So looks like the tests have been somehow at fault, as they are compiled against libpostal in some ultra weird manner. So for now I've hard-disabled them (dropped their compilation from the makefile) and the build finally runs to the end. This should be good for now, later one for the actual package review something more robust (possibly including reporting the issue upstream) might be needed. I'll open a PR for the libpostal packaging soon, just have to cleanup my changes a bit.

Anyway, as far as I can tell this concludes the missing dependencies & I can finally start packaging OSM Scout Server itself. :)

rinigus

2018-06-11, 03:53

So looks like the tests have been somehow at fault, as they are compiled against libpostal in some ultra weird manner. So for now I've hard-disabled them (dropped their compilation from the makefile) and the build finally runs to the end. This should be good for now, later one for the actual package review something more robust (possibly including reporting the issue upstream) might be needed. I'll open a PR for the libpostal packaging soon, just have to cleanup my changes a bit.

Anyway, as far as I can tell this concludes the missing dependencies & I can finally start packaging OSM Scout Server itself. :)

Sounds great! Please note that I haven't tested any 'make install' on desktop Linux. The path for directories is coded in pro-file and I was waiting till you get here and help me to fix it. So, if you know what's the common way to specify paths then please do so and send PRs :)

rinigus

2018-06-17, 07:37

New release is out: 1.7.1. With the 1.7.0, followed by 1.7.1, I focused on geocoder enhancements.

New geocoder supports aliases when you search for data using guide search (Nearby). Aliases, as suggested by @palbr (thank you!), are pulled from https://wiki.openstreetmap.org/wiki/Nominatim/Special_Phrases with the used ones listed at https://rinigus.github.io/osmscout-server/tags/ . The list is also available as an API call, via poi_types URL, allowing us later to extend WhoGo Maps and modRana to automatically provide it to the users (if the client developers will approve it). Tag aliases are also used when returning the search results to identify objects in a localized manner.

Geocoder now imports and allows you to search for POIs without names (playground, ATM, toilets, ...), Again, suggested by @palbr. This is done for guide (nearby) search. When using guide search, its also possible to separate search by name or type when looking for POI (at least one has to be specified). Released WhoGo Maps already supports it, modRana is under development. Corresponding map update has been released and available if you check for updates in map manager. Note that only geocoder-nlp maps were updated this time.

As reported by @legacychimera247, some villages were missing in the search. This is fixed now, but it looks like I have inserted these missing villages into a bit high level in the hierarchy. This will be fixed in near future.

Many more types are now imported and supported by geocoder. Let me know if something important is missing.

The latest version of translations has been also pulled into the package. Thank you to all the translators doing great job!

feedme

2018-06-17, 18:26

Hello,
I am deeply in love to osm!
But yhis darling does not allow to use sdcard .
I created a directory to sdxard ( devel-su) gave rights and osm stopped nagging "no rights" but then there is open database error.

( there is / nemo/ Maps.OSM) I also tried to copy that but again even after givin rights to/ run/nemo I have a problem , not able to create...)

running sailfish X

peterleinchen

2018-06-17, 20:51

I do not think it has something to do with X or Jolla C (which I am using).
Why using devel-su (root) at all?
Just create any directory as normal user (nemo) and assign this directory within settings of OSM. Done.

DrYak

2018-06-19, 10:03

But yhis darling does not allow to use sdcard .
I created a directory to sdxard ( devel-su) gave rights and osm stopped nagging "no rights" but then there is open database error.

( there is / nemo/ Maps.OSM) I also tried to copy that but again even after givin rights to/ run/nemo I have a problem , not able to create...)

running sailfish X

My own strategy (inherited back from older setups with other mapping software) :

OSM server set to use : /home/nemo/Maps/OSM

/home/nemo/Maps is a symlink that point to /run/media/nemo/SDCard/Maps

/run/media/nemo/SDCard is a SDXC card reformatted to a linux partition (BTRFS in my case, but will work as well with EXT4 or F2FS).

/run/media/nemo/SDCard/Maps and /run/media/nemo/SDCard/Maps/OSM
are owned by nemo:nemo,
access rights 0775 (rwxrwxr-x)

works flawlessly.

rinigus

2018-07-09, 10:59

New maps are out and available for download.

It looks like the last time I uploaded older Mapbox GL tiles. This time, all the data should be up to date, about 1-2 weeks old extract from OSM Planet.pbf.

rinigus

2018-07-12, 20:18

New version is out - 1.8.0

This version is linked directly to Valhalla via its C++ API. As a result, OSM Scout Server provides Valhalla's exported functionality in its own process. Earlier, the server had to start Valhalla separately and work as a proxy between map clients and Valhalla. Now, this extra HTTP connection is skipped.

For end users, such integration means that, after upgrading to 1.8.0, OSM Scout Server Module: Route can be uninstalled. Its not used anymore.

For map client developers, there are lots of exciting features exported, in addition to routing. Some of them are:

* map matching - for just-in-time snapping position to the road/path network; for exporting recorded GPX as a set of navigation instructions.

* time-distance matrix calculation between set of locations.

* finding optimized route between set of locations (so the order and route are optimized).

* isochrone - for example, where can I get in 30 minutes by car from here? see example at https://cdn-images-1.medium.com/max/800/0*PPRFkALqn7HoMo_8.png

Corresponding section in README: https://github.com/rinigus/osmscout-server#valhalla-api-for-routing-map-matching-and-other-services .

Note that while elevation service is exported as well, I don't distribute the data for it. I will look into it, but it sounded like it takes 1.2TB for a planet - bit excessive for mobile. If they find the way to reduce the data requirements, would be great way to get elevation-aware routing and its visualization.

For packagers (@MartinK), the server doesn't need anymore prime_server, zmq. I was made aware of this Valhalla C++ API few days ago (heard about it earlier when they started working on it in 2017, but didn't check the progress). I made a new packaging for Valhalla (https://github.com/rinigus/pkg-valhalla-lite) that is preferred for the compilation of the server. This packaging and the full Valhalla packaging (https://github.com/rinigus/pkg-valhalla) scripts were also modified to use cmake, as they switched the build system.

As always, the translators have updated the translations. I will try to remember to release point releases 1.8.x this time and get translations up-to-date. Sorry for delays.

I have added car styles for Mapbox GL maps. These are right now just increasing the priority of the gas stations, parking spaces, and such. Later we can maybe polish it further, feel free to think on how to improve them :)

I am planning to work on map matching and exploring it with the map client. Very rough POC has been done and my phone is able to map match almost in real time already with the current API (you can see the location moving as you are, but with the direction found using map matching). So, the early results are promising, but there is still much to do.

rinigus

2018-07-13, 07:23

Quick notice: WhoGo Maps (and probably Poor Maps) check whether routing module is installed when they list available routers. So, keep the routing module on device until corresponding change is done in WhoGo Maps: https://github.com/otsaloma/whogo-maps/pull/45

rinigus

2018-07-23, 08:10

New release is out: 1.9.0

This release brings OSM Scout Server to D-Bus and makes it very simple to use map matching functionality in QML apps.

Via D-Bus, the server allows now to establish a session with it by the external application and receive current street name, speed limit and the speed assumed by the routing engine for cars.

For QML, I made a QML type that can utilize this functionality: https://github.com/rinigus/positionsource-mapmatched-qml . It is expected to be a drop in replacement for PositionSource in the code of the application. When map matching mode is requested, it will establish a contact with OSM Scout Server, wake it up via systemd if users have that feature enabled, and will start providing map matched coordinates and other data. To simplify publishing, I would suggest to copy PositionSourceMapMatched.qml into the source tree of the application and include it. Example usage is shown in https://github.com/rinigus/positionsource-mapmatched-qml/blob/master/example/main.qml

PositionSourceMapMatched also simplifies app development by providing facilities for testing in the absence of GPS signal and timing information. See description in README and example.

As with regards to timing, when measured on 1+X (onyx) during ~20 min drive, it takes on average <40ms per single call from QML to map matching service via D-Bus, for processing its response, until the position and associated data are updated. The longest call over that test was <100ms.

The current DBus API can be extended to include other functionality of the server, if there is interest.

In addition, map matching support can be extended further to include other data provided by Valhalla. One example use is to utilize map matching to follow navigation progress. Valhalla's edges all have global IDs and one can just check the progress along the edge, the current edge, and the expected edge on the basis of expected route. While making navigation instructions processing efficient, that would make processing Valhalla's specific.