Importing List of POIs into Maemo Mapper 2.3
 

First of all, let me just say that this is a truly awesome program. Kudos to gnuite and the others contributing to the project.

I will be traveling to NZ next month, and have been working on obtaining offline content. Downloading maps has been working fine, so I've now turned to POIs. There is an impressive collection (40,000+) available from Zenbu (www.zenbu.co.nz), and I managed to convert the native format (.CSV file) to .GPX using a program called GPSBabel, and then imported the waypoints into Mapper. The import took quite a while!

My question is whether anyone has advice or suggestions as to how I can best map the information in the Zenbu file to Mapper. The Zenbu information does not have "categories" as such, it has "tags" that do not easily map to categories. Here are the first 3 lines of the .csv file, which shows the available fields:

What's too bad is that most of this information gets lost in the shuffle. When GPSBabel turns this into .GPX, it just takes the information in the "name" field and puts it into <name>, <cmt>, and <desc> tags, like so:

I messed around with changing the headings in the first line of the .CSV file, but that did not help. I imagine that with some tweaking, either GPSBabel or some other program could assign the tags more intelligently, and mabe even concatenate fields in some reasonable manner (e.g. put address, phone, hours, and website into the <desc> tag separated by line breaks). Any suggestions on a good program or technique for doing this?

The folks at the New Zealand Open GPS Maps Project (gwprojects.orcon.net.nz/gps/) have figured out a way to get all of this information into useful waypoints for Garmin units--including icons appropriate to the type of waypoint, inclusion of addresses & phone numbers, etc. Is there any reasonable prospect of doing something similar with Mapper?

Re: Importing List of POIs into Maemo Mapper 2.3
 

OK, after trying a lot of things, and some greatly-appreciated help from the folks on the NZ Open GPS Project website, I am much closer to being able to use the data! After jumping through various hoops, I can generate a .gpx file that contains all of the information I think can be squeezed into Maemo Mapper. The .gpx file does import into Google Earth and various other utilities. However, the POI --> Import . . . menu item in mapper does not work. I get the error "Error parsing GPX file". Unfortunately I have no idea what the parser thinks is wrong with it. Maybe something to do with the categories? I read somewhere that categories were handled through the <sym> tag, but perhaps that is wrong. Does anyone have any insight into what might be wrong with the following .gpx format?

Re: Importing List of POIs into Maemo Mapper 2.3
 

Erm, alternatively, if anyone could provide an example of a POI in .gpx format that successfully loads lat, lon, label, description, and category information into Maemo Mapper, that would be a really helpful start. (And does the category have to already exist in Mapper before loading to succeed?).

Update: I found the source of the parsing problem . . . Mapper did not like the stuff in the <gpx> tag, so when that is simplified to "<gpx>", and the <metadata> tag is taken out entirely, Mapper loads it just fine--except for the category. Regardless of whether the category already exists in mapper or not, the <sym> tag gets ignored and the category is set to the "default" category. So if anyone has a suggestion about how to successfully load category from a .gpx file, that would be most appreciated.

Re: Importing List of POIs into Maemo Mapper 2.3
 

abby_normal

If you run Windross there is always Poi_Loader :-

http://www.internettablettalk.com/fo...ght=Poi_Loader

Rick

Re: Importing List of POIs into Maemo Mapper 2.3
 

Thanks! I did know about this app from browsing the forums, and it looks quite useful. I will try it, but as the developer, can you tell me how I should prepare my .GPX file so your app will be able to read the categories and input them into the database? Should I put the category in a <sym> tag, like so:

<sym>Police Station</sym>

. . . or do I need to use a different tag, or even some other category identifier like a number? Since I am looking to import around 40,000 POIs, manually setting the category for each one isn't an option.

Re: Importing List of POIs into Maemo Mapper 2.3
 

I have no experience with importing POI but you should try a xml tool to validate your gpx file against the doctype definition mentioned in the qpx file quoted above (http://www.topografix.com/GPX/1/1/gpx.xsd).

Re: Importing List of POIs into Maemo Mapper 2.3
 

I appreciate the suggestion! For what it is worth, the example cited above validates against that schema, according to this NIST tool (Xerces/Jing parsers). But I guess technically valid is not enough to get it to play nice with mapper. As mentioned above, I finally got Mapper to accept the POIs by simplifying the <gpx> opening tag to "<gpx>" and removing the "<metadata>" tag and data altogether. It's just the category issue at this point . . .

Re: Importing List of POIs into Maemo Mapper 2.3
 

abby_normal

Yes Poi_Loader uses the sym tag and creates a new category in poi.db if it doesn't already exist. If you connect your table with USB (and poi.db is on an accessible card )then you can update poi.db in situ.

Rick

Re: Importing List of POIs into Maemo Mapper 2.3
 

Drat. Rick, I really appreciate your taking the time to answer my questions. Unfortunately, poi loader (I am using 1.0_pre) can load the attached GPX file just fine--as can the built-in "import" function--but the categories are not getting picked up. The category simply defaults to what I select at import time as the default (e.g. "Other").

I will note one other difference--Mapper's built-in import function preserves the line breaks in the <desc> tag, while poi loader converts them to spaces. That may be a "feature" not a bug, I just wanted to point it out.

NB, you probably need to change the extension of the attached back to ".gpx". I could not upload it with that extension, so I changed it to ".txt".

Perhaps preserving the categories isn't going to be possible . . . but if you happen to have an example GPX file you'd be willing to share that does pick up the categories upon loading, I'd love to see it. Maybe by eyeballing the way it is constructed I can figure out how to tweak the format of the attached (or rather, the script that generates the attached). Thanks again.

Re: Importing List of POIs into Maemo Mapper 2.3
 

try-
http://gnuite.com:8080/cgi-bin/poi.cgi

it'll generate valid pois in *gpx format all day...

hope this helps,
kernelpanic

Re: Importing List of POIs into Maemo Mapper 2.3
 

Very handy! But I have not found a way to make it generate a category tag . . . is there something particuar I need to do?

Re: Importing List of POIs into Maemo Mapper 2.3
 

abby_normal

To cut a long story short Poi_Loader has its own format ".mmp" which is just a gpx with the category in <sym></sym> tags included. Unfortunately just changing the filename from .gpx to .mmp wont work because as you have seen Poi_Loader removes newlines from a gpx before outputting its .mmp file. Its not expecting mmp files with linefeeds and the program falls over :(

The only way this is going to work ( without me releasing another version and I haven't got the time at the moment ) is for you to find a way of removing the newline in the <desk>, renaming the file to .mmp and then Poi_Loaders should import the categories ( I'm ) slightly concerned with the "/" too but suck it and see.

Rick

Re: Importing List of POIs into Maemo Mapper 2.3
 

Success!! :D

Rick, thank you for putting me on the right track. It took some further tweaking, but I've finally got it working as desired. The steps I had to follow are listed below, for anyone who cares.

I definitely understand that you don't have all the time in the world to devote to supporting poi_loader (and what you've done so far is great!). But if you ever start working on a new version, you might want to look in to some of the behavior described below. For now, as long as I've got a way to get from A to B, I'm happy!

So, here goes:

(1) Prepare GPX file. The categories are in <sym></sym> tags. I actually found that poi_loader can handle slashes in the category names just fine. The program also can handle line feeds in the <desc></desc> tags, but it will convert them to space characters on loading. What I did is convert my line feeds to a delimiter (an asterisk surrounded by spaces) just because I prefer the way it looks. It might be possible to preserve line feeds in a future version of poi_loader (because the Mapper import function supports that), if you care to get that working.

Note: neither poi_loader nor the Mapper import function seems to like <metadata> tags. I recommend deleting those tags from any GPX file to be loaded into Mapper. In general, it's probably best to delete any tags that do not contain the specific data you wish to load.

(2) Change extension of GPX file to ".mmp" - let's call it "poi.mmp".

(3) Create additional "dummy" .mmp file that has one POI entry for each new category to be added in the database. You'll see why this is necessary below. Let's call this one "dummy.mmp". (If you're only loading a few POIs, you can probably skip creating the dummy file and just use "poi.mmp" for all of the following steps).

(4) Run poi_loader, create blank new database (so my existing POIs don't get munged if something evil happens). Let's call it "poi.db".

(5) Import dummy.mmp file into database. This always crashes with an error first time around: "Run-time error '9': subscript out of range". poi_loader terminates.

(6) Relaunch poi_loader. Open "poi.db". Import "dummy.mmp". This works, but for some reason most of the POIs are not assigned to the right categories. The new categories are entered into the category table (which you can see by browsing the database), but for some reason most of the new POIs have not been assigned the right category id.

(7) Quit poi_loader and restart (this does appear to be necessary). Open "poi.db". Import "dummy.mmp" again. THIS time, the rows are loaded with the correct categories. One or 2 rows (the ones that were initially loaded with the correct categories) are recognized as duplicates, but the majority are added as new POIs (presumably because the program recognizes that the category IDs do not match).

(8) Quit poi_loader and restart (last time!). Open "poi.db". Import "poi.mmp", which has the "real" rows (in my case, 35,000+ of them). Voila! Aside from a few rows failing (actually a few dozen), everything is properly imported! This took maybe 20mins on a Core 2 Duo e6600. File sizes - "poi.mmp" was 6.1mb, "poi.db" was 8.9mb.

(9) Copy "poi.db" to Nokia SD card, tell Maemo Mapper to use it as the POI database. Everything shows up, with descriptions and categories properly handled. However, Mapper is not particularly "fast" with this many POIs. I don't know if the POI database is indexed--if not that might be a nice enhancement in the future. Other programs & hardware can deal with even larger POI databases, so a speed enhancement should be possible, "in theory."

One could possibly skip step (7), since the categories are presumably loaded into the category table correctly by this point.

So, this whole process took way more time and effort than expected (and getting the data into GPX format was even more complicated, but that's another story). But, it's nice to have it working, and encouraging to know that it CAN be done.

Perhaps in the future there will be software that can handle POI conversions much more elegantly than the existing stuff (and I've tried about a dozen programs at this point). That's not a knock on these programs--a lot of them are quite useful for certain things--but I found no single program that can convert among all POI formats reliably and elegantly. Moreover, even though GPX is supposed to be a universal standard, programs vary widely in their ability to handle the same GPX file (which is supposedly well-formed according to the schema). Some fail if certain tags (e.g. <metadata>) aren't the way they expect. Some ignore tags (like <desc>, <sym>) that you'd expect them to recognize. Some handle the data in these tags more completely than others (e.g. some can handle linefeeds, some can't).

Anyway, thanks again to Rick and the others who offered suggestions. I hope all this is of some use to someone else!

Re: Importing List of POIs into Maemo Mapper 2.3
 

For what it's worth, Maemo Mapper does not support reading categories from GPX files with an Import POI operation. In the Import POI dialog, you can only specify a singular category that becomes the "default" category that is displayed in the subsequent POI List. You can change the categories in the POI List, but only after they have been initially assigned the "default" category of your choice.

The best approach, unfortunately, without resorting to external software, would be to divide your POI into separate GPX files, one for each category, and import them into Maemo Mapper one by one, specifying the appropriate category with each import. It's not the most convenient work flow in the world, but it's the best Maemo Mapper can do at the moment.

Re: Importing List of POIs into Maemo Mapper 2.3
 

In this process did you try GPSBabel?

Re: Importing List of POIs into Maemo Mapper 2.3
 

Yes, and I was really hoping that it would do the trick. But it had the following problems:

* I had a choice of 3 input formats to begin with: .img (Garmin specific), .mp (used exclusively by cGPSmapper), and .csv. GPSBabel could not read the first two no matter what I tried. It could read the CSV file, but (as noted in the first post in this thread) that file did not contain "category" fields that I could use in Maemo Mapper. Also, whatever I tried, GPSBabel would not intelligently populate the <desc> or <cmt> tags from the CSV file. Going this route, I got a usable but incomplete set of POIs - name, lat, and lon only, no category, no description.

* Even when I finally massaged the .mp data into .csv format (this took a number of steps), GPSBabel would not populate the <desc> or <sym> tags the way I wanted. I finally had to use a custom script in a programming language called Ruby to format my GPX output properly.

Disclaimer: I know virtually nothing about Ruby, but someone from another forum kindly provided a script that I was able to tweak to my satisfaction.

Re: Importing List of POIs into Maemo Mapper 2.3
 

Thanks, I realize that now. Fortunately poi_loader provides an alternative that is a bit less labor-intensive. My POI file had 68 categories; doing them in 1 batch is nice.

Still--I said it before--I love the program! If I wanted to suggest enhanced POI importing functionality in a future version, would Bugzilla be the place to do it? I haven't used it yet, but I assume it's relatively straightforward.

Re: Importing List of POIs into Maemo Mapper 2.3
 

Why were your choices limited to those three? GPSBabel will accept almost any file format, including .xml, .gpx, or what have you, and convert to almost any other. Thus the name.

Re: Importing List of POIs into Maemo Mapper 2.3
 

One problem you might have is if you ever want to provide icons for your categories is with naming. The way Mapper works is to name the Icon (a png or jpg with a .jpg extension) the identical name as the category. Having a "/" in the category name makes this impossible.

Rick

Re: Importing List of POIs into Maemo Mapper 2.3
 

Sorry, should have been clearer. The fine folks who complied all of this data and are kindly sharing it free of charge only have it easily to hand in these 3 formats. They are set up primarily to support Garmin units, so everything is geared to generating Garmin .img files. (This is a full set of autorouting maps of New Zealand, by the way, not just POIs. I have seen their stuff working on a Garmin Nuvi unit, it works great, and is a wonderful alternative to paying $300 for Garmin's NZ maps!).

It looks like there are at least a couple of Maemo programs in the works that might have direct support for Garmin files, and that would be pretty cool too. But Maemo Mapper works now, and the ability to swap among different map repositories gives it a "wow" factor that vector-based apps do not currently support.

I'm not knocking GPSBabel. The concept of a "swiss army knife" program for POIs is sorely needed, and it supports a LOT of formats. But, alas, it could not do what I required!

Re: Importing List of POIs into Maemo Mapper 2.3
 

The reason Poi_Loader didn't work for what you needed it for is that it was designed with a completely different purpose in mind.

It normally takes one of gpx,tomtom,garmin,kml or geocache files (none of which support categories) and after asking for a category uses gpsbabel to convert them to gpx and then the app converts to mmp files ( wich is just a gpx with the category in sym tags ) for future use. You can also, by the way, dump the entire poi.db to an mmp for reloading in the future.

It was just luck that it semi-matched the above requirements.

There is no xml support in VB so all the xml translation is parsed within the app ( Thats where I removed linefeeds and also why the app wouldn't expect a linefeed in an mmp ).

I've now seen the light and I am programming in Python by preference :)

PS I'm not being defensive here just explaining.

Re: Importing List of POIs into Maemo Mapper 2.3
 

Ah, thank you. Category icons was my next project, so this advice comes at a great time! Fortunately, swapping out the "/" characters should not be too difficult.

And at the risk of being flamed for not having done a thorough search of the forums yet on this issue . . . is there a suggested or recommended pixel dimension for POI icons?

Re: Importing List of POIs into Maemo Mapper 2.3
 

The LAST thing I intended was to be critical of your app! To the contrary, now that I know Maemo Mapper has no support whatsoever for importing categories, I think it's great that it has this functionality at all! My comments are only intended to (a) let people who might be facing similar situations know what works and what doesn't work, and (b) humbly suggest functionality that someone might want to program into future versions of apps. Since that won't be me anytime real soon, I have no standing to criticize.

Basically, my concern going into this whole project was that I was asking a stupid question--that there was a simple way to get from A to B and I was just "missing" it. Having now dipped my toe into the pool of POI formats and applications, I realize it's kind of a jungle (to mix a metaphor).

Re: Importing List of POIs into Maemo Mapper 2.3
 

I use 48x48 but that was after trial and error there seems to be a lot of flexibility

Rick

Re: Importing List of POIs into Maemo Mapper 2.3
 

Doesn't Maemo mapper ask you what category you want to import a certain GPX file into? So if you put your POI's into separate files for separate categories, you should be able to import them with categories :D

Re: Importing List of POIs into Maemo Mapper 2.3
 

Yep, that is essentially what gnuite suggested, and it would work just fine for a small number of categories. But where you have a lot of categories (e.g. 60), that would involve:

(1) dividing your POI dataset into 60 separate files
(2) manually creating 60 new categories in Maemo Mapper
(3) separately importing each of 60 POI files (and making sure you don't screw up by accidentally specifying the wrong default category each time)

This kind of task is what automation was made for!!! That said, it is hardly a major issue, esp. since there is a workaround. But it would be nice to see it handled one day.

Re: Importing List of POIs into Maemo Mapper 2.3
 

Hi, all. Only just found this thread tonight, after having spent the better part of the day bashing my head against the wall trying to work out why Maemo Mapper wouldn't parse any GPX file I fed it, regardless of the source. So many thanks for the notion of trimming the <gpx> tag down to the bare minimum and eighty-sixing the <metadata>; that's sorted that out, at least.

Now, about customising the POI category icons in MM... where should I place these completed 48x48 pixel JPGs so MM will see them and use them instead of the standard coloured squares when marking POIs? I've tried /.images and /.documents/Maps/ (the default storage location of the POI.db file), but MM isn't interested in any such tomfoolery. Is there a special setting I have to use to enable this functionality?

Many thanks in advance!

Marcus