Need to Standardize POI Files

To me, it comes down to the effort required to accomplish this (for future new POI files and to update current files) vs. the amount of benefit gained.

The effort
The effort required from our volunteer POI file creators would be substantial.

The benefit
I'll put forward a guess that at most, a fraction of 1% of the users of these POI files will go to (or even want to go to or contemplate) the effort of modifying files to reduce their size. And for folks using the unaltered file, wouldn't the addition of state/township info increase the file size thereby defeating your intended goal for most users? And those that do pare down the POI files may want to kick themselves as soon as they find that they have a wedding or funeral to attend in a state they've just excluded...

For files like RestAreasCombined, RedLight and Speed Cameras, which are files updated very frequently, the modifications need to be performed with each new update thereby lessening the desire for many to go to the effort to save space.

I've looked at my collection of csv and gpx POI files and see that they're sized around 100-1000KB in size--even my Schools file which contains 170,000 POI locations(!) just takes up about 7.5MB. Now look to your GPS files and see maps, junction views, combined voices, voice recognition files, etc. that are measured typically in GBs. I just don't see a great benefit in reducing the size of the tiny POI files in the presence of these enormous files.

Many current GPS devices now auto-update maps and such from internal GPS storage to SD cards, even further lessening the need to keep files small.

I've heard many think out loud that Garmin TourGuide files may exert an extra load on the device's processor but don't remember any suggestions that lots of Custom POI locations affect a device's operating speed. Are there references for your thought? If so, that's a good point in favor of what you suggest.

For many folks, simply deleting a few foreign language files will save much more space than paring down the csv and gpx files to reduce the eventual compiled gpi filesize for Custom POIs.

Your message is good food-for-thought and I certainly have no objection to POI creators going the extra mile to include this info.

Even though the files are posted on POI Factory web site they are created,maintained and updated by individual members.The exceptions are the Red Light and Speed cameras handled by POI Factory.Some but not all were created for personal use and then shared with the membership.Each member has there own preference on how they develop them.Sure some members would voluntarily accommodate a change but not all based on this same discussion in the past.To be honest no more space then a poi file takes up.Not sure the effort would be needed especially since SD cards are so cheap.

All the favorite files I use I convert to GPX, some are easier to convert than others, files maintained by Mahoney are the easiest, the usually take about 5 minutes to convert.

Because I can't load the rest area on my Honda Accord I am also going to pull out Texas. I looked at the file and found the same problem so I put it on the back burnner. My idea look at the N E S W borders and see where the coordinates fall. Then I will sort the file by the two coordinates and pull all that fall in the area that Texas is in. May work?? But as I said it is on the back burner for now.

You could convert them using EPE to GPX files and then using MapSource delete all the ones you do not want. You can also delete the ones you do not want in EPE. But in my opinion it is easier and faster with MapSource.

I have done some in the past where I totally deleted all in the west and only kept the east and midwest.

I have to agree with an earlier post. The person who creates a POI has in some cases an incredible amount of information to collate already.

Standardization of POI files is a tricky concept to get your thoughts around.

Should we standardize on .csv file for Garmin devices? or would .gpx files be allowed?

Should the .csv files conform to what POI Loader states"

Should every file contain the phone number of the location?

Should the coordinates for the location be the official address? or at the front door? or at a spot such that an "along the route" alert will sound?

If Cracker Barrel is our model, then should all "names" begin with the state in which the POI is located? Should the "address" be in the order "city" then "street address"?

standardization of the POI file format has been kicked around before but not for several years.

The primary issues are the maintainers build files for their use and then share their personal efforts with others. Then there is the issue of different devices requiring different formats. Tom-Tom uses a CSV file having a maximum of only three fields. Rand-McNally can use up to 9 while Garmin uses 3 with an optional 4th.

We all agree the first 2 fields have to be the coordinates. That's what it is in all CSV formats. The problem comes when you add the next field or fields. Just comparing Tom-Tom with Garmin as an example, Garmin says the 3rd field is to be the POI Name while TT allows additional characters to indicate special features such as dialable phone numbers which Garmin does not allow in a CSV.

While the vast majority of CSV files found on the site have the 4 fields allowed by Garmin, not all of them are written by Garmin users. So really, we are talking about what content should be in the additional fields and again, personal preference is the major factor.

The Cracker Barrel file was presented as the OP's example of an "ideal" file. Personally, I don't like the way the file is laid out. I mean, I know what state I'm driving in, what I would prefer is the file to list the locations by city within the state. Otherwise you have to guess based on distance from your current location.

Mahoney's CSV files are known for their consistency. The first 3 columns or fields always contain the coordinates with the 3rd column containing the city and state as part of the name field. My particular files I maintain always contain the city with the exception of the Exxon-Mobil files. These are as published by Exxon with some reformatting to use state/province abbreviations and Diesel spelled out rather than the [D} in the station name.

GPX files are another animal all together. As each field is defined,they can be put in any order desired. Jon has written a script which will read a GPX file and because each field is identified, output a CSV file with 9 fields a RM unit uses in the proper order.

Trying to define a common format for CSV files is wasted effort because no two people can agree on exactly what should be in the user editable fields. "You say Toe-mah-to, I say Toe-may-to." What works for me may nor work for you. That's why I personally reformat virtually every file I load into the style I like best.

Good idea,
1) find your center of the USA,
2) sort excel column A greater than or less than the center of USA coordinate that would be in column A,
3) sort excel column B greater than or less than the center of USA coordinate that would be in column B.

Copy and paste remaining POIs to new file.

Excellent comments. Maybe I will grab some of it and put it into Glossary points (with proper accreditation of course)

I remember when I first joined, I was interested in standardization myself (an occupational tattoo us Information Technology types often get branded with). Then I started working on a file that Miss POI as well as my wife were interested in (Jo-Ann Fabric and Craft Stores) and found that it was just not that easy to get data except by the copy and paste method from "Store Locator" pages on websites. I have written Jo-Ann asking for nothing more than a simple text file that had the same information that is displayed on their website pages but have never heard back.

The other side of the coin - as you well know - is the updating effort.

I understand why Trianj made his statement. He thinks

but I do not think the speed of the units would be impacted in the slightest. I think the Garmin built-in POIs are around 9 million now.

With SD cards, people can load every one of our poi-factory files without a problem. I, for one, would never want to limit the rest areas to those just around me. As CraigW pointed out, I might just need to get to a funeral with little notice. We know a family in Canada and the death of any of several dozen people would have us on the road tomorrow.

But, as you and others have pointed out, it is very unlikely we could enforce anything, anyway. If we did have "standards", I suspect we would get a lot less files to choose from.

What he said.

- Tom -

As I download files put up by fellow GPS users I personally go on Google Earth as a KML file and figure out what states I will not be visiting in the near future and then delete those coordinates from that file and then I have the areas I am interested in.
The real question here is that there are lazy people out there that want someone to give them files that they want for free because they are "lazy" and not willing to change them.
Just be happy someone gave you that file for free. What you do with it is up to you.

I do not know how to do this.

Would you please provide a short explanation? This sounds like something that would help a lot of us.

John

Directions please

Thanks, Mary

Have not done this for a while but to get down to the nitty grity you can just get the ones on your route.

Custom poi file along a route .

1.Using Excel or Notepad make a csv file to include your start, via points and destination point for trip.
2.Open in Extra Poi Editor and save as a Garmin GPX file.
3.Open the GPX file in Mapsource.
4.Select under edit New Route.
5.On left click Insert Waypoint.
6.Go down the waypoints and highlight the first one and click ok.
7.Repeat until all waypoints are now showing. Click ok at top.
8.You will now see your route.
9.Go to file and select save as and select a GPX file.
10.Open the GPX file in Extra Poi Loader http://turboccc.wikispaces.com/Extra_POI_Editor
11.Select Merge Open under file and load the custom poi files you want to include in your route.
12.Once all files have been loaded go to edit and select Find Poi on route.
13.You will get a pop up for how far to search off route. Insert the miles for search and click ok.
14.You can go to view and select View all Pois on Map.
15.Once view is completed select close map at bottom.
16.Go to file and select save as Garmin GPX.

All of my POI files are in the .gpx format. The way I figure it, if I fill in as many fields as I can, the end user can just discard what they don't wish to view on their units. It's always better to be looking AT it than to be looking FOR it...

Thanks

The speed cameras had the state and that made it easy to plus Texas out.
I was surprised to not see the states in rest area.

Extra Poi Editor works great mostly for the Updates.

When I do updates for my Offbeat locations, I dont worry at all about the City, State and Zip. If EPE can find the proper locations, it will fill them in.. Only problem with that is that it can only do about 1000 at a time, then need to wait for a hour or so and run it again. You do this until all are found.

So, if you load any file into the EPE, then run reverse Geocashing by using the Tools menu, or just hiting F4.

It does work like a charm for me at least!

As noted above, there are many ways to do this. When I have a GPX or CSV file that does not list states, I figure out what and where will I be using it.
I then go to Google Earth and and look up the coordinates (such as if I will not go east of the Mississippi and no further north than Interstate 80 then I will delete any coordinates outside that area.) I then go to the CSV file and first sort by column A and delete those coordinates not in that area and then sort by column B and repeat delete.
I do this with all my files for if I will not be going there then I do not need that info.
I then load the file in EPE and save it as KML and look at in in Google Earth to see if I made a mistake.

Shortly after joining this site over 5 years ago I also saw the problems being discussed here. When POIs did include the state, that state was inconsistently entered even in the same file. I saw one POI that had at least 5 different forms of representation for Pennsylvania, making it even harder to extract that particular state from a larger POI file.

Someone else started a thread about standards ( http://www.poi-factory.com/node/18592 ) and I joined it, along with many others. Some of use wanted to at least make suggestions for guidelines for consistency. Others wanted to call anyone making suggestions for standards "Nazis" for daring to suggest that the best system isn't everyone just doing something different.

What happened to that thread? Miss POI shut it down. The "reason" given at the time was that there were big changes in the works, the powers to be already knew what they were going to do and so people shouldn't talk about it any more. That never really made much sense to me, I saw no need to shut people up on a discussion of technical merits. Well, as far as I know the big changes to come never came. The final post silencing the thread was later edited and a lot of what was said was removed.

I completely agree that there should, at a minimum, be some suggested guidelines for things put in a POI file. Simple things like include the state and use the two letter capitalized postal code to do so (which would make it easy to extract particular states from a national CSV file even for POI that follow the "current standard" of "just throw thing randomly into column 4". But you are likely to excite the name callers in making any worthwhile suggestions and they will be quite happy to get the discussion shut down again.

ran into the state problem myself, I built a Excel macro to deal with it, the only problem it has is when the state is used elsewhere in the same line.

I find the inconsistency in a file to be more of a problem than standardization between file,

I was going to work on a POI file for my Accord and I thought it was the Rest Area File.
I went to the Rest Area POI and they are already in State format. I went to my files for my Accord and I have the Rest Area.

I don't remember what file I was going to pull that did not have the states in it. It was not the Rest Area as that was done by the maintainer by state.

Now to remember what I was going to work on?????

Just getting old and forgetful.

Did not think "View POIs on map" was working right now.

Does it work for you?

Had forgotten about not able to view on map.Like I said it has been a while. But you can still come to the same results.

I'm just not sure what the benefit would be for the majority of the users or if it would be worth the time and effort of the maintainers to change the POI's concerned.

I wouldn't insist that maintainers make any changes. But I think it would be good if people creating new POI files at least has some simple suggestion for conventions to follow. Simple little things like use the two capital letter postal code for the state. That would avoid the problem that I saw where Pennsylvania was spelled out as Pennsylvania and also listed as "PA", "Pa", "Pa." and "Penna.". That made extracting all entries for one state in a file much more complex than it needed to be. Other simple suggestion would be to use the same name and spelling for each POI in the file, I saw a doughnut poi file with the word doughnuts spelled at least three different ways. I hit a problem trying to do a search of my POIs for a certain major burger joint where it failed to find the ones near me that I traced to the stores being in the file, but the spelling of the name was different than it was for other stores (so it didn't match what I was searching for).

We have seen these problems over and over again, I don't think it merits being called Nazis for trying to organize these suggestions for new members wanting to be helpful and create new POI files, we are just trying to help them avoid common errors we have seen made many times before. I suspect that, if standards are created then some maintainers might choose to update their current files to conform to the standards. Why suggest that anyone is going to be forced to do extra work just because we would like to help new POI file creators?

Rest Areas Combined from MrKenFL is the only file I am aware of that is up to date and has the POI locations on the exit ramp rather than several hundred feet off the road. Rest area files for particular states may not be up to date.

I believe you intended to take Rest Areas Combined and sort Texas out of it.

dobs108

I will take the combined and sort out Texas.
Thanks, didn't realize I had an old file.

I can now see where having standardized state abbreviations may be useful if you are creating a state by state file. Forgive my ignorance.

Very interesting and thoughtful commentary by all. Thanks for input. I can readily see many of the problems cited within.
If we all agree on one thing however, it is that states should be identified uniformly using the Post Office two letter identifier for each> ie. PA for Pennsylvania
That would be a great start.

Very interesting and thoughtful commentary by all. Thanks for input. I can readily see many of the problems cited within.
If we all agree on one thing however, it is that states should be identified uniformly using the Post Office two letter identifier for each> ie. PA for Pennsylvania
That would be a great start.

Very interesting and thoughtful commentary by all. Thanks for input. I can readily see many of the problems cited within.
If we all agree on one thing however, it is that states should be identified uniformly using the Post Office two letter identifier for each> ie. PA for Pennsylvania
That would be a great start.

I work with the Offbeat Tourist Attractions as a combined and as per State POI file.

One reason that I use EPE (Extra POI Editor) is to keep everything the same mostly when STATE and zip code is involved. I will not publish anything that does not have a City or State. I use this information to help me find any duplications since I have over 16,000 entries and add more each day. I need to make sure that when I enter a new place, that it is not there already with a different name or the location is off, even just a tiny bit. Sometime the "find Dups" in EPE don't find them all.

To help me with this, I create a Excel file from EPE and do muti-searches like finding duplications of phone numbers and addresses (I keep things like north, south, east, and west as the same spellings as well as Road and AVE to name a few.)

I count on tools to help me keep things in orders, After using EPE to create the information, I use GPXClean, that will fix my errors in the addresses and phone numbers if there are any. Then in excel, I use a add on called ASAP Utils, which is a very good free tool to use for this type of work.

Also, with my entries, it is not always possible to put the location marking on, lets say a driveway or the door to a building, sometimes the poi may be in the middle of the park or does not contain a building, so I do my best. That is why EPE come in so handy!

When I enter a new place, I go to the map, enter the coordinates, POI name and address, then do a street view (when avail) and "look" for the item that I am finding the location for. If it is a building, and it is marked on the google map, I use that for ref.

Some poi are harder to do than others and I do try to keep a uniformity to it for easy usage for reading as well as re-organizing for myself as well as others that are using this file. Working with a POI file like mine can bring a lot of challenges and a lot of work but still fun for me to do..

Great explanation of technique.

Thanks for the file.

John

Could not live without it.. Thanks!!

I went to EPE and saved the Combined Rest Area as .gpx.

I then open BaseCamp and made a new file called Rest Area Texas. I then clicked on the Rest Area Combined file.

I used the select file in BaseCamp and put a box around Texas and pulled these files to the new file I made in BaseCamp.

There were a few places that came in from other states around Texas as the select tool selects in a square. I used the select tool and selected the ones I didn't want and deleted them.

I now have a Rest Area only for Texas and put this in my Honda Navigation. Worked great and easy to do....

Thanks for all the great suggestions from everyone.

I did not know you could do that!

dobs108

I put the POI file in BaseCamp and as I was clicking on them one at a time then dragging the Texas waypoing to the new file and noticed I still had several hundred to check I started thinking there must be another way......

That is how I found this...

As a retired Naval Weapons analyst, I find it hard to resist a thorny analytical problem.

It seems from the comments, that what people want most is to partition a hugh file by state or states.

Just for fun I loaded the combined rest area csv file into excell.

Next I found the coordinates for a state (i.e. Kentucky) using GOOGLE EARTH for a "box" just large enough to include all of Kentucky. (36.354, -89.502 by 39.177, -82.099)

Then I sorted the file using EXCELL by longitude (col A). It was easy to select all records between -82.099 and -89.502 longitude. I copy the selected records to the clipboard and pasted them to a "new" excell file.

The next step, then, is find the subset of records between 36.354 and 39.177 latitude (sort on COL B). By selected the second subset, I now have all the records belonging to Kentucky (with a few left over that fit into the box I selected but not the state of Kentucky). As they say, "close enough for government work"

-jgracey

Not so sure I understand the issue (multiple POI files) - but will have to research GPX files - I often do some manipulation of the CSV file and think both are very small files - even of you have many POI files loaded.

I just had to open a .gpx file and select the state I wanted and drag it to a file.
My mind is getting old and this seemed so much more simpler to me. Didn't have to open any other programs than I always do when I download a .csv. file from POI Factory. I guess I am lazy..

Jgracey, you must have missed my post early on.
This will also work for states like you explained.

Disclosure: In this post I changed the words 'sort' to 'filter'.