Custom POI Standards

There are lots of posts on setting up alerts even some in Faq.Not sure about any standard to ID them but if you pull up poiloader and go to help and view alarms and tourguides.Lots of information there to help you ID a file you may want to download.

Mine are all geared around travel and the reality of reaction time to get over and nav to one.

Any POI becomes instantly more useful if a column provides address and phone number info. If these are not easily available, then so be it. Remember that not all Long and Lat coords are accurate.

Wow, this is fun an exciting to be there to witness and get involved in the history of development of GPS POI files. Being new to POI file creation I don't have much to say in detail, but I'll but in my two cents on creating standards.

1) Most everyone loves standards, even those who don't follow them. This has been the case since the creation of PC standards back in the early 1980s like .ARC and .ZIP, and with standards before the digital age.

2) Any standard you come up with is better than none.

3) Consider current hardware, but don't design the standard around current hardware. Design the standard around what you would like future hardware designed around.

4) There should be a way to indicate the POI file does meet the standard.

5) Plan for programs to manipulate the files down the road. Try and help programmers out, plan for their needs, ask for input.

6) The layout should start out more fixed, and then become more flexible, so data should follow suit. col 3 could be firm and necessary, while col 4 could be one of three suggested formats.

7) Fields should be arranged top down, like State - City, not City - State

8) Look forward, set the standard, don't follow the old ones which were never designed with POI files in mind. Zip codes are increasing in use and importance because they can tell you state and city. It's too bad they are useless to the human eye.

9) The only relevant part of the street address is the street name. That is the way we talk. Which McDonalds? The one on Main street. We don't care about the house number, we have lat and long to handle the last level of location.

10) Will your standard work on an international level?

11) Keep in simple.

This is very sound reasoning and should be kept in mind when setting the standard. Basic and looking forward.
Chuck

Great ideas. Perhaps after the holidays there will hopefully be some progress on communicating a common standard.

Good points all, but the address format of state followed by city would be a little difficult to implement if you are stripping data from a web site. As an example, here is a listing from one I'm working on right now:

308 GRAND AVE
WEST DES MOINES, IA 50265-3714
515-255-0373

What I have for input is 4 or 5 lines consisting of the firm name, location field, street address, city/state/zip and on the last line an optional phone and/or fax. The CSV layout for this would be:
Long Lat POI Name (added data such as city state) and finally street addy, city/state/zip and phone.

Manipulating the data received from a web site and reformatting it to move elements would be beyond many of the people responsible for the POI.

The only way you would be able to easily meet your suggested format would be for the thousands of web masters to adopt a standard of putting the state before the name - and then what do you do about a zip code? While it isn't needed for the POI, it does make geocoding a lot more accurate.

I understand that CSV is the easiest way for anyone to edit a file. But what’s wrong with using GPX? My standard is GPX as I can add so much more info. Just look at some of my small work. Once you work with GPX you’ll get hooked on it.

NY Water Taxi
NYC DMV
Strauss Discount Auto

Good point.

How about offering that POI Verifier up for anyone who's interested in creating some correctly formatted and accurate POIs? Seriously, if those that create files regularly used it, the file repository would be far more reliable for everyone.

Just my $0.02 worth...

I have some concerns about a proposed POI standard, based on my script-writing experience. My script (which requires BBEdit) extracts POI - including addresses and phone numbers - from a Garmin GPX file. It seems that some devices don't display this extra information, but,

¿Is there any harm including it in the standard anyway?

That's kind of what format I'm using, to create custom POI, except that I do not use a spreadsheet or 'columns' to edit my POI. I prefer working in a text editor. One of my typical POI records, which works with the Garmin POI loader:

-101.750302,37.980191,Loves - Syracuse,"Avenue A & Johnson
US 54/US 400
Syracuse, KS
620-384-7796
w/ no parking, no diesel"

Notice that in addition to address info, I included a comment line. Gravy.

Absolutely perfect example of why this discussion started in the first place.

If the above ends up in a .GPX file for Garmin users, that is fine but....

1)GPX is un-useable for owners of many "other" brands
AND
2)If it ends up being a .CSV, that huge 4th field gives some of us "others" fits.

SO.......if the owners want this to become the "Garmin POI Factory" and have everything in .GPX so us "others" will have to do a (minimum) two step conversion process, that is their right.

Somehow I think that is NOT the intent.

I believe the intent was to come up with a standard for a .CSV file that would be able to maximumize the useablity for ALL brands.

Then if you also want to post a version specific to a model line, I believe that should be allowed also.......but wouldn't really be necessary unless it has some unique information that is not in the .CSV.

Just my $.02

I still believe that just because I bought the 'bottom of the line" GPS that won't accept 4 fields, standard linebreaks, etc. shouldn't require everyone who got the "top of the line" to be forced to downgrade my standards.

Leave the POIs at Garmin's standards, then have software available to adapt these POIs to other units. (i.e. nuvi linebreaks, combine fields 4 to 3, pull the phone number for dialing from field 4 and add it to field 3, convert to other formats, etc). You can easily remove what you can't use; but you can't easily add what isn't there!

RT

Exactly true. All the available information should be there......it just should be there in a format that is easily used by the biggest number of people.

I don't THINK anyone is proposing that information be left out. It just should be presented in as "generic" a format as possible. That means NOT using the non-standard line breaks that Garmin is fond of (in field 4??) for one thing.

Some wanted 'no field 4' because their GPS won't recognize 4 fields.

If you're referring to <br>, I agree. If you're referring to the linebreaks in the sample above, those are 'standard linebreaks' shown in Garmin's POI Loader Help file and should be available. It's a very easy and fast procedure to remove them if not needed.

I have a hard time understanding why people expect everyone to adapt to their standards. When we discovered our nuvi won't accept standard linebreaks, we didn't say "hey, come on everybody, we need to change all these linebreaks to <br>; instead we looked for a way to adapt those standard linebreaks to what our GPS will accept.

RT

I've seen this thread growing since I joined here, but it doesn't seem much closer to resolving anything. Let me add my thoughts:

Yes, retiredtechnician, I agree. I'm a nuvi owner and don't fully know what limitations mt nuvi has on poi files, but I certainly agree that someone amassing this information should collect all of the standard information that most users would want and not reduce things down to a lowest common denominator.

I don't however believe that the low end users should be required to use conversion software. That would just lead to lots of confusion and result in many potentially good members of this community having problems and wandering off in confusion or disgust before they got up to speed. More on that later.

What is this Garmin standard of which you speak?

In the custom POIs here I've seen a complete lack of standards. At least one file just has lat, long, and store name, and the store name is the same for every entry, no locality information, no other info. In the Kmart list you can't tell where there is a SuperK (with grocery store), they are all just marked Kmart.

We need some standards on how someone putting together data should organize it. Obviously it shouldn't be too demanding, if someone can't get telephone numbers for a POI set they shouldn't be excluded from building the set, but things like phone numbers should be there and in a well defined place when available. POI name should not be just duplicated down the list, at the very least a locality name should be there, and a state abbreviation would be a big help, as it gives someone a way to quickly and automatically search through a large list and extract just the entries for the states they are interested in (not everyone needs or wants all of the Starbucks in Washington state in their GPs, particularly if the live in the east).

Once you have clean standardized data, it should be straight forward to write programs that repackage it to fit the needs of the various gps makes and models. Or, to put it another way, if this can't be done simply from a single standardized set of data, then the standard data likely isn't well designed. It is very important to do the design step right and get the master data organized cleanly, or users will have problems with the end result csv files that should never happen. It might even be that the original master POI information should not be a csv file at all; it might be better to build it as a database or even just an Excel spreadsheet and extract just the data fields that are needed for any particular end csv format.

I'm willing to offer my help here, but I can't dictate to the group that things must be done my way, and for this to work Miss POI or someone else needs to set up some standards. If the master files are CSV files then I'm confident that simple AWK scripts could reformat them as needed for and gps make and model. I would be willing to create those scripts, but I don't believe that end users should be required to download the scripts, install AWK, and run them. It seems far better to produce all of the formats needed and make links available for each on the site, labeled as to what gps they are for. Thus you could have "Rest Stops for nuvi". "Rest Stops for {whatever) 4 field Garmins", Rest Stops for Tom Tom" and so on. I'm thinking that all of the formats would be created as the POI is added or updated, but the forum maintainers might even choose to just generate the various formats on demand from the master data.

The more that I think about it, the more it seems like a master database is a better way to store this master information than in CSV files. Just one of the nice things about using a database is that it enforces categories on the data. You know the phone number, if there is one, will always be in the phone number field, and will not be combined with other data haphazardly. You should always have city and state fields (even if city isn't always filled in). You can cleanly build whatever csv file you want, with 3 or 4 fields, with any designated line break, from one master data set.

Again I'll offer my help here. I've done a lot of database design work professionally. Mostly this was with older products (dbase, Clipper and such) and I'm not up to speed complete with SQL but I'm not afraid of it. I don't think I'm the right person to take the lead if this is decided to be a SQL database project, but I would be glad to work with a small group of people to work out a design.

One other thing that a database approach could give is that, if the forum overlords wanted to do this, users could customize their csv files to their own liking at time of download. Rather than have a half dozen or so files in different formats for each POI, a user could go to a dialog that asked the type of gps, if they wanted to include telephone numbers, even give them the option to download the entire POI set or just extract for the states that they were interested in, then build a customized POI file and download it. Of course, this would require some changes to the POI download pages section as well, but it is a possibility if a good POI database were created.

So I guess that I propose that a database structure be built that covers what data is likely to be kept on POI files. In the next day or so I'll post back a suggested master database format. I'll very likely not get it right the first try, and I hope others give feedback on things I've missed or better ways of doing things. If we can get a good database design then we can use that to guide information collection and from there we should be able to make POIs that both are standardized, rich enough for gps unit owners that accept lots of data, clean enough for limited gps units, and descriptive enough that all POIs in a set don't have the same label.

Of course, some of you may already know why this is a bad idea, if you do please put me in my place.

Except for this. (And the related conclusions that follow.)

.CSV is already pretty universally recognized as a "generic" standard format amongst all of the various manufacturers (I think)......so tinkering with that would be a mistake, IMHO.

.CSV is a crude database (if rules are enforced!)and can easily be manipulated with Excel (or equivalent) or any word processor.

AWK?? Isn't that the sound a chicken makes when you grab it??

We started out here trying to get a standard set of instructions (suggestions) as to how to go about "changing a tire". We don't need to build a new car in the process!

I think that is the right direction.

I believe Miss POI had tried to establish a database but were unsuccessful. While a database gives us utmost flexibility, most users would not be providing data as such, i.e. CSV files that they can use. The difficulty is in extracting the data from these files into a database.

I agree with retiredtechnician that we should follow the Garmin Standard, although that leaves a lot of room for inconsistant data. The best we could do is to tighten the standards slightly, such as requiring that field 3 always be unique and suggesting ways to do this i.e. name and location.

Frovingslosh' idea is brilliant and efficient for an internal, tightly controlled business standard, but not so good when every on and their brother are creating POI files. The standard has to be easy to understand and easy to follow if it's going to be widely used.

OK, first of all let me say that I have a very rough draft of what I'm thinking about written up. I'm not ready to post it yet, I want to give it some more thought and be comfortable about what I present. I very much expect that people will hack it apart and want to make changes to it, in fact I will be disappointed if they do not. But the more I look at this the more convinced I am that this is the proper way to go.

The more that I look at the existing POI files, the more convinced I am that there is absolutely no standard at all. At best many ways of doing things that all work in some cases, and apparently fall short in others. And saying that cvs is like a crude database is like saying writing whatever you want on different size sheets of paper and putting them all in a box is like a crude database, particularly the way we use csv files.

I've seen files with nothing more than coordinates in them and the same store name duplicated on each line. I've seen files with State names fully spelled out and files with State names abbreviated to their two letter postal code. It would sure be nice to extract all POIs such as Burger Kings for a particular State such as Pennsylvania by "grepping" for a particular string, but you can't do that when some POI files have the State name spelled out and some have it abreviated and some lack it altogether, and in some cases these differences exist even in the same POI file. And to make it even worse, sometimes the two letter code for the state is all caps, and sometimes it is mixed case. Unfortunately, a mixed case search for Pa rather than PA can get a lot of false positives.

Even if one searches for PA, Pennsylvania or Pa, you still can't hope for good results, because many of the POI CSV files spam multiple lines. You're not going to be able to search a file of this type and get back anything useful, since you only get back segments of the information.

This is just one example of many things people might hope to do with well organized POI data, but are hard pressed to do with the data we have as it is presently organized.

If Garmin does have standards the nice thing that you can say about them is that there are so many, everyone can find a different one that they like and do their own thing. But that is exactly the problenm that this thread is talking about.

I believe that we can create an easy to use well documented standard that can be used by the average contributor. In fact, I believe that it can be simpler to use than the currently created POI CSV files, particularly if those files contain any extra information like a phone number or proximity alert.

I'll be glad to hear what's wrong with my approach and try to address it, but it would be nice if you waited until I made the proposal to point out what was wrong with it. If need be we could even create some custom tools to help data entry, but I think it will be cleaner and easier to just accept data in text fle or spread sheet files, but with the format cleanly spelled out so that their is a consistant standard to future POI contributions and that existing files can be brought up to.

I will acknowledge that the existing file are so amazingly haphazard that it will be very hard to just import existing csv files into a database.

Take the Dunkin Donuts USA Canada file as just one example of a file badly in need of standards. here is what a quick look should pick out (these may be far from the only problems):

Sometime there are phone numbers that are prefiles with ">+", sometime there are phone numbers that lack the ">+" and, of course, mny times there are no phone numbers.

The name isn't even consistent, sometimes it is DunkinDonuts and other times it is Dunkin Donuts, still others it Dunkin' Donuts and I expect there are other variations as well.

Usually te state is abreviated as two capital letters but in some cases it is mixed case. There may be fully spelled out state names too, although a quick scan did see any.

Sometimes the street address, city and state are in the POI twice for the same entry, other times they only show up once. Why the is no consistancy in the same file evades my understanding.

The city and state is usually there but often both are completely omitted. Zip codes come and go as well, it will be exttremely hard to pocess these files and automatically determine what is a zip code if one exists.

The top line seems to be a complete fictional POI with an illogical set of coordinates, apparently someone's creative idea of a way to label the file.

And this is all in just one POI file. For this site to do the things that it is capable of with POIs it absolutely needs standards for people to build POI files towards. I think that it also needs a good database design, but I do recognize that that is open to discussion. But the need for consistent standards to at least guide people in creating CSVs is hard to argue against. At the current time it does seem to me that we can't point at any of the inconsistencies in the sample POI here and say that something should have been done a different way, because as far as I know the current guideline is just to do your own thing and get some data in there.

Well, I'm glad to hear that she wanted that and very much hope that she joins in in this conversation. There is absolutely an issue in getting the current hodgepodge of data into a consistent database. I can't downplay that, it would take a lot of work. But that is no reason to keep doing things in a haphazard way, old CSV files can stay around unit someone is willing to convert them, which likely will involve some manual work. new files can be spared from that problem if standards are created. I can give the group several great ways to avoid that problem with any new contributions. Rather than the apparent Garmin "standard" of "just do something that kinda works for you and your particular GPS", we could give users guidelines of how to submit the data in an organized way (most of the data will be optional and no problem at all if it is omitted) and some simple ways to do it (spread sheets, simple text files, or, if desired a simple custom program, although I personally think it would be better to avoid providing a data entry program to contributors).

I personally believe that the current reason that the data is now so haphazard is simply because there is no target for the contributors to work towards, so they all take somewhat different approaches to try to get good files to contribute. But they are trying to contribute to the community, not just making a file for their own use. If reasonable and unimposing standards can be established I expect that contributors will accept them and be glad they have them.

Boy, one thing that I can say with certainty is that there is not a universally recognized standard. I'm seeing everything here at POI-factory from one line CSV file to multile line CSV files to other exotic formats, issues about how many "quoted" fields there are and where line brakes can and should be to questions about if fields need to be quoted at all. And getting past the layout of the file to what is the content of the fields, the standard seems to be "whatever you want".

Yes, the industry has often used comma separated fields, sometimes other separators than commas, or sometimes fixed space fields to help define data boundaries, But the closest thing that I can find to a standard in csv files is that they tend to use comma separators. That not enough for good database design and it is the reason this discussion was started, long before I got involved.

If there are rules I would be very glad if someone would share them with me. I can tell you from looking at many of the POI files that are posted here I don't believe that anyone is aware of any rules other than where to put the coordinates and to fill in a variable number of extra fields with whatever one thinks might apply.

Yes, that is exactly what it is.

It also happens to be a great tool for manipulating text files (or other files if you are geek enough) with pretty powerful scripting and pattern matching, but that's a different story.

Maybe if in the process of trying to do one thing the process is found to be so haphazard that a completely different goal should be considered, that isn't necessarily a bad idea. But what I'm really doing here is thinking about what needs to be captured, what is very desirable, what is extremely optional, and how to best organize it and how to standardize it. This is pretty worthwhile, particularly when not every GPS has the same capabilities.

As I looked at what was already posted here I realized that if, rather than just giving a lot of suggestions, a "template" for the data was built that a user could just fill in, then the CVS file could be built automatically from the data in that template. Most of the template would be completely optional (coordinates would not), but when the data was provided the contributor would have those instructions or "suggestion" as to how to enter the information so it was consistent with the way other contributors were submitting their data.

That template for the data can certainly be something that the user edits by a spreadsheet or even a text editor, in fact I think it should be. But because all of the fields are well defined (even the ones that are left blank), the contributor will know that they are working towards a standard rather than just putting whatever they want in any spot, and that their contributed data can easily be used to generate any of the apparently many different CVS "standard" files, or files for Tom Tom or other GPS brands.

I'm sure that there are a lot of strange rules for the various Garmin CSV files out there that I need to get up to speed on. But I'm also confident that these formats are defined and once we have the rules organized of what each family of GPS device needs we can make some great files that are good for everyone, they could provide all available data to top end users, not limiting them because some low end units have more restrictive file formats, yet also use the same data to produce great files for the low end units that meet their needs and capabilities. Why would we not want to do that if we can easily and cleanly do it?

I'll try to post a rough draft of what I have in mind tomorrow, that should at least give me a little time to be confident that I have not left any major holes. It will not be ready to adopt as a "standard" and the help of others here will be critical to come up with a good final design.

By the way, I should mention that, like this post, it will be verbose. Is there are limit on post sizes on this forum? If there is can some kind forum overlord please contact me and offer another way to get this proposal hosted, and then I could just link to that through this thread.

You can send ideas to me at maryann@poifactory.com

Miss POI

The following is quoted from Garmin's POI Loader Help file:

Creating .CSV Files

POI Loader accepts .csv files that contain longitude, latitude, speed alert information, and optional comments. You can create .csv files using a text editor, MS Excel, or a similar program.

POI Loader assumes a .csv file utilizes the following format for each POI (brackets [ ] denote optional text):

,,["][@]["],["] [comment]["]

Longitude and latitude must appear in WGS84 decimal degrees format (ddd.ddddd; negative numbers indicate West and South).

If you include quotes around the name or comment, you can include line breaks in the text.

The following are examples of Custom POIs in the proper format:
-94.81549,38.80390,Bonita
-94.79731,38.81099,Ridgeview@25
-94.74240,38.81952,Heritage Park,Perfect site for a picnic
-94.76416,38.81227,Garmin,"1200 E. 151st Street
Olathe,KS 66062
913/397.8300"

RT

fyi, I have reformatted a few files on POI for my use and often need to break old files into it's elements and then rejoin the data as needed.

I usually end up with:

long, lat, name, store #, address, suite/floor, city, state (2 letter code), zip (10 digit), phone, fax, comment.

My perferred tools are Excel with ASAP addon and notepad.

once the data is standard any combination can be created.

most of my files finish with:

long, lat, name and/or store # and/or city and/or state, address with line break(LB) and/or city + comma + state + zip + LB and/or phone and/or fax and/or comment.

you can see the format on the Walgreens poi.

Scripts can be created to format the final product if the data was in database/field format

The more that I look at the existing POI files, the more convinced I am that there is absolutely no standard at all. At best many ways of doing things that all work in some cases, and apparently fall short in others.

I think that there is something that you may have missed. This is a site where many users of different companies share files. If a model of gps only uses three lines of a csv, maybe that is why they didn't add a fourth.

I have only been here for a year, but Garmin users are not the only ones submitting pois. Don't be too critical of members that didn't submit everything your unit can use.

I agree that 3rd column needs to be unique and that fourth should have address and phone if possible. But we need to keep in mind that this format is so that all users, not just Garmin, can enjoy the files.

Daniel

Yes, no standard, and no good examples to follow.

Good point, well taken.

I'm not convinced that is necessary. Existing csv files can be edited to meet the standard or to be imported into the standard. As along as the files import OK.

Well stated, That is how we got here, that is how people think. They followed what ever examples they found.

Going forward. We know few will read the standards documentation. Most will just look at how others have implemented way points in line with the agreed standards and follow that example. The only thing we can really change in your comment above is to provide a target for contributors to work towards.

I noticed that. I only read the first sentence of each paragraph to see if it catches my attention enough to read the next sentence.

I've sent you what I have. Will wait until you have time to look it over before posting it here.

Thanks, We knew that the holiday season would be a busy time so we have not opened this project yet. We will be in a better position to start tackling some of these issues starting next week.

We appreciate all the input that we have received and will be working to build something that will take us into the future and keep us as the number one place to find POI data on the internet.

Miss POI

Hi All,

I've been reading most of these posts, and I hear almost everyone saying the same thing:

"We would all like a format of some kind to make finding POIs easier and also for those creating POIs to have a baseline to make good files."

So to that end, I think everyone needs to take a step back, breathe, and realize that we all want the same thing, but we're not all saying it in the same way.

I'm new to GPSs and POIs, but I'm an old hand at designing and programming systems, been doing it for almost 25 years

If we step back and look at this from outside of "what works with/for my GPS" and look at it from the standpoint of creating good standardized data, then we could use something along the following lines:

Create a tiered "Standard/Guideline/Whatever you want to call it"
- 1st level - The Minimum "required" info (from the sounds of it this would be: lat, long and a unique name, maybe even country and state/prov. for sorting etc.)

- 2nd level - address data, phone numbers, hours, etc.

- 3rd level - comments and addition (forum-based) information, such as who worked on it, verified it, etc.

Then people can go to whatever level they want, with the mandatory 1st level a must and create "good" POIs.

Once you have that, you can then use this data to create CVS/GPX/etc files built along device requirements (Garmins/Mios/TomToms, etc.) Think of them as "custom reports".

This way, even with only minimum data you get good "Standard" POI files, that meet the minimum needs of the simplest devices, but can also provide the data needed for the more advanced ones as well.

It's usually best to think of things in terms of what data do I have and what format do I need to output it in.

That way if you need to reference the data in some new way, we only need to create a new "custom report" to output the data that we've already verified.

Does this make sense to everyone? I purposefully left out specifics of what data is in each tier as this would be up to the designers of the database/system.

If you need any assistance with this let me know, I've got plenty of skills, maybe some of them can help out.

Oh, a youngster!

I think you and I are in agreement here. I don't use the 3 tier approach, I'm simply thinking that some fields are required (coordinates for example) some are highly desirable (State for example), some would be nice to have but optional (City for example) and some are very optional (phone number, hours of operation, and so on). But once you set this out, and how to format the fields (lets try to be consistent an not sometimes enter NC, sometimes use North Carolina, sometimes use N Carolina and even N. Carolina, and in particular don't use all of these forms in the same POI file!), then how do you compile this information? I don't know of any good way to organize it in a 4 field csv file. But if it is in some sort of database then we can generate csv files for different devices, gpx files for those who need them for their bluetooth or otherwise want them, special files for Tom Tom or other brands, and even custom csv files for people who want information included that most people don't need cluttering up their screen (such as internal corporate store numbers). And it can be done by what you term "creating reports", which is exactly what these files are, Custom format reports extracted from the master data set. I think we agree completely here.

Another advantage of this approach is that when updating data you only need to update it in one place, then run new reports to get the updated csv files, gpx file. Tom Tom format file and so on. Otherwise you end up editing multiple files, and keeping fewer files than some might like because updating a half dozen or more forms of the same data is "too much work".

What you say seems like exactly what I'm hoping to do. I'm pretty new to POIs as well, so I've created a rough draft of what I think needs to be there, but hope to get the community here to comment on it, and I certainly expect that some things will be suggested that I missed. I have sent that rough draft to Miss POI at her request, and will wait until she has time to look it over and respond before posting it here.

I do seem to have angered some people when I talk about making a database. I not yet clear on why. If you can understand the issue, please advise me.

Got background in SQL? My database work has mainly not been in SQL, but I'm expecting that it would be the right tool for this. I'm not afraid to get myself up to speed on it, but if we get a consensus I would be glad to share the tasks.

Or is SQL even needed? Maybe we could do all of this in a spreadsheet. That seems to me like the hard way, but it might make some people more comfortable, and contributor comfort and acceptance are extremely important, much more so that the platform of implementation.

I'm also thinking of how to collect the data, that is, what format contributors would be asked to submit it in. Unfortunately, csv file with 4 fields and 2 of them already used for coordinates are not a great choice for organizing data (thus the current problem). I doubt that we want to build a "data entry" program for people and tell them that they must use that. I'm thinking, at least to get started that data could be requested/accepted in any one of at least 3 forms:

1) A spreadsheet, with each field in a column. Blank sheets with the column labels would be provided. A few columns would be required, most would be optional. There are several ways to pass along to the contributors any details about the columns. Contributors could use Excel or Open Office.

2) A text file with 1 line per POI. Columns would be fixed width. The downside of this approach is t that there is enough optional data that the lines could be very long and scroll beyond the end of the page. Not my favorite way to approach this but I see no reason to not accept contributions this way. Text files with the column headers would be provided, with detailed information on each column. There could optionally be data seperators (such a perhaps | ) between each column to help users be sure the columns don't shift in large POI sets that go over 1 page (or the headers may be repeated every X number of lines, easily removed when the data is first imported into database).

3) A text file with 1 field per line. Any optional field could be omitted, so entries would only need to be as long as the number of fields that the contributor actually used (plus an ENDofPOI line), this could be as few as 3 lines based on some POI files that I've seen (lat,long, name), although I wold hope for at least an additional State field. If a field dies not change from one poi to the next, it can be omitted from subsequent POIs and it will be assumed to be the same as the previous POI entry (thus if doing a Starbucks in Washington POI you would only need to have a Store name and a State field line for the first POI, it would propagate through to all of the POIs in the file A basic template with all fields would be provided, the contributor would delete any non-required fields that they don't want, copy the remaining block, and start putting data after each data label. This strikes me as an extremely easy way to create a set of data (even easier than current csv files), and would be extremely simple to process with a trivial AWK script into a 1 line per record format that could be easily entered into a spreadsheet or a database.

Amy feedback?

The one source and multiple output idea is good. Updating multiple sources would be no fun and discourage updates.

Multiple input sources would be nice to accommodate what contributors are comfortable with.

Be sure to include an easy workflow for those of us who automate things, usually with large POI files. That is, if there are hundreds or thousands of POI entries in a single file, don't require some piece of information or format that requires each entry to be hand massaged. Currently, my output comes from GeePeeEx editor so a csv/spreadsheet format would work as long as fields/columns could be arranged to meet the new input format using common formulas in all the rows (you know, create one formula and copy into the other rows).

Tim

Well maybe not all. Remember that we can't see your face and tone of type. We often don't realize how others may read a different meaning in what we type.

From my standpoint there have been suggestions in the past on creating files. People that haven't done it before do not realized how much work it is to maintain a file system. The biggest problem is updating the changes later on. You are correct that we don't want to have multiple files, but a db that you make a change once in would be great.

I for one do not have $250 to spend on access or filemaker just to make files for this site. I enjoy being a part of the community and helping others. Miss POI and Jon do a great job for all of us. It is going to be their call as to where we go and how it happens. Life needs to be easier for them too. Imagine 140,000 kids to look after...

Daniel

'Anger' is not the best word to describe the disagreement you received from your request for feedback. I think I understand the issue, or reason for that strong disagreement. We use databases all the time. They work great. They organize our work and streamline our processes. There are a few concerns that we need to take into consideration for this proposal. First, a database may require us to purchase a license. Second, only a select few understand the database design. Third, the input field structure can be inflexible. At least this has been my experience.

If you can alleviate those three issues you'll be on your way to attaining more general agreement.

The needed info is obvious, phone no & hours change but, can be nice to have. When I create my POI's for alerts, I put in necessary info trying to keep col 3 short & to the point as this is what will be displayed on my screen. I want to keep the alert screen display as large as possible so I can read it at a glance with it not becoming a distraction to my driving, especially at faster speeds or heavy traffic. ex: Rest Area: I-35.74 E/W. As my nuvi 760 under the POI listing show closest to my location I don't need any this else in col 3 but for my actual file, col 4 will show closest city and state. By the way I keep all POI's in separate files.

If I want to list a store location, then col 4 would list more info like address, city, state and possibly phone no. ex: in the pizza file (if I had one) I would place in col 3; Domino's Mpls, col 4 would be the address, city, and state along with phone no.

I try to always remember, KEEP IT SIMPLE, STUPID!

I’m sure that this discussion would and could go on forever, but I create the POI’s for me and if I share them, you can then chose to keep them as they are or change them to your liking. At least you will have the coordinates and basic info to work with.

I'm not looking for anyone to spend a nickel on this.

I think contributors could submit in a number of different formats including text files and spreadsheet files. If the contributor chooses to use a spreadsheet then I think that a spreadsheet compatible with the free (and very capable) Open Office is a must.

Almost everyone has a text editor, and we can point people to some free and very capable text editors if they want to move up to them from simpler things that come with their OS. But even the simple ones will work fine, as long as they can handle the file size.

There could be many different ways to contribute data, whichever way the contributor chooses it would be submitted and flow into one common "database" format. Exactly what that database is, I'm not sure of yet. I think an SQL database would be the right choice. But it might even just be standardized spreadsheet files. Others may have better ideas, and I'm certainly not in charge of making this call. If it is decided that an SQL database is the best system to use, I would hope that a free open source version like mySQL was used. This might even allow contributors to submit databases directly, although I would not expect that many would want to do that. Whatever way that the data was stored at POI-factory really should not affect the average user or contributor.

It would not be my intention to get contributors to use databases directly. I was thinking that they could maintain their data in the original file form submitted: spreadsheets, text files, or perhaps something not even considered yet. Perhaps this needs to be given more thought? An alternative is that a few simple "reports" could export the data in spreadsheet or one of a few different text file formats. That way, if someone else took over maintaining a set of POIs, they could maintain it in whatever format they were comfortable with.

I would hope that any tools developed to flow spreadsheet data or text file data into final database form, as well as the "reporting tools" that create the final csv, gpx, ov2 (or whatever it is that Tom Tom uses) and other formats be available to users who wish to work with them; but a certain skill level should be expected from those users, no one at POI-factory should be teaching people to be database programmers. For the most part the factory would just accept the raw material in text or spreadsheet form, and crank out the finished product POI files in whatever format the end user wanted (and perhaps even give them choices about what fields are included in those files, such as, "do you want store numbers included?").

If he and some others are willing to work the server side scripts then inputing the "standard" would be that much easier. Almost all POI's can be converted into some spread sheet program.
So if we make the columns standard in that then the database side will be that much easier.
(for example only)
Col 1/2 lat/long
Col 3 Unique Name
Col 4 Address
col 5 City
Col 6 State
Col 7 phone
and on and on. Then the server side script can pull out the info you want for where you want it and format how you need it.
I think the big resistance comes from the current maintainers. To have to "redo" some of theses POI Files would be an enormous chore; but like anytime a new "standard" is put in place there will be some growing pains. And those big POI Files usally get written with the help of many so why not break them up and get them "updated" by the masses again?

If we make the columns standard, then we have our "standard" and don't need to spend time/effort/money on a "database"........that would not be compatible with what anybody else in the whole world is doing. K.I.S.S.

Extra cols. at the end can easily be stripped with Excel (or similar) or a wordprocessor or GeePxxx (sp). Some loaders will just silently ignore the extra columns and no manipulation is required.

WE DO NOT NEED TO REINVENT THE WHEEL, just clean it up a bit.

What you wrote about the fields is about what I have written, although I managed to do it in a few hundred times more words.

It will be tough to redo some of the existing POIs, absolutely no doubt about that. The current form of the information is haphazard and very loosely structured, an automated process will not easily transfer it into a new format.

But just because we have disorganized POI files now is no reason to continue to create POI data in hard to maintain and organize forms. And there is nothing that forces anyone to transfer all of the existing POI to a new form. If there is a Whole Foods POI now in the current csv format, it could stay in that format if no one wanted to convert it. I wouldn't even suggest that anyone do convert old files until we have six months or so experience with new POIs and a new system. I don't see any reason why existing POI can't exist as they are, but new POIs could have a common standardized data form and their pages could offer to generate any output file type with any fields that the end user wants. If people see a need to convert the old POIs to the new format, that can be done on a file by file basis over whatever time frame it takes.

K.I.S.S. is what I am all about

When I thought about the tool to enter things, I was actually envisioning something along the lines of a web-based tool:
- that could take in a few types of files and
- also have a screen-based entry system, for editing and for those that just wanted to add one or two POIs, for the group to enjoy.

It should also have something that validates the data that is provided, and allows the person to correct/add the needed data.

It should also have some sort of internal sorting/category listing feature, so that POIs can be extracted in various ways (like by category, name, state, city, lat/long, whatever, that can be decided later)

At this stage, what we really need is to template what data is needed to create the most complex GPS device data, that way we know what the max is, as the minimum is always easy to go down to from there.

How it gets built is up the abilities of this site's hosting package, those of us that can help to create it, etc...

The other way is to stay low-tech and keep to the current files, asking that people follow a set of guidelines when submitting their POIs.

This way is the simplest, but leaves us pretty much where we are right now, with data that is hard to easily sort and extract for personal use.

The database method (which non-techies don't need to worry about, as it would be transparent to you) should be done in stages, that way we can start small, to make sure that we're on the right track for our uses.

Also, for what outside groups want/need, the output/"custom reports" from the database could be used to feed their needs.

I'm up to help do whatever the group decides.

Hopefully this helps.

Well, I had not considered web based data entry. I wouldn't want to be told to use a web interface to compose my data. But yea, I an see where a web input option would be handy for inputtonf a small number of locations. It might be nice to have such an interface, but it is something that I would not aim for at the onset.

The web interface that could accept a few type of files is more of what I was thinking. And to be honest, that "interface" in my mind was simply an email attachment. But we've seen plenty of websites that can upload or download files, so sure, there could easily be a "Submit your POI file" page if someone would take the time to build it. Makes a lot of sense.

Good point. Web based submission could certainly do some basic validation (check if the points lay roughly within North America, for example and warn the user and ask if the still want to make the contribution if they don't, warn if state fields are empty, or if unrecognized postal codes are used and so on). Submissions would still go through a maintainer for a final OK but the immidate feedback would be nice.

Well, now you're getting to the output / reports / csv ....
Yes, in it's simplest form the people at POI-factory might just have a handful of reports that generate the files similar to what is available now. But I could easily see that, when downloading a POI file for a set of POI that are stored under the new system that you could get either a form or a serise of step by step questions:
You have chosen to download the North American Pothoels POI. Please select your GPS make and model: Grmin nuvi 250
Do you wish to include the entire North American POI set? N
Do you wish to select by states? N
Do you wish to select by a distance from your home location (I show it on record as (-78.7400,35.6123)? Y
Within how many miles would you like to include from this location? 250
We have 3671 POIs that meet your selection.
Would you like to include street names where available (2975 of the POIs include street names)? Y
Would you like to include City names (3104 of the POIs include the City name)? Y
Would you like to include the county name (8 of the POIs include the county name)? N
Would you like to include the state name (all POIs include the state name)? Y
There are no phone numbers available for these POIs.
Would you like to include the extra comments when available (852 POIs have extra comments)? Y

Your file is now ready, click here to download.

Yes, that is where we seem to be. Of course, if we find a field that has been overlooked we can certainly go back in and include it, as long as we do a good job on the design now and don't paint ourselves into any corners. But to get this rolling it seems pretty important to get a well defined set of fields specified, and that data layout would seem to be based on both the data in current POIs and the current capabilities of the GPSr hardware. Imagine that 10 years in the future most GPSrs are web enabled. We might add to the database design a website field for each POI, if someone smarter than me doesn't think to add it at this chance. There should be no problems with that as long as we do it right, and people haven't been storing lot of website addresses in the comment field because we failed to anticipate the need.

Lets hope we can get a consensus on this.

That would be nice but there are only TWO votes that really count.

I mention that (obvious) fact now only in the hope that when the management does make a decision that NOBODY will see fit to disagree and argue about it.

Yes, we need to keep our priorities straight.

1) Retain the motivation to build and maintain POI files

2) Retain interest in downloading and using POI files.

Do I have the order correct?

You are some what correct that we would not need a database. I guess it could be done in an spreadsheet and uploaded as that then each person could pull the data they wanted and convert it to whatever file type they liked. As it is easy to join columns too
But my thinking was if we made the input the same across the board people could use the raw data how they see fit or the server side software cold very easily(less work on the programmers) pull the columns you asked for and generate the output for you. Standardizing the input is the wheel that were trying to fix

For what it's worth, I took a couple of evenings and made and submitted a POI file based on this concept. There is no consensus on a database design yet, but I went ahead and created some data labels, grabbed some addresses from my state government's website, cleaned it up and labeled each filed, geocoded it by street address and zip at itouchmap.com/latlong and then converted the resulting text file that defines all of the fields with a very simple awk script. The first .csv file was very simple and contained no line breaks, but then a quick modification of the awk script produced a .csv file with more information, formatted in 4 lines with < br > type breaks.

The process went very cleanly, and I now have the data (in a simple text file form) that is very specifically labeled and that I can use to generate .csv files, .gpx files or other format files, all while maintaining one set of data and not several different format copies.

Here is a sample of how I formatted one POI of data:
CITY: Raleigh
STREET: 3070 Wake Forest Rd.
ZIP: 27609
LAT&LON: 35.824222,-78.621868
PHONE: (919) 872-2815
HOURS: Mon-Fri 8:00-5:00
COMMENT: Holly Park Shopping Center
END_of_POI

The information is all here, I saw no real extra effort over putting it into a .csv file, but now I can manipulate it into other formats like .gpx also.

Note the coordinates line. At first I was labeling lat and long separately on two lines. But the site I was geocoding from gave it in one string, and in reverse order from what Garmin uses. Not liking to do work, I quickly realized that rather than put the effort into splitting the line into two fields, or ever swapping around the two coordinates to suit the Garmin format, I could do it faster and with less chance of error if I just copied the data that the website gave me into the text data file and let the awk script parse the string to generate the individual lat and long fields and then put them together again in Garmin order. The extra code to parse the original lat,long data into the two internal fields was pretty simple, it took only 1 line of awk code.

And, of course, I can use the awk script that I wrote to quickly parse any POI data file that conforms to this format and produce a POI .csv file with it.

I certainly hope that we can get a consensus on how to break out data and a form to store it in. Even this simple text file form for would seem to offer some nice benefits over just throwing most of the data into .csv field 4 in whatever order it happens to land.

I followed this thread for some time. It seems like a lot of discussion going on about column 3 and column 4 and what should go where. And also it seems to end up with something that has to work for the low-end devices. If so then probably the majority of users will still be unhappy.

I am using GeePeeEx and I like its way of using a header record for the csv files. NO, NO, NO I am not suggesting that everybody has to buy GeePeeEx (sorry Phil AKA Hornbyp, maker of GeePeeEx).

The header record describes which fields are included in the rest of the records.

Sample file for El Torito restaurants:

Lon,Lat,Name,Addr1,City,State
-117.888731,33.817958,El Torito-Anaheim-148,2020 East Ball Road,Anaheim,CA
-119.054456,35.365496,El Torito-Bakersfield-149,4646 California Avenue,Bakersfield,CA

The two entries are each just one line, even though it displays here as two.

The minimum info for a store or restaurant should be those in the sample. If the creator wants to add more fields, he/she can do that, but the field names in the header record must be the defined names, e.g. phone for phone number, description for more info about the poi.

Sample file for Historical Markers in Texas:

Lon,Lat,Name,City,State,Description
-97.603450,30.330300,TX04002,Decker,TX,Decker Swedish Evangelical Free Church & Cem
-97.546066,30.520950,TX08054,Hutto,TX,Hutto Cemetery

For historical markers you cannot always provide a street address, but a city and state should always be provided.

From a file structured in this way a user can extract whichever fields he/she needs for their device. Fields can also be combined to form a column 3 or column 4. Maybe the extracting could be part of downloading the file.

This could be a simple way of defining a standard for the POI files. Users can work on the files using text editors or spreadsheet programs. For files that already meet the minimum requirements you would only need to add the header record.

NOTE. If this format is any way proprietary to GeePeeEx and Hornbyp, I am hoping that somehow that could be resolved.

Regards
JCA

If I am understanding this thread as a whole this is a good example of what we are trying to avoid.
Examples are the same on block 1-3 but the trouble starts after that. Where it is understanding that there is no address for some markers, do we still leave it as "block" 4 and have it blank and move on to City, State etc..?
jcavaa I am not calling you out you just happened to post what I think is an example of what were doing wrong instead of me looking into some of my files. I think we are trying to come up with ONE standard way of entering data vice if it has an address do it this way if not then this way and if something else than this way.

GPX is not propriarty to Phil or GeePeeX, not even to Garmin. It is simply a form of XML.

GPX format would be a workable way to define all of the fields. The good thing about GPX format is that all data is labeled, not just thrown together like it often is in field 4 of .CSV files. The down side of using gpx as a "standard" is that, unless you expect everyone to use a paid tool like Phil's, gpx files are hard to create, maintain, and extract data from.

You are completely right, City and State information should almost always be available and should be ratained and included in any POI. There is little sense to just have a list of coordinates and a store name with no other information.

But there is a lot of other information that I see in some POI files but not others. While this information should pretty much always be considered optional, it sure would be nice to keep and it is good to retain it it and do it in a standard way. This includes things like a pgone number, even a fax number, a street address, hours of operation, store number, and so on. One of the things that I think we should do here is build a good list of the data fields that could be in POI files and standard ways to format them.

When I say standard ways, look at the State field for an example. While most POIs seem to use the two letter postal abbreviation for the state, some spell out the same name and some use other forms of a state name. For example you might find "Miss" or "W Virgina" or "W. Virgina" or rather than PA for Pennsylvania you might see "Pa." or "Pa". This makes it hard to extract just one state from a national POI file, since you can't always be sure how the state will be represented, and it might not even be represented the same way in all places in the same POI file.

Rather than use hard to maintain GPX files as a standard, I would much rather see us use either spreadsheets, a database or even simple text files. I already have an AWK script that can translate a text file with well labeled fields into a CSV file. (It's a very simple script less than 20 lines).
I'm looking at the gpx format and will write another script that uses the same text file to produce GPX format files. I'm expecting that my text file will eventually be imported into a spreadsheet or a database (I'll modify my AWK script to make a comma delimited 1 line per POI file to make this easy) and then the POI files can be generated from that, but for now the text file approach is serving me well, The important concepts are simply that the data be organized into standardized consistently labeled fields, that data like city and state be not discarded when available and be kept in s consistent form, and that we get some consensus on what the fields of data should be. Otherwise any organized system is n better than just saying that people should throw whatever they want into field 4 of a CSV file.

I am not suggesting to use GPX files as the standard. I am suggesting using csv files, which are text files, that have a header record that describes the fields of the next records in the file. This makes it into a free format, but that can accommodate all the fields we would need. Some fields would be required and some would be optional.

The sample file with historical markers was just to show that in some cases the required fields e.g. street address would not be applicable. But I would for sure want to have it required for stores and restaurants and wherever else it can be provided. I don't think that means that I'm talking about having two standards. If the one standard required the POIs to have a street address, then a file for historical markers would not be able to follow that standard.

I totally agree with you with regards to state name. That must be part of the standard to have the two letter state abbreviations.

JCA