Goecoding - how to deal with poor input data?

This is why POI files are a constant work in progress. The POI creator will rarely have all necessary information to include every location.

As I see it you really have two choices:

(1) Discard the locations with missing info. The file can be updated at a later date if another member provides the missing information.

(2) Include the locations but use general coordinates, such as the center of the town/city. I do not like this choice because a person will just drive to this spot without realizing it is not the actual address.

When you have poor input data, you could post the info here and ask someone in that area to get you a good address and coord. Most times this helps but is another time consumer. I hate the companys that have their location on the web site but you can only get them by city or zip instead of by state. Good Luck

Well, a 3rd choice is to do a in 2 above, but to at least include a comment that the data is not precise (something I did on a previous POI file in a few cases where I only had a street name). This depends on users actually noticing the comment though, perhaps I may do something similar here where I have much poorer data but include the warning right in field 3 of the CSV file, not as part of field 4.

Neither dropping the location completely or routing the user to a bad point without warning seem good ideas to me.

What I have done when they only provide a PO Box address is to try contact the company for additional details (better address). Failing this I removed those entries from the list.

It may not be perfect but it is already enough work to code those that have "valid" addresses.

Dom

Also, does the website has mini-maps for each location, or does it simply list the street name and/or PO Box? If the website has a mini-map, then you look up the intersection on an outside mapping program to obtain the coordinates.

I have MapSource and MS Streets and Trips installed on my computer, so sometimes I will look to see if the POI is already loaded. I will then copy the coordinates onto a new file to create a custom POI file. Also, I will visit the company's website and verify that the location still exists. (A lot of times the pre-loaded POIs will be out-of-date.)

-Dave

I know as one download and using a file I don't want coordinates in there that are wrong. I would just as soon the be missing.

I think you should move them to a separate POI file and work on them one at a time. You should be able to look them up on the internet. I do that all the time with local restaurants, search for 'company name city' and look for a link with the address.

I don't expect every file to be 100% up to date. Currently, the data in my GPS is years behind in roads and new business.

I just looked up Panera Bread on my 265WT which has the 2009 map. It still does not show the location I've been getting coffee at since about 2006. It is in the 'Panera Bread Locations Wisconsin' file.

I have a MagicJack, so I could make a dozen or so calls to help. I'll PM my email address if you want to send me a list of phone numbers.

Thanks, I'll take you up on that. Have phone numbers for most, although a few with dubious addresses don't even include them. It may be a couple of days before I have things cleaned up enough to be obvious whet the trouble points are (I may as well geocode everything that I can so if more questionable ones show up I only bother you once), but I'll send you the list when it is ready.

What have others done in such cases? I'm tempted to drop completely the location that only gave a P.O. Box, but I realize that this might be a disservice to someone trying to use the POI file who was routed to a much farther location when there was a closer one near but I omitted it due to lack of location information.

I don't know what type of establishment you are trying to geo-code but in my case, I'm currently geo-coding motels, hotels, B & B's, etc. Many do not have good addresses. I look the specific location up in Google. Each establishment almost always has their own web site and usually with a good address and a map to boot. I don't know if this would work in your case as it may depend on what type establishment you are geo-coding.

If the phone number is provided, sometimes a reverse lookup will yield a usable address.

These are a small "chain" of independently operated grocery stores (although this seems to be a more general problem than just this on set of POIs). A few have websites, most don't. The address data is already from the national website, so not much luck there. Some locations are actually already in the Garmin POIs, but like most Garmin POIs they are very incomplete, and useless to look up unless you are in proximity, so that's not a good solution for a large group. Will recheck Google earth but don't believe they are there. Thanks for the feedback though.

Great Idea. I didn't think of it, but that very well might get me a better address.

Just wanted to say that this tip is proving quite helpful. I've had several cases where geocoding fails or gives obviously bad coordinates that are not the desired location, about half the time I can get good results with a reverse look up and a bit of extra work. About 35% through my geocoding.

Glad to hear that some of my gray matter is still working. I'll have to go tell my wife, who wonders, sometimes.

It can be painfully slow to research addresses this way, but if its all you got, its all you got! I was working on a POI file with some of the same problems and just thought to myself, "I wonder if reverse lookup...?"

For unknowns, I have dropped them but listed them with the POI posting and requested help. Someone graciously helped me out with one Holiday Station store recently.

Tim

Frovingslosh, thanks for the post - I'm in the process of creating my very first POI file and find myself in the same situation.

I geocoded my address list with GPSVisualizer and it failed to match on about 11% of the addresses (48 of 448). Their site had some tips on how to deal with that and by following them I was able to geocode about 10 of the 48 that failed.

From the replies to this post I've come to a geeral conclusion - post what I've got, then continue working on the file and repost when I have something better.

I missed out on RetiredTechnician's POIVerifier (http://www.poi-factory.com/node/20659) so I tested the file by loading it to my GPS. Looks good, so I'll submit it shortly!

You may want to download Google Earth 5 to add a additional tool for finding coordinates.You can zoom right in on the area and the pointer will show the coordinates.You can go to tools,options and find show lat/long decimal degrees and put check mark in it.This will change how coordinates are shown.

You can also google the business to find out what the street address is (ie - XYZ Company, Anytown, Ga).

Taking advantage of the fact that this site has something like 70,000 members who want to help? A posting here that you need help getting coordinates for the specific spots would probably find someone nearby who would be more than willing to give you the coordinates of the curb entrance to the parking lot.

I have Google Earth 5, but have been using an interface to the on-line Google Maps that returns lat & long in an easily usable form.

There indeed are benefits of looking up points manually this way. Sometimes you find that a looked up address comes back as a point on a country road with no buildings withing miles. This can be taken as an indication that you have the wrong coordinates. I've also found a number of times when I look up some street and then zoom into the street level view that I've been positioned on a completely different street. I was surprised at how often this happens.

At this point I've done a lot of work on a 500+ location POI file, but I have almost 20% locations that are still bad. I'm not willing to release the file that way, with either with the bad locations just dropped or with the incorrect coordinates. Reverse telephone lookup and trying to determine if the results are any better that the data that I already have is very laborus, but it seems like the next step. As one can see from when this thread was started the project is stalling because of the poor quality of the published addresses.

And, unfortunately, while Google Earth, Google Maps, or other resources can sometimes tell you when you have bad data, for most POIs (including this set), it is much harder to determine from the computer view where the actual location is, store roofs just don't convey enough data and Google's street view us incomplete and would take too long to locate an incorrectly coded location. So while Google is a good resource for finding geocoding errors, it is less useful for finding addresses that don't geocode properly.

I can not agree with you on that.Just yesterday using my gps to a location that turned out to be wrong.I went to Google Earth and went right to the entrance.

Using Google maps my first choice to make gpx or csv files.I have found to work very well.Putting in just the name ,city and state will get you to picking from a list.Comparing the phone number from the data you have gotten from the web site will lead you to the correct one.Then zoom in to verify the location.

Could you provide more details? I'd be interested in getting lat&long from Google Maps.

I've been compiling POI data in a labeled text file format. Basically I put labels on all of the fields, then let a script compile the data into CSV or gpx files. If we could agree on a standard for a database then I would expect similar scrpts could create csv and gpx and more directly from the database, but there is no aggrement to standardizing data and how to store it yet, so this is my precursor to that, it keeps the data well defined rather than just throwing everything into "field 4".

Here is a sample of how I organized the data for a POI that I contributed:

CITY: Mount Olive
STREET: 1130- E North Breazeale Avenue
ZIP: 28365
LAT&LON: 35.211936,-78.062267
PHONE: (919) 658-9921
HOURS: Mon-Fri 9:00-5:00
COMMENT:
END_of_POI

I use http://www.itouchmap.com/latlong too look up the lat & long. They actually use Google Maps. At first & had separate Lat and Long lines, but it too much needless work to grab the data from the website and break it into two fields, so I realized I was doing it the hard way and should just modify my script to accept the one line as Google formats it and break it into the two fields. I used the scripting language AWK which is pretty handy, it only took one line of code to make this conversion:

/LAT&LON:/ { c=index($2,","); lat=substr($2,1,(c-1)); lon=substr($2,c+1,length($2));}

Your email is listed for contact, so I'll contact you that way. It's not actually posted (a good feature to prevent spam), but if you want the full script please respond and once I have an e-mail address that I can send attachments to I'll forward to you a write-up I have, the AWK script (only about 20 lines and a few of those are comments or blank) and a sample input file.

Frovingslosh - despite your affinity for text files and scripts, there is zero chance that either will be the accepted norm.

The average user has no idea how to create or modify scripts. Scripts may seem simple to you, but they aren't simple to the average user.

I'm not attacking you or anything. Just trying to point out the facts. Because you seem fixated on your AWK scripts. They will simply never be accepted by the majority.

I certainly wouldn't research 500 pois but what I like to do in my POIs is to look at Google Earth and see how close it is to the lat/long that I got. That way I can change it to what I visually see.

I'm not sure what your problem with me is. I was asked a quetion. I responded to it. I'm not trying to force anything on anyone (although certainly would like to see some standards adopted for POI files, I find it cumbersome to see POIs that don't even include State information in them, or that are for a single chain but show the chain name in several different forms and spellings).

I've never suggested that average users should write scripts, I have a script already written, and would be glad to modify it to meet any needs that it does not cover. However, I might be over estimating the average user here, or you might be under estimating it. I've contributed 2 POIs so far, with bad address data stalling my third project. I had to change exactly one line in the script for the second POI contribution. Here is that line, do you really believe that the average user can't understand how they might change it to vary what the program outputs, say for example to remove the hours information, as I did?

field4 = POIstreet ", "POIcity "<br>" POIhours "<br>" POIphone "<br>" POIcomment;

You say your post is not an attack, yet in the same paragraph you call me "fixated". I'm not clear why you would do this. And I'm hardly fixated on AWK, although I've known about it for over 15 years I only started using it for a few things last year. And I would really rather see the POI factory standardize on spreadsheets or a database and use spme other language to create POI CSV and GPX files from those. But if I was going to compile POI information I was not willing to just throw everything into field 4, I wanted to keep it organized. My approach to create a text file is just something that worked for me. I do think the "average" user here should be able to edit a text file, and I've offered my crude AWK script to the few who have asked for it.

Frovingsloth - I don't "have a problem with you". I never said I did. In fact I went so far as to specifically say I wasn't attacking you.

I'm sorry you took what I said the wrong way.

If you want to keep disussing AWK scripts then by all means do so.

I have no idea what to do with a script. Most of what you say I don't understand. This has nothing to do with you personally, but not all of us have the same back grounds.

I have a college degree in agronomy, and have self taught myself (with the help of others) everything I know about computers and electronics. I am not stupid, just ignorant.

Maybe if you were to point folks like myself to a page that explains how to use AWK or scrips we would better understand what you are trying to do.

If you can explain your better mousetrap, then maybe I can use it. Until then I will be making files with GeePeeEx and Excel. It may not be the fastest, but it is the most accurate and simple for me to understand and use.

Daniel

GeePeeEx and Excel are fine, but neither are free and I wouldn't want to tell people that they have to buy things to contribute POI files. There is a free alternative to Excel, OpenOffice, but I know of no free alternative to GeePeeEx. Still, keeping the data in a spreadsheet keeps it very well organized, and avoids the "everything thrown in randomly to field 4" tend that I have a problem with. But maybe the GeePeeEx form of a spreadsheet would be a great way for people to contribute data to POI Factory, and the factory could produce the desired GPXX and CSV files with their copy.

I've sent you an e-mail at your contact through POI factory. I can't include attachments in that, but if you respond to it I can send attachments directly and will be glad to send you more details on awk and my script as well as a file that I used to create a POI set.

I've already been criticized more than once here for being too verbose, so I'm not going to try to convert anyone by posting in painful detail how I'm doing this or to teach AWK. A great free copy of AWK called GAWK is available here: http://gnuwin32.sourceforge.net/packages/gawk.htm and there is plenty of documentation on the web including here: http://www.gnu.org/software/gawk/manual/gawk.html

The documentation can be a bit intimidating for the first time user. My advice would be to just hit me with an e-mail and ask for a copy of the script that I'm using. I didn't extensively comment it, as it was only for my own use, but would be glad to go back and comment every line if anyone wanted that. As it stands it's only about 20 lines, only about 16 of them are needed (could perhaps be trimmed to as few as 10-12), and I hope it is straight forward enough that the average user here could edit it if they wanted to make changes (would work just fine without any changes for most users). It's just a text file, so it can be edited with any editor that can produce a text file.

If you go to this site you can add position finder to your Google maps. http://maps.google.com/gadgets/directory?synd=mpl&pid=mpl&ba... .Once you put the check mark in position finder you can click on a location and it will show the coordinates.Also I will email you how I use Google maps to do custom pois.

Maybe I am taking this too personally, but it seems like you could say "I'm not attacking you or anything. You're an idiot. But the world needs idiots ..." and then later respond, "see, I started the paragraph with "I'm not attacking you....".

I'm not wishing that I had a soap box to preach the merits of any particular programming language from. But I am surprised that whenever discussing ways to better organize data so that the problems of "everything thrown into filed 4" might be lessened that someone always has to naysay with "you can please everyone so don't even try to improve things" posts. I would hope that people who are not interested in the topic could just go and participate in other threads. There are certainly a lot of threads at the POI factory that I consider uninteresting to me, I just don't read them. But I've tried to discuss this before, and the end result was Miss POI closing the entire thread.

I would be glad to show people what I'm doing, if they hit me with an e-mail I'll send them a response with a couple of attachments. I would be happy to see a thread where we discussed using this technique to better organize data and create CSV and GPX files from it. I'm pretty new to AWK myself and might learn a lot as well as spreading information to others. But I don't want to pick fights with those who naysay whenever this is brought up, and I don't want to offend Miss POI by doing an end-run to get past the thread that she has already shut down.

I was quite suprised by the accuracy for coordinates on Google Earth V5.

I am very dismayed with the street addresses on google maps. Punch in my home address and it shows me across the side street. Punch in a street address in downtown Williamsburg, VA and some are a block off. There are no numbers on 99% of the houses in Colonial Williamsburg as it is a "Restored area" and they did not number houses in
the 1700's. So if you only have a street address, you may never find who you are looking for, unless you go to the city's tax assement/property web page and locate the exact building there. Oh yeah, there may be a 200, 200A, 200B and 200C address on the same lot and they are all separate buildings.

Lots and lots of street address errors in Google maps. Don't rely on them for accuracy.

I can tell you one solution will not work while geocoding. I use Google earth, Google maps, MapSource, my gps, store locators for chains, several different geocoding sites and I still can't be completely sure I have the coordinates coorect. You can't even trust the stores own maps. (I think they probably come from a second party)I do the best I can with the tools and info I have and hope for the best. You will find the community at large will help with corrections and additions as needed. If a location just gives a PO Box, I would say they don't want visitors at that location. I think you are on the right path as far as putting in as much info as possible, but you have to remember not all gpsr's will be able to display all the info in column 3 and possibly column 4.

So, you are not sure you have the coordinates coorect, HUH?
Was that intentional?)

Happy snow day to some of us.

Just another of the many typos I make. Firefox catches most... (I doo make that typo quite frequently when typing coorect coordinates.but not when typing coordinates correct...)

Yes, I've certainly noticed the large number of POIs that you are maintaining. thank you very much for that.

Yes, although cynical, I had that thought too. But I would hate to offer for others use a POI file where I knew I omitted locations, just as I would not like to look up in a POI file my nearest location, only to find out later that there was a location 20 or 30 miles closer that I could have gone to if only I knew it was there (even if I had to make a call to find the correct location). So I'm just not willing to put my name on a POI file that I know is grossly inaccurate or grossly incomplete. My 500+ location POI file is just going to have to wait until I can resolve issues with nearly 100 locations.

The first step is to compile it as completely and accurately possible. As for displaying it, I'm creating a POI file with all of the relevant information in field 4. I say relevant because I decided to omit some data that I did capture, such as zip code, although I could easily modify the script that produces the CSV file to include the zip if anyone wanted it.

I'm creating field 4 with the <br> Garmin format, which seems the most widely used format. But the script could easily be changed to produce true line breaks, or any <br> can simply be replaced by a CR/LF in the file with most editors. I expect that these two forms of CSV files should work for most Garmin users. I don't know about other brands of GPS but would be glad to look at issues that might affect them. I did look at producing TomTom OV2 files, but since these are binary files and there are apparently converters that will translate csv to OV2 and I would have no way to test any OV2 that I produced, I saw no reason to pursue this, at least while I'm the only one using this approach.

I've been putting pretty limited information in column 3, wanting to avoid screen clutter if alerts were used. If anything I may have too little rather than too much there. Do you have any specific thoughts on what I might be including in a CSV file that would be too much information? I was a bit concerned if all nuuvi's could display 4 lines of text from field 4 but I posted a question on that issue and the responses came back suggesting that I was way under any limit that might be there, with people saying they had 12 or more lines in field 4.

A script could also easily produce a GPX file. I've been dragging my feet on this a bit and instead working on other things, but if there is any interest in this I'm sure it could be completed in a couple of hours at most.

Ok, I stand Keerected.

I often see the word "noon" typed for the word "none". That's explains why my backspace key is getting pretty damn worn.

I made a poi and the description in column 3 was truncated on my Nuvi 270 but not on the nuvi 770 (different screen widths) It isn't that there is too much info, you just can't see it. However if you click on it, the next display (column 4) does have and display the information.

So do you try to ID the location completely in column 3 if the information can not be displayed.. i.e. Super Wal-Mart #5154: Rosemead. or just put Super Wal-Mart? I put in the info even though it won't be displayed initially but is available if you tap the location.

The argument could just as easily be made to just enter the store name in column 3, when you click on it you get the rest of the information anyway... Which is the way I did the Petco poi...

Either way you do it someone will disagree... (As I have stated previously."You can't make everybody happy"

Well Bob Dylan said you can.. Either by arriving or leaving.... But that doesn't take into account the people who just don't care one way or the other... Is that a double negative?

Reply removed; reposted in new thread (http://www.poi-factory.com/node/20835).