Introducing GPXClean for Windows!

 

Before I get long winded, I wish to thank BobKZ and Box Car (or is it 'Sox Car' wink ) for helping me beat up the very early beta versions (like last week!). It's a very small package that doesn't require an install. Just save somewhere where Windows (sorry Mac guys...) can find it and run. It's basically a filter program that will parse your .gpx files one line at a time, making the pre-described changes in each record. Cleans up case issues (all caps, lower, etc), removes extra spaces, expands out abbreviations, etc. Read on...

What GPXClean does (without asking):

  • What GPXClean will do: Compresses multiple spaces into a single space in the 'gpxx:StreetAddress' & 'gpxx:City' fields
  • Removes spaces from the beginning and end of anything in the 'gpxx:StreetAddress' & 'gpxx:City' fields
  • Log to a file any changes made that aren't case related (upper to lower, i.e.)

What GPXClean can do (optional):

  • Change the character case in the 'name' field
  • Change the character case in the 'gpxx:StreetAddress' & 'gpxx:City' fields
  • Change abbreviated road types into their long names (Rd to Road, Ave to Avenue, etc.) and remove trailing periods
  • Add/change/remove the country field
  • Modify the format of the phone number (to either xxx-xxx-xxxx, (xxx) xxx-xxxx or xxx.xxx.xxxx)
  • Append ' - ' to the POI name, to make it more unique
  • Make sure mcx stays McX (McAllen, McDonalds, McMillan, etc.) if 'normal' is set in Name (-poi) or City & Address (-case) fields
  • Make sure states are capitalized if 'normal' is set in Name (-poi) or Address (-case) fields
  • If -case is 'normal', it will capitalize directions (N, S, E, W, NE, etc.) and remove the period
  • Log to a file any format changes in the phone number

The best part about it is it's free. My time, my server, your benefit. Let's start spring cleaning early!

http://camerabob.dyndns.org:5190/poi-factory/gpxclean.html

Enjoy all

-Bob

P.S. -message me if your find any errors anywhere with the package or web site. I'll do my best to correct them.

--
Striving to make the NYC Metro area project the best.
Page 1>>

sounds great

A late Christmas present. Something we all can use.

I will have to tinker with it later. Got company showing up soon for a few days.

--
Harley BOOM GTS, Zumo 665, (2) Nuvi 765Ts, 1450LMT, 1350LM & others | 2019 Harley Ultra Limited Shrine - Peace Officer Dark Blue

GPXClean

THANKS camerabob.

I appreciate the work and for sharing.

Thanks camerabob.

Thanks camerabob.

--
Charlie. Nuvi 265 WT and Nuvi 2597 LMT. Android Here WeGo - Offline Maps & GPS.

Are you sure you won't need a donation for chocolates.

Thanks camerabob. As you say on your site EPE is the program of choice and this will cleanup any mistake when making the files (usually caused by lack of sleep and too much beer) LOL.

--
All the worlds indeed a stage and we are merely players. Rush

LOL!

d-moo70 wrote:

Are you sure you won't need a donation for chocolates.

My wife is currently gambling her chocolates in Las Vegas... Hopefully, she'll bring back more than she left with... LOL!

Nice program, camerabob!

Thank You

Thank you camerabob. I'll have to try this out.

--
Garmin Nuvi 765T, Garmin Drive 60LM

Chocolates

turboccc wrote:
d-moo70 wrote:

Are you sure you won't need a donation for chocolates.

My wife is currently gambling her chocolates in Las Vegas... Hopefully, she'll bring back more than she left with... LOL!

Nice program, camerabob!

As long as she gets her chocolates, that is important. What she does with her chocolates is her business. laugh out loud

--
Harley BOOM GTS, Zumo 665, (2) Nuvi 765Ts, 1450LMT, 1350LM & others | 2019 Harley Ultra Limited Shrine - Peace Officer Dark Blue

Question

If I just run the program will it clean without me inputting a location?

--
Garmin Nuvi 255W

No.

benparlament wrote:

If I just run the program will it clean without me inputting a location?

It needs something to work with. The least (the first time around) is the source file (-i source_file). You could edit the GPXClean.ini file and stick the source file in there at the end. If you do that, it will run just fine with only the program on the command line. It won't scour the computer looking for data to chew through.

--
Striving to make the NYC Metro area project the best.

Too many late nights of online poker

d-moo70 wrote:

Thanks camerabob. As you say on your site EPE is the program of choice and this will cleanup any mistake when making the files (usually caused by lack of sleep and too much beer) LOL.

Also because the web sites I/we scour data from would give an Oreck a run for it's money... They SUCK... I never knew that there were so many different ways to abbreviate crap before I took this on...and all in one file to boot!

--
Striving to make the NYC Metro area project the best.

A question

How much space do you save cleaning your gpx files?

Needs an option to standardize abbreviations

My preference is to not expand Ave to Avenue, as that's more characters to absorb when riding and looking at a POI. I would like the abbreviations made standard, e.g. changing ST. to St and no period. This is the way I see the street labels in Google Earth or Garmin map database.

Your problem with Saint, Doctor and Senior could be minimized by detecting a space in front of St, Dr and Sr and immediately following as your condition for processing. So 123 St Charles St would correctly parse to only process the second St in that field.

I'd also like to see abbreviations carried to street names with ordinal numbers, e.g. substitute 1st for First, 2nd for Second. Again, this is similar to the street naming in GE or Garmin maps.

--
Zumo 550 & Zumo 665 My alarm clock is sunshine on chrome.

what about...

dave817 wrote:

My preference is to not expand Ave to Avenue, as that's more characters to absorb when riding and looking at a POI. I would like the abbreviations made standard, e.g. changing ST. to St and no period. This is the way I see the street labels in Google Earth or Garmin map database

I'll chew on that.

dave817 wrote:

Your problem with Saint, Doctor and Senior could be minimized by detecting a space in front of St, Dr and Sr and immediately following as your condition for processing. So <gpxx:StreetAddress>123 St Charles St</gpxx:StreetAddress> would correctly parse to only process the second St in that field.

That wasn't my concern. I was hitting Dr. Martin Luther King Ave (or Blvd), or St Charles Place. St wasn't in the string twice... Both are preceded and followed by space. There's no simple logic to parse that.

dave817 wrote:

I'd also like to see abbreviations carried to street names with ordinal numbers, e.g. substitute 1st for First, 2nd for Second. Again, this is similar to the street naming in GE or Garmin maps.

How far down the line do ya go? 1234th street?

--
Striving to make the NYC Metro area project the best.

Probably none

polaris_silvertree wrote:

How much space do you save cleaning your gpx files?

More than likely by expanding names it may grow a bit. This isn't about space conservation, it's about making a more easily human readable POI file.

--
Striving to make the NYC Metro area project the best.

Great Information....

Thanks.

--
RKF (Bethesda, MD) Garmin Nuvi 660, 360 & Street Pilot

I'd like more control over changes

camerabob wrote:
dave817 wrote:

Your problem with Saint, Doctor and Senior could be minimized by detecting a space in front of St, Dr and Sr and immediately following as your condition for processing. So <gpxx:StreetAddress>123 St Charles St</gpxx:StreetAddress> would correctly parse to only process the second St in that field.

That wasn't my concern. I was hitting Dr. Martin Luther King Ave (or Blvd), or St Charles Place. St wasn't in the string twice... Both are preceded and followed by space. There's no simple logic to parse that.

Search for " St</gpxx". That will skip " St" not followed by the end of field delimiter.

camerabob wrote:
dave817 wrote:

I'd also like to see abbreviations carried to street names with ordinal numbers, e.g. substitute 1st for First, 2nd for Second. Again, this is similar to the street naming in GE or Garmin maps.

How far down the line do ya go? 1234th street?

In NYC, 150th St but I've not seen streets with 4 digits.

I've not been able to turn off change POI name case shift, so results are not what I'd like:

*Windy City H-D changed to *windy City H-D
AD Farrow Co H-D changed to Ad Farrow CO H-D

Adamec H-D of Orange Park changed to Adamec H-D Of Orange Park (prepositions should not be capitalized) ref. http://www.rasmusen.org/w/capitalization.htm

There are a lot of personal preferences in this stuff also, such as I prefer to see a County road as "CR-A", not "Cr-A". I've seen GE use "Co Rd A" so I'm not consistent either smile. Possibly where I'm headed is that if you are using Perl, can you break out your rules into a SED.txt file that GPXClean would read?

--
Zumo 550 & Zumo 665 My alarm clock is sunshine on chrome.

A problem with that as well

dave817 wrote:

I'd also like to see abbreviations carried to street names with ordinal numbers, e.g. substitute 1st for First, 2nd for Second. Again, this is similar to the street naming in GE or Garmin maps.

A good idea, but there is a problem with that, insofar as different cities use both of those naming conventions eg. Second, and 2nd.

--
nüvi 3790T | nüvi 775T | Those who make peaceful revolution impossible, will make violent revolution inevitable ~ JFK

thanks...

Thanks for all the hard work you guys do and everyone's contributions on this site.

I agree to an extent.

dave817 wrote:

There are a lot of personal preferences in this stuff also, such as I prefer to see a County road as "CR-A", not "Cr-A". I've seen GE use "Co Rd A" so I'm not consistent either smile. Possibly where I'm headed is that if you are using Perl, can you break out your rules into a SED.txt file that GPXClean would read?

Yes. I wrote it for me. wink It does (did) what I wanted. Everything else is gravy. How granular can you get? As granular as I'm willing to put my spare time in. Keep up the feedback. As time permits, I'll do what I can. (tough crowd...)

--
Striving to make the NYC Metro area project the best.

Version 0.86 is now out

Changes made in v0.86:

  • Allow you to define your own descriptions for the list of common road types

  • Allow you to define a list of common words that will retain their case (prepositions, i.e.)

See the readme.txt file on the web page for more information.

http://camerabob.dyndns.org:5190/poi-factory/gpxclean.html

--
Striving to make the NYC Metro area project the best.

You could always...

..dig into the .ini file and add them there yourself using the -special="CO". Then EVERY occurrence of CO will be raised. Hence my issue. 'Co' is something I'm probably not going to touch. Could be County, Colorado, company... I could spend a month of Sundays trying to program that in. (or out) I'll dig into this beast I created to see if I can have it skip all occurrences of 'Co'. This simple little script is now over 1200 lines long...

--
Striving to make the NYC Metro area project the best.

ty

Thanks, this will be very useful.

It always starts small... LOL!

camerabob wrote:

... I could spend a month of Sundays trying to program that in. (or out) I'll dig into this beast I created to see if I can have it skip all occurrences of 'Co'. This simple little script is now over 1200 lines long...

It always starts small... LOL! Then, it becomes a monster... smile

Thanks for sharing, it is

Thanks for sharing, it is very much appreciated...

--
NickJr Nuvi 3597LMT

What!!!

What GPXClean cannot do:
Clean .csv files
Convert between file types
Wash windows
Cook dinner <---DEAL BREAKER FOR ME!!! LOL

So far I like it, got to spend more time playing with it but I like what I see.

Thank

--
(formerly known as condump) RV 770 LMT-S, Nuvi2797LMT, Nuvi765T

Give me a sample file (or a name)

dave817 wrote:

*Windy City H-D changed to *windy City H-D
AD Farrow Co H-D changed to Ad Farrow CO H-D

The sample files I grabbed were Shell and BoA_ATM. Spent a couple hours converting them to .gpx first (They're both .csv. If you guys want my hard fought work, let me know. I'll send off the field correct .csv files.) They had 10k+ and 16k+ records. I wanted to see how my little program would work, and also get some differently formatted text. They went through OK, and what didn't go through allowed me to add those discrepancies into the program. I had yet to run into a POI name or address with an asterisk in it, that's why it wasn't raised.

--
Striving to make the NYC Metro area project the best.

I should have asked the master...

turboccc wrote:
camerabob wrote:

... I could spend a month of Sundays trying to program that in. (or out) I'll dig into this beast I created to see if I can have it skip all occurrences of 'Co'. This simple little script is now over 1200 lines long...

It always starts small... LOL! Then, it becomes a monster... smile

...before I jumped in with both feet! Grasshopper is learning wink

--
Striving to make the NYC Metro area project the best.

Just put up version 0.87

Added the ability to change ANY term to any other term anywhere in the file. Have at it!

Also corrected: The 'save case' terms would get overwritten when executing the program. They will now be preserved.

-Bob

--
Striving to make the NYC Metro area project the best.

Thanks!

I new to this and I've learn more in 4 days that the previous time with my GPS...
Now I'm using more of my Free Web-bing time to check this site
Thank to all of you guys that put your time and effort to "educate" other people.
Hope that I will be able to help as well...shortly.

--
Enjoying each turn...even when lost!

Version 0.88.0.4 posted

The custom changes are now logged in the changes .txt file. (I forgot that bit at midnight...)

The case changes for CO are now being ignored. If you wish to change 'CO', add it to the -begin case- area (safe if preceded and followed by a space), or to the -begin custom- area (NOT SAFE. Will make the change anywhere in the file. Use with caution...).

I think with what's out there now, most folks could go through most files without major hassles. If you find a term that you think should be added as a default definition, let me know. I wrestled with 'Interstate', but one file had streets named Interstate. May have been the service road, but that was enough to turn me off of the idea.

0.88.0.1 Corrected the 'skip CO' feature.
0.88.0.2 Algorithm to skip custom if no definitions. Custom slows the processing down by 50%
0.88.0.3 Corrected code to allow spaces in the output and change file names from the command line
0.88.0.4 Corrected code so the case sensitive and custom options will run without either .ini file existing

--
Striving to make the NYC Metro area project the best.

Defrag

camerabob wrote:
polaris_silvertree wrote:

How much space do you save cleaning your gpx files?

More than likely by expanding names it may grow a bit. This isn't about space conservation, it's about making a more easily human readable POI file.

Is it more like a de-fragmentation program?

--
Garmin Nuvi 260W Garmin Nuvi 1490T If you think knowledge is expensive, try ignorance.

This can now be done, more or less

dave817 wrote:

My preference is to not expand Ave to Avenue, as that's more characters to absorb when riding and looking at a POI. I would like the abbreviations made standard, e.g. changing ST. to St and no period. This is the way I see the street labels in Google Earth or Garmin map database.

See version 0.90, posted 1/21.

--
Striving to make the NYC Metro area project the best.

Not at all

jamstyle wrote:
camerabob wrote:
polaris_silvertree wrote:

How much space do you save cleaning your gpx files?

More than likely by expanding names it may grow a bit. This isn't about space conservation, it's about making a more easily human readable POI file.

Is it more like a de-fragmentation program?

It has nothing to do with the way the file is saved. It's all to do with the content in the file...the part us humans see. Making it look...'normal'. First letter capitals, no multiple spaces, etc.

--
Striving to make the NYC Metro area project the best.

Have you tried

dave817 wrote:

I've not been able to turn off change POI name case shift, so results are not what I'd like:

*Windy City H-D changed to *windy City H-D
AD Farrow Co H-D changed to Ad Farrow CO H-D

"GPXCleane.exe <POI_name> -poi off"?

--
Striving to make the NYC Metro area project the best.

Version 0.89 is out

Added support for symbols, for whatever that's worth...

The ability is there to add, overwrite, or remove existing symbols. I will build a web page for a list of valid (as far as the program is concerned) symbols. There are way more available in Mapsource, but I didn't have the time to grab them all and program them in.

--
Striving to make the NYC Metro area project the best.

Come in handy

camerabob wrote:

Added support for symbols, for whatever that's worth...

The ability is there to add, overwrite, or remove existing symbols. I will build a web page for a list of valid (as far as the program is concerned) symbols. There are way more available in Mapsource, but I didn't have the time to grab them all and program them in.

This option(if I understand correctly) will come in handy for eliminating the apostrophe in the combined Walmart Sam's file. Currently if you spell the name Sam's like Sams there will not be any Sam's found. It is a pain to have to enter the apostrophe just to search for Sams. Thanks.

--
Nuvi 2460LMT

Thanks -

Thanks, camerabob, your hard work is much appreciated.

That cna be done

In the -begin custom- section. Just add "Sam's = Sams" (without the quotes). The program will strip every occurrence of that darned apostrophe, and send them into the bit bucket.

--
Striving to make the NYC Metro area project the best.

Version 0.89.0.1 posted

-Corrected skewed logic in the symbol manipulation area.
-Added spell check for all the symbols recognized from within Extra POI Editor and MapSource.

--
Striving to make the NYC Metro area project the best.

By the way...

...the symbols are the POI/GPX symbols. (Trail Head, Residence, Waypoint, etc.), not the punctuation within the file (I now comprehend the question presented earlier....)

--
Striving to make the NYC Metro area project the best.

Doesn't seem to work

camerabob wrote:

In the -begin custom- section. Just add "Sam's = Sams" (without the quotes). The program will strip every occurrence of that darned apostrophe, and send them into the bit bucket.

I have tried Sam's = Sams but nothing changed.

I have also tried the below but no change.

-begin custom-
Sam's = Sams with " & apos ; s " in Sam's
-end custom-

The poi file is Wal-Mart_US&Canada.gpx .

--
Nuvi 2460LMT

I have been a member a long

I have been a member a long time and this forum has to be one if not the best on the net. So many users devote their valuable time to help out one another, it is great to be associated with them and I am sure as you become more up to date on your GPS you will also contribute as well.

--
NickJr Nuvi 3597LMT

It's automatic

The program converts ' to ' automatically without having to do anything. Run it once and check the output. Because I have to possibly capitalize after an apostrophe, I convert them when I find them. Try Sam's = Sams.

--
Striving to make the NYC Metro area project the best.

Different Program

@camerabob,

Your working so hard and fast on this program it isn't even the same program as when you introduced it a week ago. smile Way to go!

After a hard week at work I think might get a chance to give it a try this cold and snowy weekend.

--
Harley BOOM GTS, Zumo 665, (2) Nuvi 765Ts, 1450LMT, 1350LM & others | 2019 Harley Ultra Limited Shrine - Peace Officer Dark Blue

dang HTML

camerabob wrote:

The program converts ' to ' automatically without having to do anything. Run it once and check the output. Because I have to possibly capitalize after an apostrophe, I convert them when I find them. Try Sam's = Sams.

Should say "& apos; to ' "

--
Striving to make the NYC Metro area project the best.

You should see my first draft

bear007 wrote:

@camerabob,

Your working so hard and fast on this program it isn't even the same program as when you introduced it a week ago. smile Way to go!

After a hard week at work I think might get a chance to give it a try this cold and snowy weekend.

It looks anorexic at 300 lines. wink That was way before you guys even knew it existed....

--
Striving to make the NYC Metro area project the best.

Done that.

camerabob wrote:
camerabob wrote:

The program converts ' to ' automatically without having to do anything. Run it once and check the output. Because I have to possibly capitalize after an apostrophe, I convert them when I find them. Try Sam's = Sams.

Should say "& apos; to ' "

I copied the "& apos;" from the original input file and put that in as I said in my prior post.
I notice that after running (with or without Sam anything in the .ini) the output file has an " ' " and not the "& apos;".

Another thing I notice is the word Street does not change to ST although I have Street = St. Other changes in the dictionary section don't work either. The change file shows a lot of changes but not the changes I'm putting in the GPXClean.ini file.

--
Nuvi 2460LMT

check the output screen

It may be finding the Walmart....ini file. This is the first place it looks. If it exists, just delete it and run again. Then in the future, aim your changes to this file.

--
Striving to make the NYC Metro area project the best.

Could you test?

camerabob wrote:

It may be finding the Walmart....ini file. This is the first place it looks. If it exists, just delete it and run again. Then in the future, aim your changes to this file.

Could you test on Wal-Mart_US&Canada.gpx ?

Sam's = Sams doesn't work for me.
Street = St doesn't work for me.

--
Nuvi 2460LMT

I see your point. Download v0.90, just up

Thanks for the find mmullins98.

There was an AND statement when there shoulda been an OR... Happened when I added code to bypass the custom to speed things up. Give the new version a whirl.

Also, make sure you change "Street " to "St ", or gpxx:StreetAddress will get changed as well.

Just what I want to be doing on a Friday night.... mad

Fixed the .ini file dropped the dot.

Even though the custom area is fixed, you can skip it and just put the changes in the road type area. From the readme on the latest version:

  • Version 0.90 - January 21, 2012
    -Changed the code so that the full length names in the address field will be altered as well as the abbreviated names. (Street = St., i.e.)
--
Striving to make the NYC Metro area project the best.
Page 1>>