EPE acting wierdly

 

Very strange thing with EPE today. I downloaded the Speed Cameras and Red-Light Cameras files today as I do every week. And as I've done every week for the last ten years or so, I opened the Speed file first, deleted the "poi-factory " part in every line, saved and closed the file, and used EPE to convert it to a .GPS file. Everything was cool.

Then as I've done every week for the last ten years or so, I opened the Red-Light file, deleted the "poi-factory " part in every line, saved and closed the file. But this time when EPE tried to open the file, the upper-left panel filled in, as usual, but the lower-left panel remained blank. Hmm.

So I downloaded the RLC file again, this time not removing the "poi-factory ", and this time EPE handled the file correctly. I went back into the downloaded .csv file and just added one space to the first line, backspaced over the one space I just added and saved and closed the file. EPE failed again. Now remember, I've been doing this very same technique for ten years and EPE has always worked correctly. Today EPE worked correctly for the altered Speed Camera file and failed every time for the altered RLC. I did the test several times and EPE failed every time. It never failed if I used the unchanged file.

Anyone willing to try to reproduce the problem?

Phil

--
"No misfortune is so bad that whining about it won't make it worse."

No problems here

I assume you are using something simple like notepad or excel to remove the text. Try the other one of these to see if that's your problem. Maybe they saved the csv with DOS vs. Unix line feeds.

I use VIM and make drastic edits to the files, converting the 4 columns to more columns so I can populate the State, City and text comments. I also convert Canada metric speeds to US measurements. Then I import into EPE as a 5 or 6 column csv. I add Symbol (Custom 4/Custom 5) and Display (SymbolOnly) to each POI and save. It takes minutes and gives me a nicely formatted gpx. I saw no problems with the RLC file.

--
Zumo 550 & Zumo 665 My alarm clock is sunshine on chrome.

Worked for me

plunder wrote:

Anyone willing to try to reproduce the problem?
Phil

I downloaded RLC. Opened the file in Notepad++; deleted " poi-factory Jul19 17" (not including quotes); saved the file, and then opened it in EPE. While the first second or so I did not see the lower left pane, it appeared. EPE showed 3,521 entries.

So - I cannot reproduce your problem.

John

I don't care that much

dave817 wrote:

I assume you are using something simple like notepad or excel to remove the text. Try the other one of these to see if that's your problem. Maybe they saved the csv with DOS vs. Unix line feeds...

I use Notepad, but I gotta tell you, I'm not interested in trying new things. All I can say is that it's been working forever and the Speed Camera still works. It's this week's RLC with any change apparently that causes EPE not to work.

Phil

Edit: I just rebooted my laptop and downloaded both the Speed and Red Light files again. Using Notepad I made exactly the same changes that I mentioned above and the results were exactly the same: the Speed file worked correctly while the Red Light file failed to display data in the lower left-hand frame. I'll wait until next week's files become available.

--
"No misfortune is so bad that whining about it won't make it worse."

Phil

plunder wrote:

...
I use Notepad, but I gotta tell you, I'm not interested in trying new things. All I can say is that it's been working forever and the Speed Camera still works. It's this week's RLC with any change apparently that causes EPE not to work.

Phil

Edit: I just rebooted my laptop and downloaded both the Speed and Red Light files again. Using Notepad I made exactly the same changes that I mentioned above and the results were exactly the same: the Speed file worked correctly while the Red Light file failed to display data in the lower left-hand frame. I'll wait until next week's files become available.

Using Notepad, I did duplicate your results. I tried "Save as" using both ANSI and UTF-8

Educate me

What is the reason or advantage to removing the poi-factory text from a file here?

To save space

CraigW wrote:

What is the reason or advantage to removing the poi-factory text from a file here?

Merely to save 12 bytes per record. No nefarious plot.

Phil

--
"No misfortune is so bad that whining about it won't make it worse."

Thanks

plunder wrote:
CraigW wrote:

What is the reason or advantage to removing the poi-factory text from a file here?

Merely to save 12 bytes per record. No nefarious plot.

Phil

Thanks. Just curious.

Removing

I have downloaded files from POI factory such as Walmart, McDonalds which have locations all over North America.

I don't really expect to go west of Kansas so I have taken the time using MapSource to remove all locations west of Kansas, going north and south. In the grand scheme of things these do not make one heck of a difference in file size once converted to GPI files.

So I cannot see removing 12 bytes will matter much in the total file size except to make a lot of work and as we see now frustration to the attempt. My opinion anyway.

--
Nuvi 2797LMT, DriveSmart 50 LMT-HD, Using Windows 10. DashCam A108C with GPS.

No argument here

Melaqueman wrote:

I have downloaded files from POI factory such as Walmart, McDonalds which have locations all over North America.

I don't really expect to go west of Kansas so I have taken the time using MapSource to remove all locations west of Kansas, going north and south. In the grand scheme of things these do not make one heck of a difference in file size once converted to GPI files.

So I cannot see removing 12 bytes will matter much in the total file size except to make a lot of work and as we see now frustration to the attempt. My opinion anyway.

I don't disagree with you at all, well maybe a little. I merely found it interesting that after all these years, something has changed so that EPE is having a problem. My reasons for doing things the way I do have no bearing on the problem.

Phil

--
"No misfortune is so bad that whining about it won't make it worse."

Worked for me too

I got the same results as John.

Using EPE V5.99 on Windows 10 Pro build 16241

jgermann wrote:
plunder wrote:

Anyone willing to try to reproduce the problem?
Phil

I downloaded RLC. Opened the file in Notepad++; deleted " poi-factory Jul19 17" (not including quotes); saved the file, and then opened it in EPE. While the first second or so I did not see the lower left pane, it appeared. EPE showed 3,521 entries.

So - I cannot reproduce your problem.

John

--
Nüvi 2595LMT

Update

plunder wrote:
dave817 wrote:

I assume you are using something simple like notepad or excel to remove the text. Try the other one of these to see if that's your problem. Maybe they saved the csv with DOS vs. Unix line feeds...

I use Notepad, but I gotta tell you, I'm not interested in trying new things. All I can say is that it's been working forever and the Speed Camera still works. It's this week's RLC with any change apparently that causes EPE not to work.

Phil

Edit: I just rebooted my laptop and downloaded both the Speed and Red Light files again. Using Notepad I made exactly the same changes that I mentioned above and the results were exactly the same as last week: the Speed file worked correctly while the Red Light file failed to display data in the lower left-hand frame. I'll wait until next week's files become available.

I took this week's RLC and Speed Camera POIs and ran exactly the same routine as in past weeks/years and got exactly the same results as last week: EPE handled the Speed Camera file nicely but mis-handled the RLC. Same deal: EPE didn't show the file in the lower left-hand panel.
I decided to press on as in the past, so I set the proximity to the usual 600 feet and saved the file. The file appears correct with the proximity showing up in the saved .GPX file. Not that anyone be me cares.

Phil

--
"No misfortune is so bad that whining about it won't make it worse."

Humor Me, please

@Plunder,

Will you try your routine using Wordpad, please?

John

Further testing

jgermann wrote:

@Plunder,

Will you try your routine using Wordpad, please?

John

Okay, John, I simplified reproducing the error down to the nub. Here's the deal:
1. Download the Red Light file as r0.
2. Download the Red Light file as r1.
3. Using a fresh copy of EPE each time, I can open both r0 and r1 correctly (as expected).
4. Close EPE.
5. Bring up EPE.
6. Open r0 with NOTEPAD, immediately save and close it without making any changes.
7. Open r0 with EPE. Lower left panel does NOT get displayed.
8. Close EPE.
9. Bring up EPE.
10. Open r1 with WORDPAD, immediately save and close it without making any changes.
11. Open r1 with EPE. Lower left panel DOES get displayed.
12. Ran the same tests several times are results are consistent.

Conclusion: The act of merely opening, saving, and closing RLC with NotePad causes EPE to not display the lower left panel. Opening, saving, and closing RLC with WordPad causes EPE to correctly display the lower left panel.

Notes:
1.As far as I know, it's only the RLC file that displays this effect.
2 Even with the lower left panel NOT displayed EPE still handles the file correctly.

Phil

--
"No misfortune is so bad that whining about it won't make it worse."

@Phil

Thanks a lot for that work.

By playing around with files saved by Notepad, I finally got EPE to display the POI-Edit dialog for the first entry [which is "RLC - Pacific Hwy & 196th" (without the quotes)]. What I found was a "Long:" of:

-122.31507

I will later try to replicate the sequence of steps that got me to this screen. What I started with was clicking on the second entry - at which point EPE filled out the lower left panel for me.

When I was double clicking on entries, I found that the POI-Edit dialog would not display. BUT - when I would hover on the Icon in the TaskBar, I would see that the dialog was there. I could get the dialog to display by right clicking on it and choosing the "maximize" option.

Then, rather than my normal practice of doubling clicking on the .csv file (because I have told windows to always use EPE when I open such a file), I started opening EPE and then opening the .csv file by name. Somehow all the activity has now caused me to be able to double-click my test .csv files and be able to double-click on the first entry and get to the POI-Edit dialog (even though the lower left pane is not displayed).

I need to run an errand but I will be back. I would love to understand what in the world Notepad is doing.

BOM

Notepad is adding the Unicode Byte Order Mark (BOM) for UTF-8 to the beginning of the file when saving it and apparently EPE chokes when it sees that character sequence. I suspect the file I/O routines in the old Visual Basic environment (that EPE was developed in) are not Unicode aware.

I don't know why Notepad is doing this on the red light camera file and not the speed camera file, but there must be something in the data that makes Notepad think it needs to add this UTF-8 identifier. You can read more about it on Wikipedia: https://en.wikipedia.org/wiki/Byte_order_mark

--
Alan - Android Auto, DriveLuxe 51LMT-S, DriveLuxe 50LMTHD, Nuvi 3597LMTHD, Oregon 550T, Nuvi 855, Nuvi 755T, Lowrance Endura Sierra, Bosch Nyon

@Alan

Thanks for that reference. I would have never found that.

Now, we need to find out why Notepad is saving the RedLight-Cameras file in UTF-8 format when Phil saves it.

Phil, were you doing a "Save" or a "Save As"?

What I did was "Save As" and I added characters after "Redlight-Cameras" (such as NP UTF or NP ANSI) so that I could keep track of what worked and what did not, because I noticed that when I did the "Save As" that I could choose encoding.

save as

When doing a "save as" in Notepad in Windows 7, there is an encoding drop down menu with the choices:

ANSI (this appears to be the default)
Unicode
Unicode big endian
UTF-8

There is no such option doing a "save." It would be speculation to assume that the last encoding choice would carry over into future "saves." But:

https://answers.microsoft.com/en-us/windows/forum/windows_7-...

dobs108 smile

I do a "Save"

jgermann wrote:

...Phil, were you doing a "Save" or a "Save As"?...

I do a "Save." And let's not forget that this has been working for 10 years up until 2 weeks ago. Really weird. Oh, and my environment hasn't changed any time lately.

Phil

--
"No misfortune is so bad that whining about it won't make it worse."

Work around.

jgermann wrote:

Thanks for that reference. I would have never found that.

Now, we need to find out why Notepad is saving the RedLight-Cameras file in UTF-8 format when Phil saves it...

I would guess that Notepad looks for certain characteristics in the data characters and sequences contained in the file to determine the default encoding type for a Save. If that is the case, something has apparently been recently added in the red light file that makes Notepad "think" this UTF-8 tag needs to be added. My Notepad.exe file (on Vista 64) is dated 7/9/15, so I don't think there have been any changes to Notepad in the past couple of years.

Since plunder posted this on July 21, I assume the problem first occurred with the red light file posted on July 19. If someone still has the file from July 12 (assuming that Notepad does not cause the issue in the July 12 file) we could probably start comparing the old file to the newer file to figure out what is triggering this Notepad behavior.

At least knowing about it and using the "Save As" with "ANSI" as identified by Dobs108 and jgermannn is a good work around. And the good news is that we are now certain that it is a Notepad issue and not an EPE issue.

--
Alan - Android Auto, DriveLuxe 51LMT-S, DriveLuxe 50LMTHD, Nuvi 3597LMTHD, Oregon 550T, Nuvi 855, Nuvi 755T, Lowrance Endura Sierra, Bosch Nyon

same date

alandb wrote:

...My Notepad.exe file (on Vista 64) is dated 7/9/15, so I don't think there have been any changes to Notepad in the past couple of years...

This is the same date as Notepad.exe for windows 7 x64, fully updated.

dobs108 smile

Same for me

Me also.

Another test

I ran a rather simple test. I used a file compare program (ExamDiff) to compare a newly downloaded copy of RLC.csv with the current RLC that I had "saved" a few days ago.

There was one difference in the compare: the first record in the "saved" file has some bogus character in the first byte. The newly downloaded RLC had no such character. Its first character is the "-" character in the longitude.

So I took the new RLC, the correct one, and merely saved it. Using ExamDiff I compared the same two files and now the file that had been correct had the bogus character also.

I then deleted just the first record but the bogus character had merely migrated to the second record.

As a final test I looked at the Speed Camera file and after changing and saving it, there was no bogus character in the first byte. So whatever is going on appears to be unique to the RLC.

Phil

One final test: I took fresh copies of the RLC, opened one with WordPad and the other one with NotePad, saved each one, then compared the two with each other. As expected by now, the WordPad copy was correct while the NotePad copy had the bogus first byte.

--
"No misfortune is so bad that whining about it won't make it worse."

Unicode Byte Order Mark (BOM) for UTF-8

plunder wrote:

... the WordPad copy was correct while the NotePad copy had the bogus first byte.

Actually, the characters of the BOM should be in the file based on the fact that Notepad has determined or been instructed that the file must be saved as UTF-8.

An update

I just tried this week's RLC file and it was the first time I actually tried to load the POI file into my Nuvi 1450. It failed. First of all, I have no idea what a BOM character is, but I can now attest that it hoses the POILoader.

I wonder what's different with the RLC file compared with every other POI factory file and all of the ones that I've created. Oh, well, I think I'm running out of care.

Phil

--
"No misfortune is so bad that whining about it won't make it worse."

Invalid characters

jgermann wrote:

Thanks a lot for that work.

By playing around with files saved by Notepad, I finally got EPE to display the POI-Edit dialog for the first entry [which is "RLC - Pacific Hwy & 196th" (without the quotes)]. What I found was a "Long:" of:

-122.31507

I will later try to replicate the sequence of steps that got me to this screen. What I started with was clicking on the second entry - at which point EPE filled out the lower left panel for me.

When I was double clicking on entries, I found that the POI-Edit dialog would not display. BUT - when I would hover on the Icon in the TaskBar, I would see that the dialog was there. I could get the dialog to display by right clicking on it and choosing the "maximize" option.

Then, rather than my normal practice of doubling clicking on the .csv file (because I have told windows to always use EPE when I open such a file), I started opening EPE and then opening the .csv file by name. Somehow all the activity has now caused me to be able to double-click my test .csv files and be able to double-click on the first entry and get to the POI-Edit dialog (even though the lower left pane is not displayed).

I need to run an errand but I will be back. I would love to understand what in the world Notepad is doing.

Just noticed this thread. Looks like some invalid characters in the coordinates. EPE is not validating the field value at all. It just passes it from one file format to the other. EPE is not too intelligent and does not filter errors which hides many errors in the data. Still EPE should catch errors and produce a log or error message to help you guys find the problem faster.

Let me know if you need me here.

Turbo

I don't think it's an EPE issue

turboccc wrote:
jgermann wrote:

Thanks a lot for that work.

By playing around with files saved by Notepad, I finally got EPE to display the POI-Edit dialog for the first entry [which is "RLC - Pacific Hwy & 196th" (without the quotes)]. What I found was a "Long:" of:

-122.31507

I will later try to replicate the sequence of steps that got me to this screen. What I started with was clicking on the second entry - at which point EPE filled out the lower left panel for me.

When I was double clicking on entries, I found that the POI-Edit dialog would not display. BUT - when I would hover on the Icon in the TaskBar, I would see that the dialog was there. I could get the dialog to display by right clicking on it and choosing the "maximize" option.

Then, rather than my normal practice of doubling clicking on the .csv file (because I have told windows to always use EPE when I open such a file), I started opening EPE and then opening the .csv file by name. Somehow all the activity has now caused me to be able to double-click my test .csv files and be able to double-click on the first entry and get to the POI-Edit dialog (even though the lower left pane is not displayed).

I need to run an errand but I will be back. I would love to understand what in the world Notepad is doing.

Just noticed this thread. Looks like some invalid characters in the coordinates. EPE is not validating the field value at all. It just passes it from one file format to the other. EPE is not too intelligent and does not filter errors which hides many errors in the data. Still EPE should catch errors and produce a log or error message to help you guys find the problem faster.

Let me know if you need me here.

Turbo

Turbo, I don't think it's an EPE issue. I'm the OP of this thread so I know what the issues are.
For the RLC file ONLY, if I make any change to it, no matter how insignificant, then save it, some kind of character gets added as the first character in the first line. This character appears to stop EPE from displaying its lower left-hand frame.
Even though that lower box is not displayed the file gets updated correctly. For instance, if using EPE I add the proximity parameter with some value, the .GPX file is built correctly with the value I entered.
However, even though file LOOKS correct, it also contains that unknown byte in the first line. This in turn causes an error trying to load the RLC file into the GPS.
To summarize, this doesn't look like an EPE problem. This problem ONLY happens with the RLC file and only if any change is made (even one byte) and then saved. I use NotePad for my work, the RLC/NotePad combination has worked for me for YEARS, and jgermann has been able to reproduce the problem. I used ExamDiff to display the errant byte. If the RLC is not saved by NotePad, everything works - EPE displays correctly and the file loads into the GPS correctly.

Phil

--
"No misfortune is so bad that whining about it won't make it worse."

Possible improvement

Hi Phil,

EPE checks for illegal coordinates value before displaying the map otherwise it would have crashed the javascript running google maps. This is why the map does not show up (bottom left window).

I just added some Lon/Lat value checking in EPE 6.02 (not released yet) when importing from CSV/XLS files. Any illegal character should be detected and a message displayed indicating the line number of the faulty POI. I am also checking if the coordinates are valid (from -180 to 180 degrees). Any values outside the range will also pop an error. This should fix an error I saw in another thread.

This being said, this improvement applies only to CSV or XLS files. Not GPX. That would be some extra work.
Hope this will be useful.
Turbo

Question for you, Turbo

Turbo, do you have any insight on why the RLC file is having a bogus byte added to the first line? Out of the thousands of POI files here at the Factory, why that one? And why now?

Phil

--
"No misfortune is so bad that whining about it won't make it worse."

No idea but...

No idea but if you send me a link to the file I will have a look at it. Thanks.

Turbo

It's the Factory's Red Light Cameras file

turboccc wrote:

No idea but if you send me a link to the file I will have a look at it. Thanks.
Turbo

It's the normal RLC file that gets updated along with the Speed Cameras file every week.

1. Download the RLC file.
2. Make a copy of it.
3. Open the copy with NotePad.
4. Make ANY change (add a blank to the end of a line, i.e.).
5. Save the file.

To see the problem, I use ExamDiff to compare the original to the copy. The first byte of the copy has garbage in it.

I've been working with the RLC file for almost 10 years and the problem just showed up a month ago. No system changes made that coincides with the problem and other folks have reproduced the problem.

Phil

--
"No misfortune is so bad that whining about it won't make it worse."

Not finding file

Hi Phil,

I searched for the camera file but is it possible you need special membership to get this file? I looked on the Red Light & Cameras page but could not find a download link. Just a Subscribe Now button.

Ow! I see. I must be an active member. I think I have been active only 1 week for now. Will take another 2 weeks. Alternatively, email me the file at turboccc at Hotmail . com

Thanks. Turbo.

A usable workaround

For whatever reason, NotePad has been adding a BOM (whatever that is) as the first byte in the Red Light Cameras file. That causes an error in EPE and also causes the POILoader not to load the RLC file.
Today I used NotePad++ to update the RLC and that solution works. I think it's time to put this thread to bed.

Phil

--
"No misfortune is so bad that whining about it won't make it worse."