Creating a POI file from scratch - Raw Data

I personally would recommend GeePeeEx Editor. You can create detailed POI files (including dialable phone numbers) and the program includes a google map function so you can view your coordinates through a google sat image.

The trial version is fully functional but is limited to five waypoints.
http://geepeeex.googlepages.com/home

I use Google earth for my personal Lists and where available, I use the street-view to identify the exact building. Takes a lot more time than running the addresses through a Geo-code program, but my results have been pretty much spot-on! Unfortunately not all locations have clear maps or street-view, so you still end up with best guess, unless you actually visit the location and mark it yourself.

I can't tell whether you are planning to use the exact coordinates of the building or not. Rarely is that a good idea -- you usually want the coordinates to at the curb (or driveway entrance) as opposed to a building that is set back too far from the road to give an alert. If you don't want alerts and you need to find a specific building in a campus or large industrial complex, that might be an exception, but you will probably get only a straight line as a route once you leave the nearest street.

Depending on what poi file I'm working on, Imay be running 2 or three programs at once.

When I am updating the Diners, Drive-ins & Dives gpx (http://www.poi-factory.com/node/15558) I have GeePeeEx Editor running and a web browser with at least two windows open. One is the Food Network's webpages to provide me with the restaurant's details and the other is Google Maps. With Google Maps I search the address and I have "Street View" turned on. If the location has been photographed I can actually read the signs on the building. If need be I can "walk up and down the street" reading the signs until I find the location. I then right click to get "Directions to" that location, which outputs the Long/Lat. These are then copy/pasted into the appropriate fields in GeePeeEx. A third window allows me to Google the business by name and read reviews or look at the business' website if the street view photographs have not yet been taken at that location. Additional browser windows are opened as required. (I too could use the Google Maps feature that GadgetGuy2008 suggests above, but I feel I have a little more flexibility doing the way I describe.)

The third program I run is Google Earth. I'm building POI files that do not have addresses so I find the location I want, zoom in with Google Earth and create a POI at the correct location. I then save out these POIs as a single KMZ file, which when unzipped into a KML file can be imported by GeePeeEx. Any information I've added into the placemark field in Google Earth is automatically imported into the text field for the waypoint by GeePeeEx too.

I am doing much the same as you in compiling an accurate POI file. I use a different set of programs and check each address individually. My main program for geocoding is itouchmap.com (Option 10) which allows me to enter a street address. This brings up a Google Earth display of the address with a pointer that can be moved. I position the pointer on the exact location I want encoded, and then scroll to the bottom of the page. There I have my choice of either DMS or decimal degrees of the pointer location. I can also use this page to enter a set of coordinates and have a pointer placed on the map.

This is absolutely great for finding and visualizing the errors from batch geocoding programs.