• Website and Flash Development
  • Content Management Systems
  • .NET, PHP And More!

Blogs

Screen Scraping Made Pretty Easy

Many public libraries offer a valuable but often overlooked resource – free or discounted museum passes.  Here in Western Massachusetts residents have access to the combined materials of the entire CW/MARS regional system of more than 140 libraries.  Courtesy of our library cards, my family and I have made free trips to the Holyoke Children’s Art on exhibit at the Springfield QuadrangleMuseum, Amherst History Museum, Historic Deerfield, Magic Wings, Brattleboro Art Museum, Smith College Art Museum, Springfield Quadrangle and Boston Science Museum.

I became curious to know the full range of museum passes our library system had to offer.  This information is available via the C/W MARS search engine, but the more than 50 results are spread out over 5 rather verbose pages.  What I was really looking for was a concise, browsable list of available museum passes with links to detailed information. 

A powerful, elegant PHP library called Simple HTML DOM Parser makes constructing such a page relatively easy.  Contributed as open source by S. C. Chen, it features a “Find” function that takes arguments modeled after JQuery’s easy-to-use  syntax.  For instance, this small PHP snippet is virtually all that is needed to dump all the link destinations in a web page:

include_once('/simplehtmldom/simple_html_dom.php');
$html = file_get_html('http://some-site.com/some-page.html');
foreach($html->find('a') as $element)
       echo $element->href . '<br>';

I wrote a program that used Simple HTML DOM Parser to loop through each page of search results from a CW/MARS Library Catalog search query on the term "museum pass." The program parsed each page and extracted all HTML nodes with a class of "briefcitTitle" (find('.briefcitTitle')).  Each node contained a link to an individual museum pass page which was extracted and written back out as a table row. With a couple dozen lines of code I was able to put together the list of museum passes I was after:

Museum Passes Available for Loan at Western Mass. Libraries

OK, Use the Map Now

My recent blog post described an inaccuracy in the Google Map generated by entering the Studio4 Technologies office address.

Either by coincidence or, could it be the amazing power of the blogosphere, I find the map of "592 Main St., Amherst, MA" is now correct.  Our map marker is at the corner of Main and North Whitney, right where it should be. 

Thanks Google technician, wherever you are! 

Google map showing correct location of Studio4 Technologies office

 

 

 

 

 

 

 

 

 

 

 

Don't Use This Map!

Google Maps is the application that first clued me into the power of Ajax and it has been my mapping service of choice for quite sometime.  Therefore it is with some dismay that I must point out a glaring inaccuracy in one of its maps -- namely the one generated from the Studio4 Technologies street address!

 


View Larger Map

The pushpin marking the supposed location of 592 Main St. is nearly a half mile off.  The office is located in the Nacul Center, a former Methodist church that has been remodeled into studios and an art gallery, at the corner of Main and North Whitney Streets in Amherst, Mass., and more than 3 blocks west of where Google Maps would have us!

I have attempted to correct the location using Google's process for moving a marker, but the error exceeds the 200 meter limit that Google allows without human review and I am still waiting for their human to respond to my change request after several weeks.

Pay us a visit any time, but, at least in this case, you will be better off using a map from Bing or MapQuest to find your way.

Google Maps API Key Issues Across Development Environments

So, one of our latest projects allowed us to dive into the Google Maps API. What a great utility! It is really easy to do incredibly complex mapping functionality. Can you imagine how difficult this stuff would be 5 years ago?
 
For the API to work properly, the developer must use a key that is generated for a specific URL. But what happens if the site is accessed via different URL's? For example: your staging machine is internal to your network and you access it via and internal IP address, but your client accesses it via a sub domain of your main website. Or, you have an automated deployment of code from source control, but the key that you have checked in relates to you local development environment. In this post I will show you how to create a logical work around so you never have to worry about this again.
 
First, we need to determine all of the environments that the Google Maps API will reside in. This may include Local, Staging, QA and Live. Go to http://code.google.com/apis/maps/signup.html and register these URLs to receive you new API keys.
 
Second, we need to determine what language you will be using for your site. This is very likely to be ASP.NET, ASP or PHP. In this example I will demonstrate in Classic ASP for now, just so we can get the gist of what is going on. If anyone is interested in doing this in other languages, let me know and I will add them to this post.
1. Create the conditional statement that determines what environment you are using. Something like this:
if Request.ServerVariables("SERVER_NAME") <> "localhost" then
 

end if
 
2. Now place the Google Javascript with the corresponding key for this environment.
if Request.ServerVariables("SERVER_NAME") <> "localhost" then
response.write("<script src="http://maps.google.com/maps?file=api&amp;v=2&amp;sensor=false&amp;key=ABQIAAAAnfs7bKE82qgb3Zc2YyS-oBT2yXp_ZAY8_ufC3CFXhHIE1NvwkxSySz_REpPq-4WZA27OwgbtyR3VcA" type="text/javascript"></script>")
end if
 3. For any additional environments just add a new else if into the mix

 
if Request.ServerVariables("SERVER_NAME") <> "localhost" then
response.write("<script src="http://maps.google.com/maps?file=api&amp;v=2&amp;sensor=false&amp;key=ABQIAAAAnfs7bKE82qgb3Zc2YyS-oBT2yXp_ZAY8_ufC3CFXhHIE1NvwkxSySz_REpPq-4WZA27OwgbtyR3VcA" type="text/javascript"></script>")
else if Request.ServerVariables("SERVER_NAME") <> "StagingURL" then
response.write("<script src="http://maps.google.com/maps?file=api&amp;v=2&amp;sensor=false&amp;key=ABQIAAAAnfs7bKE82qgb3Zc2YyS-oBT2yXp_ZAY8_ufC3CFXhHIE1NvwkxSySz_REpPq-4WZA27OwgbtyR3VcA" type="text/javascript"></script>")
end if

 
There you have it! It is a little time consuming at first, but trust me, you will save a lot of time and headaches if you prepare for site environment transitions ahead of time.

 
Hope this was useful! if you have any suggestions for future content, feel free to send them to us.
 
 
 

Welcome to our new site!

The have been a few changes in the company the last few months and we thought this was a great time to revamp the website. What do you think? It is a work in progress, but we are very excited about it. It is giving a new space to try out some new technologies. Please come back to check in on us, as things will be filling out and changing dramatically.

Thanks!

The Studio4 Technologies Team

Syndicate content