We recently were put in touch with a real estate agent who is helping us to scout out potential rental properties in the NC area.  We aren’t going to be really looking seriously until the summer, but figured: why not start now.

Fortunately, Melisa was able to set up a great personalized site for us that shows listings as they appear in the market in our target areas.  The searches are broader than the areas we are interested in, but that’s ok because it gives us a broader view of the market.

And because it’s an aggregated list, it gives me a chance to do some cool-ish things with the data.

Selenium + Python Regex

For those that haven’t used it, Selenium is a great library for dealing with browsers.  I have found it especially useful for grabbing data from pages that do not expose it in their basic HTML.  It is also great when you have any sort of authentication required to access a site.

In other words, it was made for this application.

Using Selenium, I can access my personalized link, and I can iterate through the various “searches” that have been set up.  Since a search corresponds to an area of neighborhood, this makes it easy to avoid having to see every listing in the whole RDU area.  (In some cases, this is impossible because, like this site, the lower records don’t load unless you scroll down, which is tricky with Selenium).

Once I have the new search results loaded, Selenium allows me to grab the visible text from the search area.  A simple Python regex returns a dict with each listing detailed out.  I am grabbing address, MLS, price, sqft, beds, baths, and the description of the listing.

Storing is boring

If it’s an address that I haven’t seen before, it goes straight into the database. If it is a listing I’ve seen before, I just update the “last seen” value in the DB and go on my way.

My DB contains listings from lots of different aggregator sites.  I will be very curious to see if the listings from this special, personalized set up end up in the DB before a site like Zillow gets them.  If not, then hopefully this special search has more listings.  If it doesn’t, then I can more or less ignore what it’s sharing since I should have already seen the data elsewhere

But this piece is a bit boring.

Enhanced Interrogation Techniques

Where it gets interesting is the extra stuff that I can do once a listing has made it into my database.

The first, is to grab the coordinates.  Google has an API that allows for address-based lookups of long/lat.  One of my next steps is to start plotting all the listings in D3 as a way to make sure that the areas we are interested are actually getting scanned via the searches they set up.

Having the address and the Google APIs also makes it possible to do things like find the closest grocery store.  I would love to be able to bike to get groceries wherever we move.  And with the address in hand, I can immediately look that information up.  That way, along with the alert that a new listing is available, I can tell if it meets that sort of requirement.

Last piece is to compare the listing to others in the area.  Since I know the square footage and the rental price, it’s pretty easy to calculate a relative ranking for the listing. In other words, how does the price / sqft compare to other listings in that neighborhood.  Having this in near-real time will make it easier to evaluate the alerts that pop up.

Um, why?

In the best case scenario, because I am able to collect more enhanced information about a listing very quickly after it is posted, I may have a leg up on others looking at the area.  And that would be without the risk of a bad decision made quickly (in theory).

But if all that comes out of this is that I built some cool scrapers and data viz, it’ll have been a Monday well-spent.