Project: SERP Tracking & SEO Analyzer
Finally, I've had a chance to play around with one of the Google APIs. I decided to take my crack at the Google Web Search API since search is the cornerstone of Google's operations. After reading the documentation briefly (and being surprised that they included .Net sample code), I started brainstorming what tools I could build. Eventually, I decided on a SERP tracker/analyzer.
Now, I have quite a long way to go, but I have successfully called the API, extracted the data from a search results and stored the information in the database. For the time being, I am writing the application in VB.Net and utilizing MSFT's SQL Server 2000; however, I have every intention of porting this to other languages and platforms. Finally, I plan on releasing the source freely since the Google Wed Search API has a limit of only 1,000 queries per day and a maximum of 10 results per query. With those limitations, the only way that the "community" can really benefit from this is to run their own specific relevant term searches.
Hopefully, the feature list will include (but not be limited to):
- Port application to .Net, PHP, Python, MS SQL, MySQL, flat text file
- Allow users to dynamically update their API Key
- Allow users to dynamically change their search terms and search depth (without violating API terms and conditions)
- Store historical SERP data
- Web interface to display contextual charts and graphs based on filtered criteria
- Evaluate high ranking pages and upwards moving ranked pages to determine why they are so effective including comparing page rank, page titles, page summary, keyword density, META tags, keywords in headline tags and more commonly used SEO techniques.
- Track daily application usage (to prevent API ToS violations)
The picture above shows, in simplistic form, how the SERP data could be stored in the database. Eventually, by normalizing the tables outward, I can do further analysis on the pages as I parse the sites returned as the Googlebot would see them. I'm really excited about the potential for an application like this and the SEO implications. I'm sure many tools like this exist, but I hope to deliver the application and the resulting data in an efficient and unique user experience while also making it incredibly easy for someone to setup my application using their own API key and search terms.
Comments
SERP Tracking & SEO Analyzer
This sounds like a great tool. Do you have anything in beta for us to mess around with yet? I
I would love to find/build a tool that does reverse engineering of the SERPS. Have it tell us what terms we rank for as opposed to telling it to search for specific terms....I have seen simple ones that read your Meta tags and skim your site and then compile a list of "terms". Then it cross references those terms against the SERPs.
Interesting SERP Tracking Concept
@Prof SEO Agency -
Unfortunately, the tool that I built (that I have running flawlessly here in production at work) is no longer "supported" by Google as they stopped providing SOAP API keys. Therefore, you could see my source code, but without your already obtained SOAP API key, you wouldn't be able to execute any queries.
Thanks for an excellent idea. There would definitely be ways of doing this for sure - you could index all your site's META tags, look for keywords and phrases with high densities on your pages as well as page titles and then query Google to see if you rank for these terms. It's basically the same concept as my program instead you add an initial step of taking your site, analyzing it and coming up with a set of search terms automatically as opposed to through manual intervention. For more obscure phrases, you probably want to take a look at the statistics provided by your web server's referral logs and such tools such as Google Sitemaps which tells you your "average position" for a term. Your idea definitely gives my program the potential to have "longer legs".
Competing Tools
So, that didn't take long. I actually found Ranking Monitor which seems to do basically what I am trying to build. However, I feel it doesn't explore the full range of potential analysis angles and graphing options. Plus, I see the dreaded "Purchase" option which anything I build here wouldn't have. So, that being said, I'm going to press forward just for my own coding benefit.
Any Updates on the Ranking Monitor - DB?
I've been trying to wrap my head around actually developing ways of creating a dashboard or "ticker" for tracking rankings and associated traffic info. Because I work with a bunch of websites, this makes a lot of sense to me, keeping track og rankings, keywords and campaigns in general. Since my work is only as good as the data I work from and improve on, keeping my eyes on keywords and ranking positions is key.
I would love to explore ways in which this can be done without too much headache, and with a bit more accuracy. Thanks for posting about this very relevant and pertainent topic. Let me know if you've got any tips on ranking monitoring, etc.
Thanks again!
SEO Tool
Great to see that you are working on a tool for yourself. I would love to be able to get a copy when you are done. I do have a couple contributions that may be helpful. When developing this tool, combining metrics such as backlinks, PageRank and/or Webrank, as well as backlink text analysis could make for one powerful tool.
While staying on the path of a purely SERP tracking application - I would certainly make it as flexible as possible to allow for customization and scaling for whatever the search engines throw out at us.
Thanks, and keep in touch!
Do Not Fear ...
... I'm still working on this tool.
So far, I'm collecting data for certain search terms as well as site statistics including indexed pages and inbound links. You'd be surprised how much information I have already been able to gather just from these three pieces of information. Google SERPs for the term "hagrin" (not highly competitive) have been changing almost daily (top 10 results) which surprised me greatly.
My next step is to start working on the web interface to easily display and access these statistics. In addition, I want to take the Top 10 for a certain SERP and start cataloging certain criteria including search term/keyword density, META tags, URL, HTTP last modified date, Domain age, # of inbound vs. outbound links, etc. (all the SEO goodness that we have all come to love).
Development is slow with work, training for these ultramarathons, playing on 2 (sometimes 3) soccer teams and potentially starting Muay Thai classes a few days a week, but I'm still pretty excited about being able to create a free tool that maybe someone might use (in addition to myself). If anyone has any ideas as to what an app like this should and shouldn't do (that I haven't already mentioned), let me know and I'll try and add it at some point.
Owned by Google
So, as I was continuing development on this tool (I had refined the code to avoid errors, timeouts, etc.), Google has now removed the SOAP search API, the ability to get API keys and have basically crippled the reporting system I was building. Now, the tool still seems to be working, but I can definitely see the tool ceasing to function in the short-term. Instead, the SOAP search API has been replaced with a newer, more restrictive AJAX API - a move that can only be seen as attempting to stop the SEO professional.
Many may say that HTML scraping may become more popular, but then you run the risk of angering the Google gods as HTML scraping is a violation of the terms of service.
I'll be investigating the new AJAX API, but I highly doubt that the new API will satisfy the requirements I have.
Any Update on the SEO Tracking Software?
Hey,
I just wanted to follow up on your software. Any new updates with the API problems? Google makes it so difficult for us to track rankings, they should just bite the bullet and provide a product of their own to export rankings. I like within the Google Webmaster tools how it displays the terms that were typed in to "call your site" the most. So even if your site was ranked 250th for "software development guru" you know it is "relevant" because Google is already ranking you, and then you can optimize for that term and improve your rankings. Essentially you are reverse engineering your SEO efforts....
SEO Software
... the long answer is yes I am still working on it, but the short answer is I'm not really sure that, in it's current application, it makes much sense to continue development.
Since this tool centers around an API that is no longer being maintained by Google, it almost seems pointless. This is shown by the stats I gather for my own site which show the number of inbound links decreasing while Google Webmaster is staying fresh and seemingly more relevant.
Since I refuse to do any page scraping (violation of ToS), I'm a little bit at a loss as to how I want to proceed. There are definitely a lot of great things I could see done (and that I did including supplmenetal pages, ratios, etc.), but the Google SOAP API isn't the right tool for the job anymore. I'm very open to any suggestions you may have ...
HTML SEO Scraping
I tried HTML scraping, and it worked quite well, except that when page contents change, the scraper must be rebuilt - and this can be a great big waste of processing. It is not right on Google's behalf to limit SEO's but on the same token, they are a business.
I think if the data can be archived then scraped once it is in the posession of the data tool, that should still be within guidelines.
What do you think?
Thanks!