You are hereDoes Digg Belong in Google's Index?


Does Digg Belong in Google's Index?


By hagrin - Posted on 04 January 2007

I have to thank Search Engine Journal for posing one of the better questions so far of 2007 - does Digg belong in Google's index? (Actually, as you read the SEJ article, Allen Stern seems to have posed this question first.)

So, Does Digg belong in Google's search index?

First, a lot of people have weighed in on this topic since the initial people posed this subject and almost all of them are just plain wrong not because of where they sit on the issue, but more because the facts they used to support their arguments do not make sense or are completely false. What are some of the arguments for both sides and what are the misconceptions?

Pros

  • Helping Users Find Content - This would be the strongest argument for including Digg results within the Google index. Although many people seem to be incorrectly using the term "pagerank" (see here), the general idea is solid. Some pages on lesser authoritative sites (based on not only PR, but backlinks, keyword density, domain age, robots.txt exclusions, etc.) that hold the original content may get lost in the Google index and having the Digg result appear in the index improves the chance that the Google user will find the content he/she is looking to find. Generally, the rule states that you want to do anything that improves the user experience and helping users find the content they need should be the goal of any search engine.
  • Digg Mirroring - One of the greatest benefits of having the Digg version of a story appear in the search results is if a story disappears from the original site, very often the Digg comments will contain a link to a mirror of the original content keeping it alive past just the lifespan of the original website. However, would the average user know that? Obviously, no since most average users have never even clicked on the "Cached" link within the Google search results.
  • Don't Like It? Customize Google - Many people suggested using the -site: digg.com command with all your searches; however that's extremely inefficient unless you're using a Greasemonkey script. But why not just create your own Custom Search Engine and put Digg on your excluded sites list? The fix is easy and more people should really take advantage of the CSE offering from Google.

    Cons

  • Other Indices Do Not Exist within Google SERPs - Probably the most compelling argument for why Digg shouldn't be included in the Google search results is that other indicies, like Yahoo!'s search results do not appear in the Google index. This is obvious because search indices don't have "value added" or original content - they contain the page's title and a short description. Of course you're saying - but is Digg an index? Many will argue that yes, Digg is nothing more than an informative/popular index of links. Although there are no hard numbers to confirm this, it would appear that most Digg stories are submitted with the title and the description 100% copied from the original, linked site. Even if we could prove the previous statement, many would say that the comments associated to the Digg submission provides the original content to differentiate itself from other indices. However, the prevailing opinion of the Digg commenting system is so low that many, including myself, consider it broken, highly useless and completely inferior to similar sites like Slashdot.
  • I'm Tired of Clicking - Another popular argument seems to hold that the user experience is diminished because to actually reach the desired content the user is looking for by having to click on the Google search result and then the title on the Digg page. In addition, users unfamiliar with the Digg interface may not understand that the content actually exists after one more "hop". Although the Digg interface is similar to the Google interface (a blue link followed by a short description), I would like to see the functionality improved to something like Reddit's RSS feed where the direct link takes you to the original story and there is a "More" option to read through Reddit comments - something I almost never do.
  • Original Content < Digg Scraped Content? - As a webmaster, I see something inherently wrong with what amounts to no more than a user powered scraper site ranking higher in Google SERPs than the original content. However, it's important that this is not a deficiency of Digg, but more a "feature" of Google's search indexing algo.
  • Duplicate Content - Let's say that the original content URL and the Digg link both appear within the Top 10 results for a search term. How is duplicate content enhancing the user experience? Answer - it's not. There's little difference between this scenario and those generated by spam blogs.

    Misconceptions

  • Digg is more informative than most sites - The Digg fanboys will be all over this point, but the fact remains that many of the stories submitted to Digg are either blog spam, incorrect or written by authors looking to profit through their site. The reason why Digg attracts a high percentage of these types of sites is because of Digg's power - its massive, fanatical user base, the backlinks a promoted story will receive and the other benefits that relate to increasing your site's popularity. However, just because something is popular doesn't mean that it should be considered an "authoritative source" for certain topics. This isn't a problem specific to Digg, but to much of the Internet and its users - no one is sure exactly who they should trust.

    Conclusion
    So, if you've been reading carefully, you've noticed that I haven't taken a side. Where do I stand on the issue? Simple - create and use Google's Custom Search Engine option. Now, Google has to do a better job of promoting this highly valuable tool to "Joe Internet" because many of the usability issues deal with the average user and the CSE option isn't known by more than 1% of the Internet population I would gather (percentage not based on any facts, just perception). How could it be made more mainstream? If you've ever used a site like Match.com to look for hot steamy love, you can filter out your searches by eliminating certain people from continually showing up in search results with a simple click of an X. Google could implement something similar and save those preferences based on a user the same way they save search history, CSE optimizations, etc. Therefore, whether or not you agree or disagree with Digg's inclusion, you should know that you can put your own solution into action and determine Digg's influence in your Google searches.