Project: NCAA Basketball Statistics Scraper & Handicapper

By hagrin - Posted on 29 November 2006

Since my SEO Tool has been successfully collecting data and I haven't decided how to display the information, I decided to start collecting data for another project. I remember reading that there was a discussion over the public domain of sports statistics and I personally believe that the data should be free and available to download. However, since it is not and data distribution sites want big money for such information, I did what any good programmer would do - develop a web scraper to parse available data, restucture it and hopefully release it freely to the public (this last statement is probably a no go, but I'll do the necessary research to figure out the potential copyright issues).

So, one would probably ask, once you have the data what do you plan on doing with it? To be honest, the only real use I see outside of eay-to-use public distribution would be to provide the public a "system" for handicapping future contests. Many sites like that post trends post statistics on an average basis which really misleads someone looking at the matchups. I believe that stronger trends exist when evaluating other factors not necessarily measured or taken into account as well as looking at standard deviations of data and potentially the median of such data.

I actually set myself a deadline for this one - January 1, 2007 so I would check back around then for the SQL scripts needed and hopefully the web interface that allows people to evaluate certain matchups (although as I write that, I'm thinking that it might just be better to automatically evaluate all the games for a day and then rank them - no need for user intervention). Obviously, I could track the accuracy of my predictions and tweak the formula as I see certain trends to improve the accuracy.

First PS3 Reviews

By hagrin - Posted on 21 November 2006

November 17th, 2006 has come and gone and the PlayStation 3 madness has started.

Ars Technica has posted their PS3 review and I definitely found a lot of their comments very interesting. Primarily, although the PS3 is rather large in size, it's freakishly quiet and definitely outshines the Xbox 360 in this department. Second, there was little innovation in the controller except for a complete Wii ripoff of motion sensing movement and gameplay (which they describe as "gimmicky" at best). Ars sums it all up best by saying "This is starting to be a theme for the PS3: some great features that are hindered by poor software support". Ouch.

Ars ended up giving the PS3 a 6/10 as an overall score.

Other reviews are out there and around, but this is the best review on the entire Internet. Trust me, I read almost all of them. I bet those who paid $3000 for a PS3 on Ebay opening day are wishing they had waited now.

PS3 Pre-Orders Being Cancelled at GameStop

By hagrin - Posted on 13 November 2006

On the heels of my last story, Joystiq is announcing that GameStop has given the bad news that they will not be able to fulfill their Playstation 3 pre-orders. As Joystiq accurately reported, even if you were one of the lucky 8-12 people who had their $100 accepted and received a pre-order receipt for a PS3, GameStop may not be able to honor your PlayStation 3 pre-order due to lower than expected initial unit numbers.

GameStop, in attempts to perform damage control, will call all pre-order holders and for thos unlucky enough to not get a console on release day, they will give you a free game or DVD (less than $20 in value however). Although JoyStiq is unwilling to place the blame (and probably rightly so as we don't have all the facts), I put the blame squarely on Sony overpromising retail stores and then lowering initial release numbers when they realized they couldn't meet their own expectations. Do you really need any more reasons to boycott Sony at this point?

Playstation 3 Shipments Lowered

By hagrin - Posted on 08 November 2006

Initial Playstation 3 shipments are being halved as Sony cannot meet their forceasted shipment predictions. This news comes right before their launch date, right before Thanksgiving and right before the lead-in for the holiday buying season. As Neowin reported, such major retail chains such as Best Buy have had to dramatically shift their marketing strategy to make up for the lack of PS3 inventory by pushing the Xbox 360 and Nintendo Wii to its customers. As many have heard me say, this announcement comes as no surprise to me as Sony has been disappointing in all business areas this last year - enoughso that I have officially stopped purchasing Sony products.

However, Ebay sellers are rejoicing today. As you can imagine, with shorter supplies higher prices follow and those who have their pre-orders filled will be able to turn around and sell the console for a hefty markup. If you're lucky enough to get a launch date PS3, sell it - you'd be fiscally irresponsible not to sell.

Want to Lose Weight? Become Poor

By hagrin - Posted on 06 November 2006

Over the last year or so, I have managed to lose 50 pounds and become one of the more active people I know. When people who see me now, who haven't seen me in a long time, they always ask "How did you do it?". Books, professions, websites and diets have all tried to answer this question in hopes that they would be able to solve one of life's great social problems. My solution, however, didn't come from any book or training manual, but from a lifestyle change. What was that lifestyle change?

I lived "poorly".

The above declaration is intentionally misleading (in a lame attempt to capture my audience's attention). However, the statement does hold some truth when put in the context of my spending habits and my ability to "rough it". First, let's flashback to October 2005. I had just come off my second broken foot in less than a year and had not been able to do much physical activity. My weight at that time was 235 (although many say I never looked that heavy) and I had taken a "I don't care" approach to my body thinking I still looked OK. However, inadvertently, I heard someone close to me discuss my appearance with their boss and found out that they found me "unattractive and fat". Yikes, the F word - a word that still stings even to this day. After hearing the assessment on my body, I looked in my full-length mirror and didn't like what I saw.

That night, I ran 8 miles at the gym.

I couldn't walk up or down stairs for the next couple of days, but continued to crank out about 30 minutes of running a night at the gym. The pain persisted, but the pain drove me to continue my running quest and I started lifting light weights at high repititions. These were humble beginnings, but they were the foundation of who I am today.

The problem with this recollection as an advice story for others is that the anger of being called fat only lasts for so long and doesn't produce the lifestyle change that is needed to break bad habits and institute newer, better routines. So exactly what clicked inside of me that made this change a permanent one that still drives me over a year later?

Live like you're poor.

So, how does living poor exactly translate to losing weight? Here's a few tips I used -

  1. Leave your wallet at home when you go to the gym. You may leave the gym starving, but if you bring your wallet with you, you will be more apt to make poor impulse food buying decisions. Plus, you'll save money by not being able to spend any.
  2. Bring your lunch to work. Eating out (umm ...) is expensive so bringing your lunch will help you save. In addition, if you constantly eat out, you will be presented with more unhealthy food options than you would if you brought a planned lunch.
  3. Walk or ride everywhere. When I decided to ride my bike and take the train to work instead of driving, I was able to lose weight while saving money on gas and car maintenance (my commute is 50+ miles each way). Cars are very expensive to maintain no less purchase or lease.
  4. Forget the "not-so" convienant stores. Convienant stores are typically more expensive than supermarkets and definitely stock more unhealthy food than a supermarket would. Limit your shopping to supermarkets - try going before work when there aren't that many other customers.
  5. Plan cheap dates. Yikes, well maybe you shouldn't take dating advice from a single 28 year old. However, instead of going out to each rich foods for a dinner date, take that romantic walk in the park, walk the boardwalk at the beach and stay active. Trust me, you'll want to be with someone who likes being active.

Good luck with your weight loss activities! I'd love to hear some more tips or words of advice.

Hiring Programmers Today

By hagrin - Posted on 02 November 2006

This morning I read a very insightful blog post over at Mini-Microsoft concerning the trouble hiring talented programmers and a potential solution. Mini MSFT draws the picture of the accomplished programmer who has a strong resume, strong background, good experience, but then fails the simple coding problem during the "whiteboard" test (for those who don't know, the whiteboard test basically entails coding out your answer on a whiteboard as opposed to a computer). Mini then questions if you're not going to take the chance on hiring this candidate and your talent pool all seems to come up short in some way, how can you strengthen your potential employee base and offers up the idea of a bootcamp like training called "Microsoft Academy".

This is an interesting dilemma and a problem that directly faces me as I progress through the technical side of my career.

(Note: the below stereotypes don't apply to everyone and skill sets and aptitude will vary on an individual basis, not on a "how did you learn" basis.)

I'm self-taught. I've never taken a computer class, never completed a training course or seminar and have learned from reading books, reading the web, Google code searches, code samples and asking other programmers I worked with (although now, I work alone and don't have any other programmers to turn to). With 5+ years experience now, I have run into most common corporate environment related programming tasks and can expertly navigate my way through the software development process. However, although I was able to eventually develop an efficient solution to a problem, each new hurdle required research and trial and error testing.

So, wrapping up Mini's post with my own situation into one neat package, how does the current hiring process by many software firms actually relate to discovering and hiring the best programming talent? Does the firm's ability to groom talent play a role in who you can and cannot hire?

Taking a critical look at my own skill set, I came up with the following list of Pros and Cons of someone who is slef-taught and who may not do well on a whiteboard interview examination due to inexperience with solving a particular problem -


  • Being self-taught shows the ability to pick up new skills outside of the classroom and on the programmer's own time. This ability relates more to what occurs in the real world life of a programmer when technology is highly dynamic and not static like shool textbooks.
  • Shows a level of self-motivation to be able to learn a craft outside of formal education.
  • Won't require the same teaching overhead that others who don't learn well on their own need.
  • Quick absorption of new material and technology released


  • Self-taught developers very often figure out "a" way of getting something to work and don't always know the most efficient coding path to solve a problem. I know I have run into this situation numerous times during code reviews later on after my development is done.
  • Time wasted on research and trial and error for topics and problems that many formally schooled developers would have tackled as part of their cirriculum.
  • May lack fundamental concepts in programming which prevent implementing technologies that are not documented well (an example of this for me was developing sophisticated web parts for SharePoint 2003)

With all the above, should you hire a programmer that has proven himself in other environments, but fails the whiteboard test? The answer is maybe (what a cop out). If your organization can take an individual an provide him ample time to "get up to speed" on a subject, the self-taught, whiteboard failing developer might be the best fit for your organization. If you need contract work done, you probably want to hire the developer who has direct experience with the solution you're looking to implement because firms hiring contract workers do not have the time to groom talented individuals.

In conclusion, if you can groom talent you can be more varied in your hiring selections and potentially garner greater rewards. Otherwise, if you're limited by budget and/or time constraints, your hiring pool will be much smaller and limited to those who possess a mastery in a certain skill set. For me personally, since I am self-taught, the company that takes a slight risk in me and gives the time to acclimate will find themselves with a highly versatile, highly motivated programmer who is always learning.

Would your company/department take the risk on hiring someone with my background? Let me know - that's what the comments section is used for.

Google Talk - Offline Messaging Added

By hagrin - Posted on 01 November 2006

Finally, they added it!

Google Talk became the best IMing program in the world today when they added support for offline messaging. Now, if you send a message to a user who is offline, that user will receive the message when they sign online again or check their Gmail (the best feature). When users check their Gmail accounts, chats will show up as unread messages (how cool is that?) and intergrates amazingly well with the chat logging integration between Gmail and GTalk. All you need to do is make sure you have chat logging enabled and you'll be receiving offline messages in no time! A wonderful addition by Google which leverages the power of Gmail and GTalk.

Now, I need to stop being lazy and hookup Google Talk for my Blackberry 7290.

Google Data APIs Now for PHP 5

By hagrin - Posted on 01 November 2006

The Google Code blog announced the availability of Google Data API support using PHP 5 through the Zend Framework. Zend, who I know better for making one of the best PHP IDEs around, has developed their Google Data Client Library which "provides a PHP 5 component to execute queries and commands against Google Data APIs from your PHP applications".

Now, I just need to get a webshost who will upgrade to PHP 5 (not to mention, update their version of python).

Google Offers Custom Search Engine

By hagrin - Posted on 24 October 2006

Google has finally announced their custom search engine tool which allows users to build their own search engiens and then customize the search results. This tool allows users to specify search engine results to meet the requests of users looking for specific topic related results.

What does this mean for people who use Google now? Honestly, not much. The real world applications I see are that websites will use Google's custom search engine to power search results on their own site; however, they will tailor results to push their content to users. To be honest, although it makes search easier, searchers using the appropriate search operators could yield similar, if not more objective results. Although Google allows for social interaction in developing customized search results, I can't see how allowing anyone to develop an offshoot search engine will ever create the type of community that could really drive more accurate, efficient results.

I will tell you what this means though in the near future - portal scraper sites powered by Google setup by the average user. How so you ask? Google allows user sto setup custom search engines, but also add their Google AdSense code to the results. Knowing that, webmasters can create specialty topic sites, create a custom search engine, add their AdSense code and then profit without creating any unique content (edit - as I have been reading today, no API is needed since webmasters can use an IFRAME and then their site's CSS to stylize and seemlessly integrate customized search results).

Before anyone tries to hammer me over the definition of a scraper site (displaying other people's content as your own) and they say that SERPs aren't really scraping, I would argue that Google is the ultimate scraper site and this customized search engine only enables the same business model on a much smaller scale. Think I'm off my rocker? Well, read Matt Cutts' proposals for creating industry leading niche search engines. He talks about how someone could be first-to-market, create an industry approved and leading search engine and then profit share the ad revenue with Google meanwhile Google nor the webmaster has created any unique content while displaying targeted advertisements.

All in all, it's good to see Google opening up their search results to the public to be modified and fine tuned, but we'll see this application's real impact over the long-term.

Malware Tools

By hagrin - Posted on 23 October 2006

The security community came together and SANS has posted this comprehensive list of tools to analyze and combat malware. This is a fantastic post and hopefully one that gets around the net to all system and netowrk administrators out there dealing with malware infections of all kinds. The tools on the list are mostly freeware or shareware and almost all are for Windows machines (since malware is only seriously prevelant on Windows networks and machines - although some of the tools are cross-platform). If you run a network or even if you want to protect your home PCs, make sure to check out this list of tools.