An interesting thread on WebmasterWorld forums discusses ways to diagnose a possible Google penalty. If you suspect your site / URL is being filtered out / penalized (e.g. you have lost some Google rankings), try doing the following:

  1. Dig your internal traffic analytics for traffic drops or bad trends;
  2. Perform [site:yoursite.com] search in Google to see if Google reports same number of indexed URLs;
  3. Check if Google Webmaster Tools report any problems;
  4. Take a look if Google toolbar PR has changed (graybar PR mightbe a signal of penalty);
  5. Check your site ranking for a “domain-name” search (without theTLD).

    6. If such a query puts you in the fifties, it’s almost (!) guaranteed to be a penalty.

  6. Check if Google has blacklisted your site as unsafe for browsing (type http://www.google.com/safebrowsing/diagnostic?site=mysite.com with your domain at the end).
  7. Check the source code for any injected links to obvious “bad neighborhood” sites;
  8. Check your other sites. In some cases Google seems to devalue the WHOIS domain owner and this results in a simultaneous ranking drop in several sites under the same domain account.
  9. Copy a long phrase from your page and search Google for the exact match - make sure this very page (not the home page, not other sites scraping yours) is ranked #1;
  10. Check if your URL can only be seen by adding &filter=0 to the search URL - or clicking on the “show omitted results” link (this means you were hit by the duplicate content penalty);
  11. Any other? Please share your way to diagnose a penalty.

searchenginejournal.com

And here’s a quick video featuring Google search quality team members explaining how to deal with the penalty without going into extremes