I like to look at it, not from the site owner's perspective, but from Google's perspective, TheDuke. They gain/lose advertising revenue based on numbers of users (ie advertising). If they start showing a bunch of lousy SERPs, their users will go to Yahoo or MSN.
Relevant backlinks likely indicate a relevant page for the SERP. Therefore, it's in Google's best interest to count relevant backlinks higher than those less relevant.
Google isn't in this for the small business, democracy, and apple pie. Google wants to make money--however possible.
The sooner you can get your site in-line with their model, the faster you can reap rewards.