An airline can never be sure it can fly any passenger, ever. Weather can make the flight too dangerous, etc.

Sort of. There's a clear economic benefit to allowing airlines to oversell flights and take advantage of the fact that sometimes people don't show up when they have reserved seats: it lowers ticket prices for everyone and contributes to Delta's profit margin (the relative amount to each depends on supply and demand -- specifically the elasticities of supply and demand). IDB compensation is a method by which the costs and benefits of allowing oversales are placed on one actor (Delta in this case) so that that actor's choices to maximize it's own profit will (hopefully) simultaneously maximize societal value as well.

I say hopefully because the logic requires that the IDB compensation be an accurate transfer payment of the loss suffered by passengers denied boarding. In the end, it seems to be fairly reasonable. Obviously in some cases it is clearly too low, but in others it is too high. Given the number of people on these boards who relish receiving VDBs, I'd say there are plenty of people being "overpaid" for their denied boarding experiences and will freely admit that many cases exist where people are underpaid. In the end, it seems about right. And having a simple rule offers a lot of benefits...measuring actual damages would be impossible anyway.

The problem with weight and balance oversales is that they're not intentional on the airline's part. The airline isn't really deriving an economic benefit from them, so forcing them to pick up the tab for the flip-side of the coin is a bit questionable. With bigger aircraft, as I said, you can make the case that it actually is intentional: they have many more levers to pull to get the aircraft within weight and balance (off-load cargo, etc. as I mentioned before). With RJs, there's much less wiggle room. They're not flying cargo (at least, nothing substantial), and all RJs have similar operating envelopes, so they can't easily put a different aircraft on the route or something. (Only option would be to go with mainline aircraft...and then you really would see decreased service, at least decreased frequencies.)

And as to your first point, I do agree that there's some point at which an airline shouldn't be selling a ticket for a flight. If the plane could only take off with a full load of passengers if the weather were unseasonably cold with a 20 knot wind right down the runway, I don't think they should be able to use "weight and balance" as an excuse to IDB passengers without compensation. And I have heard of cases (though, not that rediculous) where I thought they were over the line.

And don't think I'm an airline apologist....I helped my friend argue Delta into paying him the $400 they owed him when he was IDBed in MEX due to "hot and high" weight restrictions. Their local staff told him he was owed squat because it was weather related. I set him straight and made him email Delta and not take "no" for an answer.