Look at it another way..

The casino hands you an envelope with 5K in it, and they say: inside this there are 15 $100 bills with a blue 'X' in the corner that you will have to send to the IRS.

You say thanks!

You then give the envelop to the dealer and say:

Inside this envelope are 15 $100 bills with a blue 'X' in the corner that you must give to the IRS as this is your Income Tax payment.

The IRS then receives a note from the Casino that they passed you the envelope, and the IRS asks you for 15 $100 bills with a blue 'X' in the corner.

You say, oh no, I gave them to the dealer, he is going to mail them to you lickity split... they say "oh ok cool, as long as we get our 15 bills with a blue 'X' " or they say, 'oh no sir, our tax code states that since you touched the envelope from the casino we want 15 from you and 15 from him?

Of course, they say the latter. Which is why I think this is unfair. Get it?