DelRay Chris,
Say it like it is, brother! Yes, the Govt. per diem is not all that great. Still, we do get very good rates in most cases and this helps a bit. Still, yes there are times when we wind up in Motel 6's, etc.
Surprisingly, on the food you can pretty much get by on per diem. However, there are clearly no splurges or lobster dinners, etc. Your San Fran example was a good one, and there I get about $200 for the week for food, and spend about $400. -However, that is splurging and it makes sense that I should have to pay out of pocket if I get carried away and eat out at nice places. Point being that we do not have as much leeway as the open market, and I think that that is appropriate, actually.
As for the miles, yes, most agencies let you use them for upgrades and do not monitor. Using them for upgrades is a dang fine option, and I for one (and many others) do just that. Yes, many use them for personal reasons and it is one of those "don't ask, don't tell" type situations. Still, it is not worth risking getting your rear chewed for doing so; and first class is sweet anyway!