In my experience (which is on United only), no. The agents may ask for the health certification, but otherwise they'll just want to quickly glance at the bag and then ask for your money. However, I would strongly recommend that you not take the cat in-cabin if he can't comfortably stand and turn around in the bag. He will get severely cramped and the trip will be miserable for him. Due to unforeseen circumstances I had to transport my previous cat in a too-small carrier once, and I would never, ever do it again. It is cruel to the cat.

I suspect they would notice, yes, but perhaps won't do anything. However, I again strongly advise against this. It's very difficult to stuff the bag under the seat in the first place, and getting it out is also a major pain, even if your coach seat has extra legroom (e.g. Economy Plus). Doing so will agitate the cat; based on my experience, the cat would much rather just stay down there the whole time rather than be taken up and down.

As for the potty, cats can hold it for many hours. He'll be fine on a transcon - it's only 5 or 6 hours. I don't think he'll require any sort of potty break. Moreover, most cats won't use the potty in unfamiliar surroundings - he'll be much more interested in investigating the weird smells, instead.

IMHO, you are worrying too much and trying too hard. First, make sure the cat can fit in the carrier. If not, I would strongly recommend not bringing the cat - it will be a miserable experience to be stuffed in a too-small bag. If the cat does fit, then I'd recommend just putting him under the seat and leaving him there throughout the flight - simpler is usually better. I've flown both cats and dogs (and do so every year), so I've got plenty of experience in this arena.

Good luck.