I have never questioned anyone checking in wine. We have some medallions here in CVG that travel to SFO with two wine boxes empty, then come home with them full. They each hold 6 bottles. So I would say checking in twelve is fine. Just make sure the boxes are suitable for wine bottles.
I have also done alot of travel and have never been questioned when I brought wine back. It does sound like the agents were misquoting the 5liter rule.
Sorry I coudln't respond earlier. I had a hard time getting on this site yesterday.