OK I FIGURED IT OUT - for those of us who live in APARTMENTS!
That was the problem!
So, this is what Barclays does: You give them your home address, like 1234 Main St, Apt 999.
Barclay's system immediately ignores USPS convention and every other source of logic, and they put the apartment on the FIRST LINE instead of the second line like they are supposed to, and put your street address on the second line instead of the first line. Basically, their system looks for anyone with an address at an apartment, and swaps them, contrary to USPS convention.
So, in their AVS (Address Verification System) your address is saved as this:
Apt 999
1234 Main St
So, when Amex goes to authorize it, the AVS comes back as "no match."
So, first I went into my Barclay's profile, and fixed my address: I put the street address first, the apartment second, and saved it.
By the next day Barclay's had swapped them back!
So I went in and tried to fix it again. Then I went to Serve and tried to authorize it immediately. Still wouldn't work - because although its fixed in your online account, it is STILL not fixed in their AVS system - which is probably updated once a day, AFTER they "fix" your address!
Now, I run a business and have a merchant account, so I started running my card for $1 to see what was going on. First I tried with my address "1234 Main St" and was DECLINED. Then I tried with my address as "Apt 999" and was APPROVED.
So I realized their AVS is not updated. So I go to Serve, and try "Apt 999".
FAIL. I have no idea why.
So THEN I try, and this is insane, just "Apt".
Yes, when it asked for address, I put:
APT
And bingo, Serve approves me.
What a crap company Barclay's is.
TL;DR: if you live in apartment and try to load your Barclay's card to Serve, go to your Profile in Barclay's, look at ONLY THE VERY FIRST NUMBER OR WORD THEY HAVE for your address, and ONLY put that in - nothing more, nothing less. Do not try to fix it, do not try to change it, do not put anything else in. For street address you put the first "word" or "number" on your street address in the profile as barclay's tells you they are. Barclay's word is god. Do not attempt to correct them.