Sounds like the first agent had actually done everything and the second agent shouldn't have been scanning at all - just sending the bag down to the aircraft. The incorrect error message confused them I guess - but not sure why they didn't read the very clear timeline on the boarding pass and investigate further! Especially since judging from my occasional experience in a Ryanair check in queue this shuffling is done by at least half the people checking in.... normally they keep the boarding pass until you return to that same person for them to check your weight/payment is now OK

You could potentially try and pursue EU compensation from a denied boarding angle - could be tricky to prove it

Out of interest did you retain the original bag tags/labels? This would help in their investigation I suspect if they ever bother to do one