The bottom line is that you are not entitled access based on your AA status if flying EI and the agent was entirely correct. This will change if EI join OW but that still does not seem to be on the cards in the near future.

As you rightly say, you can just stick to AA flights if you want your lounge access, or of course fly with other OW airlines with a stop. The fact that you'll limit yourself to AA flights is completely irrelevant to EI or rather, they will benefit because AA will pay them for your lounge usage, and in fact, since that self imposed restrictions may mean you may occasionally miss out on the cheapest available fare and pay a bit extra to be on that AA flight, EI will pragmatically benefit even more.