Welcome to Flyertalk!
You should be able to use the lounge in Terminal E without problem.
I am not yet FTL, but I flew GVA-ZRH-DUS (Schengen-only flights) in business class two weeks ago and had a 2-hour layover in ZRH. I went easily from Terminal A to Terminal E. No questions asked at passport control. The lounge agent just asked me "You know you'll have to go back for your flight?" and that was all.
Even the Swiss website mentions the possibility of using the lounge in Terminal E :
https://www.swiss.com/us/EN/fly/airp...a#t-page=pane2