You will have to go through passport control anyhow as you enter the EU Schengen zone in Madrid, there won’t be any further passport control for your flight to Vienna nor in Vienna. Ask nicely to check your bags through in New York, your chances are 50/50.

Can be quite a walk and a little train from 4S to bagage reclaim which is in T4 (T4S being a satellite of T4). Bag delivery is often slow in MAD. From bagage reclaim to checkin is just one floor up. Queues at Y desks can be long, J is generally ok.
I would say you need 2 hours to be comfortable.