If I understand you correctly, you will transfer from the NEV-IST flight to the IST-VIE flight. As the incoming flight is domestic, you will arrive in the domestic terminal and will have to go to the international terminal for the next flight. While it's only a short distance, it means that you will have to "go outside" in any case; specifically, you will have to go through security and immigration at IST.
In principle, TK should be able to check your bags through to Vienna, assuming that the 80 minutes you have is above the minimum connection time (I don't know what the MCT for domestic->international is). But I think that 80 minutes is a very short connection time, and since you're on two separate tickets, the risk is, in my eyes, not worth it. Any slight delay of the domestic flight and you lost your international ticket as a no-show. I would definitely leave more time.
Shuly