In theory, you should get your Schengen visa from the country that will be your "main destination", or where you will spend the majority of the time. In practice, most likely nobody will care, once you enter Schengen, you can freely move around. If you want to be safe about it, make your Poland stay longer than your Latvia stay