If you apply for a Polish visa, Poland should be your main destination or both countries should be your main destination, but you can enter anywhere. If Latvia is your main destination then really you should apply for a Latvian visa. Exit doesn't really matter as long as the exit is recorded properly.

There is a possibility Poland will refuse the visa if you present travel plans showing your will spend all the time in Latvia. However you can "change" your travel plans after getting the visa (or to be safest, after you have been admitted into Schengen). Once you are in Schengen nobody will require you to stay in any particular country (unless you are put into an extremely rare situation where your visa is issued only for one country).