Welcome to FT! I like your enthusiasm and hope it's the start of a life-long love of traveling! Here's my two cents based on your questions/comments which I've bolded

1. What are the best countries in your opinion that aren't too costly and why?

Well this depends to some extent on your hobbies, likes, dislikes, etc. For someone staying in hostels or couchsurfing, backpacking around, etc. a lot more countries can be visited cheaply. Whereas someone who only wants a certain level of hotel and amenities could find almost every country to be costly.

While not perfect or 100% up to date, websites like Lonely Planet typically have a rundown of average daily costs in their country and city specific information. I wouldn't take those as ironclad truth, but it will give you a general idea of the magnitude of cost differences.

Many online travel "magazines" and blogs run articles on cheap destinations or how to do country X economically. Again I wouldn't plan a trip on those articles as they tend to be generalized, but it can give you a starting point for further research at least. Just try googling some key phrases and you'll find some stuff.

Exchange rates obviously factor into the equation, which isn't the greatest news for those of us coming from the U.S. in many cases lately. While an unfavorable exchange rate doesn't mean you can't find ways to visit said country affordably, it does raise the bar. (Likewise a favorable exchange rate doesn't always mean that prices won't still be high and partially negate your exchange savings.) Sites like XE.com and others are good for researching that.

2. In connection with the first question, what is a good amount to take with me if I go for 2-3 weeks?

Little to none, as mentioned upthread, in most cases. Use your ATM card to withdraw when you get there.

3. How long in advance should you buy the plane ticket(s) to get a good deal?
I tend to travel more on award flights so I don't have a real good general answer for that. It does depend on the particular routing and time of year, too. I'm sure others with more knowledge can chime in. I do know that the question has come up many times in the context of various destinations, so if you search around a bit on FT, you'll likely find some answers inputs on that area. Try the destination specific forums, maybe.

The budget thing isn't quite known yet. Apparently after I come back from training (whole year) I will have around $20k+ saved up. My starting budget should be quite nice but I don't want to spend that all. I guess the budget could be $4k tops per trip

I assume you'll have a day job besides your reserve duty? Otherwise your $20K won't last very long even for your day-to-day living, not to mention traveling. Plus having vacation time and ability to use it - variable depending on your job. I'm not saying not to travel, but just plan carefully and pace yourself as necessary.

Spend some time all over FT - there's lots to learn about airlines, hotels, destinations, trip reports, even a budget travel forum. And of course other more general travel resources out there, too.