Just to clarify, what makes a country "difficult to visit"?
In my mind, it takes several conditions for that:
1) Need for a Visa before arrival (in the case of Bhutan, Visas are not difficult to get, the hotel is actually taking care of that and I won't even need to visit Bhutan consular services or mail my passport - I will simply present a confirmation and get my Visa on arrival..)
2) Long journey (Australia is far, but flying there is pretty straight forward and one can basically get there from EUrope or USA in 24hrs...). Some countries are only accessible from a few airports, and connections are such that one must spend a day at least waiting for an early flight...
3) Cost of Travel: The more expensive a country, the more difficult it is to visit
4) Security Concerns/Political Turmoil/War: In most cases, this is just temprary. For instance, I wouldn't travel to Syria now, but in maybe a year or two, things will have settled down.. But Somalia has been having trouble for a loooong time!
Countries that in my mind are most difficult:
a) North Korea: Visa Required (and takes some time to get), Must travel through Beijing, so must obtain Chinese Visa. Little flexibility regarding dates and extra stop in Beijing, though probably very pleasant, makes it expensive and longer...
b) Somalia: Few flights, Visa required, huge safety concern
c) Tiny Pacific Island: Visa not necessary, but must fly to Australia, flights are not scheduled very often, so it will take time and cost money
d) Haiti: Easy to get to, but unsafe, politically unstable, and very poor infrastructure since the earthquake (getting to the safe part of Haiti where cruise shipds stop doesn't count as a Haiti visit!)
e) Antartica: No Visa, but if you want to sleep on site, the few tour operators are very expensive...
f) Iraq: (Not the Kurdish part, but Baghdad) requires visa.. Easy to get to from Europe but requires "balls" :-)
g) Yemen: Visa, Safety concerns, Political Stability issues
h) Western Sahara: Not a country yet (or maybe. depends on your definition)...
Finally, to expand on what has been said, KSA issues a 5yr Multiple Entry Visa for US Citizens. Obviously, one needs an Invite, but it is not that difficult to obtain. Just look through your network and you'll probably know someone who has a good relationship with a Saudi that can issue the invite
Belarus: again, Visa is very easy to obtain. The hotel can issue the invitation so getting a Visa is just a matter of making a request at any consular service with the appropriate invitation