I've always been friendly enough with my coworkers to have no fear of any of them compleining about me "always being on vacation" (because they all know I only take one per year), nor is it any big secret where I go (almost always to the same place unless I'm going to a family wedding or funeral), and it's usually pretty important for a bunch of them to know how long I'll be gone and when I'll be back, so that pretty much covers all of the bases.
As for getting called while I'm on vacation, that's pretty rare for me. At the last 2 companies where I've worked, there was a prevailing attitude - NOBODY calls me while I'm on vacation! I share that attitude, and those left behind always respected the sanctity of the vacation and simply assumed that anyone who was on vacation was unreachable (even though our cell numbers and personal emails were readily available).
The company where I worked before that had a different view, and I did get a few calls on my cell while I was on vacation. But that's what caller ID is for; I simply didn't answer.