Never been "harassed", but have had to wait while the "security officer" before checkin (outside the US) checks every page (they usually give up after 10 or 15 pages of various stamps/visas), or the immigration "officer" looks over every stamp I have...

I had one poor bored dude stop me coming back into the US, check about 10 pages, and then ask me about my last trip to Japan... I often connect through Narita, but it has been years since I actually went into the country.... I looked down, and he was staring at a visa for PRC.... Oh well!