I was asked for my passport # for the first time last night, after about a dozen or so transactions on my card in the last week. I made a couple of S/.150 charges in Lima and they didn't bat an eye or ask for ID, but S/.50 at a hole in the wall in Cuzco warranted a doc #. Doesn't seem to be any rhyme or reason, other than perhaps the comfort level of the merchant on a chargeback.
I, like mikew99, don't carry my passport book around with me, and thusfar it hasn't been a problem.