I would think some suspicious immigrations officers, not just in the USA but around the world, may ask you what used to be in place of the torn VISA sticker and if you don't provide an answer he/she sees fit, you will be asked to go into the little room.
I would just leave it alone.