I usually leave the tags on clothes as long as it doesn't itch....the only ones that come off are the ones sewn into the side of shirts, they always seem it itch. I take the tags off of everything else - washcloths, towels, blankets, scarves, gloves, etc.
I even go so far as to remove all of sticker price tags from body washes, soap and shampoos because I hate when they partially peel off when wet.