No that doesnt make you a plonker, many of us can probably recite the safety briefing but I always put my paper down or whatever I am doing more out of courtesy for the crew.

I'm sure the crew always watch who pays attention and IMHO have had great service because of it, so I will continue spare a minute.

I would be interested to hear any crews thoughts as to whether they differentiate or not, of course they might even do it without thinking but I am sure they are watching during the briefing.