While some might not have a problem with that, in this day and age where one mis-placed word can constitute a threat of terrorism or whatever, I'd rather not take the risk of being faced with assault charges against another passenger. If one or two "looks" or requests to stop are not honored, at that point it's time to involve the FA or even the captain. As a pax, you still have the right to bring things to the attention of the captain.