I know what you mean. I think this is what happens when they have been washed too many times. Sometimes, it is the first thing I check when I sit down and I quickly switch with my neighbor before I sit down. (assuming that one is better)
One time it was so bad, because all the "cotton balls" were on one side, I asked the FA if she had a better pillow and she actually found one.

These days, with polaris, I ask for the gel pillow and stick it on top of that pillow so it sort of solves the problem.