I don't think so: there are a number of people who are actually not literate enough to be able to read English, even if they speak English. Maybe you can have a look at the statistics about reading abilities in your country. It can be surpringly high, even in first world countries.

I was also asked once while traveling if I could understand English, I do not find it offensive, I thought that the person who asked me just needed this information to know how to deal with me in this situation and adapt to my communication abilities.