Because when you state "I guess the larger point is this: . . . .," you are suggesting/implying that "the larger point" is shared by all.

If you wanted to say that "the larger point" was merely the point that you are trying to make, then you should have stated "I guess that my larger point is this: . . . ."

Which, from the above posts, is what you apparently thought that you said, but did not.