Agreed. Even if you're "visiting" the cabin, you're enjoying the benefits of a class you did not pay for, earn, or were comped. Regardless of whether it is mealtime or if the "guest" consumes anything, being in the cabin for anything other than a couple of minutes for a valid reason means they are still enjoying the comfort of the premium cabin.

Further, given that you've simply described this person as a "colleague", what is it that you have to talk about that you need to trek from one cabin to another during a flight that you couldn't chat about in the gate area, baggage claim, or in a post-flight e-mail? Why not have your colleague visit you in your cabin? If it is something you might think of mid-flight, then send a note via FA.

The only thing that burns me up worse than friends/colleagues coming to visit are the families traveling in different class cabins (i.e. dad and teen in coach, and mom & baby in C of F -- adult child in E- or older parent in F), and the family members in coach feel that they have the right to just come up and "sit with" their family in the higher classes. I have gone off on that many times, and each time, the person was redirected back to their seat by the crew, if they didn't leave when I asked them to. Hey, if you can't even use the bathroom in a different cabin, you sure shouldn't be able to occupy any other type of seat in that compartment either.