Walking to Ministry of Crab, you will probably see a couple of touting tuk tuk drivers. I would just point and say, "I'm going over there" across the street. I never thought anything of it. I found everyone in Colombo to be very nice. The scam people aren't anything to worry about in my opinion. Just keep walking. We took a tuk tuk tour in the late afternoon/evening when it cooled down and that was a really great way to see the city.