Sounds pretty dodgy to me. Did you end up using a toll tunnel or bridge. Hopefully not (most likely Queensboro/59th St) as it would have been an extra charge. I would have guessed without toll, the taxi would have been about $25-$28 depending on traffic.
If you wish to make a complaint, you would most likely be aided by the taxi #. Its the medallion on the taxi sign, as well as should be posted (with the driver's picture) inside the car on the plastic partition (usually behind the driver). Complaints should be filed at
311 TLC complaints website.
I hate that taxis (and likely this doorman) that take advantage of visitors to the city. Leaves a bad impression on the people who come to the city!
In the future, I would just grab the first in the queue, take note of the driver's taxi # (and name if possible).