I've often wondered the same thing. I would do my best to never get "argumentative" with the agent if presented with a similar situation (thus far, the closest it has gotten has been uncomfortable questions about any arrests, etc.) but I would like to know to what extent I can 1. Stand my ground or 2. Shut up and just refuse to answer any questions that aren't directly related to establishing my citizenship for purposes of entry.