It's illegal for the government (US) to use the social security number as an identification number in records (except, I suppose as used for Social Security).
However, they *can* and in some cases are required to ask for it to ensure that you are not failing child support payments. For example, I heard that many states are now linking their drivers licenses to SSN's, which before was not done. This, from what I hear, was done to ensure that people who failed to pay child support would not be issued drivers licenses.
I think the same type of thing is for the passport, howver, maybe they are using the "child support" type of things as a reason to legally ask you for your SSN number... and then they use it or record it for whatever other purposes they want (hey, I'm not paranoid, but I also don't think that all national security gov't agencies follow regulations).
Anyway, I suspect that they accidentally put your SSN number in your passport instead of your passport number.
I'd give them a call and get that fixed, or else it may not be accepted if a foreign government sees the mismatch in numbers.