According to the last CBP guy that checked my car, the fact that I was merely driving south (away from the border, as is everyone at the checkpoint) was probable cause.

For that matter, I got delayed once by 45 minutes since I refused to tell them where I was going other than "south", since, as far as I was concerned, once I had showed them my valid US passport there was no immigration reason for them to question or detain me.