You can definitely file a complaint. It probably won't result in much, but it may make you feel better. If you come back again with the proper documentation, I don't think you will have any problem. You can also apply for a visa in advance, which gives you the right to appeal to an immigration judge if you are denied entry. And yes, I believe you can apply for Nexus.
If it helps any, you're not alone here. This kind of treatment is not unusual from CBP, and happens to US citizens as well.