The overnight delivery service provider that you use to send your passport application to the Department of State must be capable of delivery to a P.O. Box
http://travel.state.gov/passport/get/fees/fees_837.html
That doesn't mean they are going to refuse delivery if you get the right street address, but the statement above is pretty clear.