I'm guess for the USVI, its all considered a duty free port, so you need to clear customs, but not immigration?

This would seem to indicate that while US citizens don't require a passport for travel between the 50 states and Puerto Rico, a foreigner would:

http://travel.state.gov/travel/cbpmc/cbpmc_2223.html