Depends. I can see it regarding the taxi on the Singapore side, this can speed things up. However on the JB side one needs to negotiate a flat rate with the cab driver - and a lot of the cabbies don't speak English. I also believe that JPO is outside the JB city zone - not sure what sort of surcharges will kick in for that. Taking the city bus to go to JPO (the JP01) should be fairly straightforward since this has nothing to do with going within the sterile zone or the Causeway back to Singapore.

I suppose it's a question of whether or not the OP is feeling adventurous. (OP - one thing. If you do this, I suggest you dress down. You don't want to look too well to do.)