Fusion 2.0 supports 8GB per VM, I have not tried to allocate that to a VM when not having the physical memory. VMWare Workstation (PC side) DOES allow the VM to page memory to disk if you allocate a size greater, but it can be dog slow and cause thrasing on the host due to limited physical memory.

I'm assuming you don't have Fusion 2.0?