Generally, it's the CPU that does the decoding. As you have found out, the Pentium 4 2GHz is inadequate. It might be able to squeak by when decoding 720p h.264, but 1080p h.264 is definitely out of the question.

When the stars of OS, video card, driver and codec align, it is possible to offload some or all of the decoding task to the video card. AFAIK, right now the development focus is on Windows XP and Vista through the DXVA framework. I don't know if it works on Linux yet.

I think 512MB RAM is adequate, provided that you close other applications during playback. If your software is using CPU to do the decoding, then you will get most benefit from a CPU/motherboard upgrade. In my experience, an AMD 2.5 GHz dual core or Intel 2 GHz dual core is fast enough to decode 1080p h.264 using the free and popular but inefficient ffdshow/ffmpeg codec.