512kbps would be more than enough in theory. In practice, what is more relevant is how the traffic is prioritised, so that other (non-VoIP traffic) cannot flood the available capacity and cause the quality to drop dramatically.

What sort of router are you using?