You will need an alien registration card (外国人登録) or a certificate on registered matters (記載事項証明書), which you can get the same day as your alien registration card application.

As far as I know, most banks will not open accounts for those whose status is that of a "temporary visitor".

e.g. Shinsei Bank will not: http://www.shinseibank.com/english/powerflex/oa_fc.html

So -- in summary -- both of your sources are partly correct. Opening a bank account requires certain paperwork from the city office, and it is possible to get that paperwork done as a temporary visitor. However, banks will not accept the paperwork if it has been issued for a temporary visitor.