The rule of thumb is, you can photograph anything outside (temple/shrine buildings, outdoor statues, gardens etc), but not inside any temple/shrine buildings with religious altars, and you'll usually see signs to this effect inside them. Not only is the light damaging to artworks, but it's disrespectful to any worshippers (& staff) if there are distracting flashes going off indoors. Museums usually prohibit flash photography much for the same reason, although non-flash photos may be allowed.

When in doubt, ask. No Japanese needed, pointing to your camera will suffice.