The train might get you to Kyoto a little sooner, but flying is cheaper. I think it comes down to what you are more comfortable with. Since you appear to have reservations about traveling without reservations, I suggest you fly.

17:30 flight with ANA is your best option - You have enough time for the connection. After immigration and customs, make your way to the domestic area of the terminal to check bags and go through security (T1 South). Don't go through security too early. There is little to do in the domestic gate area.