I will always swap with someone when they ask; however, it is annoying when the swap is to the first row because there is on underseat storage.
I would ask the person in the first row to move back to the second row instead of asking someone from the second row to move up to the first row. I think you will have better chances of swaping if you do it that way.
I know in your case you did ask the person in the first row (the old guy). But it sounded as if you asked him first.