The fact that he was talking on the phone isn't relevant. It wouldn't annoy me any less if he were having the same conversation with his seatmate.

I agree with bigred93

I would have made it quite obvious that I was taking notes on his conversation, especially if he was name dropping, and such.