This is a common enough "problem" that SeatGuru often highlights as a "negative" that certain exit row seats "can be very cold".

Seems to me it would have to be some sort of seepage into the aircraft.

I've certainly noticed it many times, and I always try to grab a blanket in those airlines that still offer this amenity when I'm booked into the window/exit seat.