Obviously, in proximity to an exit, the a/c's insulation between cabin and fuselage skin is disrupted by the exit and its fittings, allowing the area to be potentially "colder" than nearby seats (but not much). Rarely have I even noted the interior Plexiglas window being much colder, but it's obviously a few degrees cooler than the cosmetic interior covering. As for airflow from the exit sealing and gasket, "No Juey, Jose!". Give or take a few degrees, it's -50F below out there, and a little viewing of the "gear" worn by WWII bomber air gunners, at lower, warmer altitudes will give you some idea as to what real "cold" might seem.

What you likely felt, given some separation of interior panels, was airflow from a leak, crack or split in one of the cabin air ducts, generally a good bit cooler, as evidenced by the temperature of the air from the overhead "nozzle" above your seat, as yet unwarmed by all that human use, adding heat and humidity at every breath.