I don't see how you could possibly be denied entry into T4 if you're actually departing from there -- regardless of what other terminals you visited earlier in the day.
Edited to add: It's a moot point, as the new Capital One JFK lounge is not in T5: it's in T4!
https://viewfromthewing.com/first-lo...ne-lounge-jfk/
("First Look Inside Capital One’s New Flagship JFK Lounge Opening 24/7: Fresh Ess-a-Bagels, Murray’s Cheese Experiences & Meals On Demand")