I do this often - just be sure your conversations across the aisle are not so loud as to disturb other passengers around you (those in the aisle seats in front of/behind your seats), and if you are leaning toward each other while chatting, watch out for FAs walking up and down the aisles.