Thwe is a significant difference between socializing with colleagues as a recreational matter and socializing with clients because it is part of your business development obligation.
Nobody should be remotely concerned about accepting or declining the former. The latter is something which is part of your job and if it is the business practice in your space to spend a lot of time and money, that is just the way it is. Sometimes it pays to sound out the client. There are a lot of people who would rather go home to their kid and maybe talk business over a quick hamburger.
The good news is that more and more businesses are prohibiting their people from accepting "entertainment" and, for those doing this overseas, there may be local anti-bribery statutes as well as foreign statutes such as the US FCPA.