Saint Antoine is part museum part hotel, so to describe it as "modern luxury" is basically false.
Ritz Carlton Montreal has the best hotel rooms... except at the top suite level where the edge goes to St James. Ritz F&B is excellent and frequented by locals. Not too stuffy. Location is very different between the two. The various boutique options discussed in the thread jerry.a.laska linked are also worth a look.