Haven’t flown AirAsia too many times, but once on the same flight I saw many clearly oversized bags squished in AND a roller bag removed by cabin crew for being oversized (and the passenger charged on the spot).
So basically, anything could happen. My guess is the more your bag looks like it fits (soft, backpack, fits perpendicular) the less likely you are to be challenged.