My wife got a scolding from Canadian immigration a couple of weeks ago for traveling with our baby but without me (or a letter). I admire her for asking the agent point blank if he was going to refuse her entry. He didn't but I'm sure she now has a red mark on her permanent record.
Mike