Here's a suggestion. Travel to where the time zone difference will be minimal, at least the first time. Adults can get cranky as they adjust to jet lag; imagine how that would be for a sensitive infant.

I think you should wait and assess the child's personality before making firm plans.