At least in the beginning of T4 there were some access-related rules preventing passengers from accessing with non-T4 boarding passes – someone with more current knowledge can tell what the situation is like today.

In any case, you'll be subject to an extra security check in T4 as the security check there is central and not at the gates.