A nested ticket is as follows:

You book a return ticket from A to B and back (Say out on Tuesday returning following Tuesday). Some tickets (e.g. CE R fares require a Saturday night stay, which this booking complies with). However, you need to come back from B on Wednesday and fly back to B on Sunday, so you book a B to A return ticket for those days. Again this is fine for the Saturday stay requirement. You now have nested tickets: A-B-(B-A-B)-A.

You fly out on the first ticket and then back on the second, without staying a saturday night, then you fly back on the second ticket and return on the first ticket, again avoiding a Saturday stay requirement. The issue (point that the ailines try to prevent with their T&Cs) is that if you do not stay over a Saturday night, the fare would be higher, thus nested tickets are avoiding higher fares in the airlines' view contrary to their ticketing rules.

I hope this explains it.