I'm in a similar situation - I'm a non-Schengen citizen, but with a Schengen residence card. Most airports I've been to stamp me in and out of the Schengen zone, but I've also noticed that Frankfurt never does, once I show them my card.

The initial interaction always begins with me handing them my passport. They then flip through the book, looking at the ~100 Schengen stamps, trying to figure out if I've somehow over-stayed during what appear to be multiple "visits" to Europe. Once I get the plastic residence card out of my wallet a few seconds later, they quit looking at the stamps and just wave me through.

So ultimately, as long as you don't lose your residence card, the stamps appear to be unimportant.