In general a bag connected to an oxygen delivery device, be it a mask or cannula serves as an external reservoir of oxygen. For example, with a simple nasal cannula your nasal cavity serves as the oxygen reservoir. Even when you're not taking a breath in the cannula delivers new oxygen to your nasal cavity. When you breathe in, room air (21% oxygen at sea level) is entrained along with whatever concentration of oxygen you have in your nasal cavity (>21% oxygen) and delivered to your lungs.

You can move this reservoir externally as well by using a mask over your face (the space between the mask and your face plus your nasal/oral cavity become the reservoir) or a mask with a bag which adds an external reservoir. The bag on the aircraft oxygen mask is simply providing a reservoir of 100% oxygen that you can then breathe in. If you breathe in faster than the oxygen delivery device is delivering oxygen the bag will not inflate since you're drawing in all of the delivered oxygen plus ambient air entrained around the mask edges. If you're breathing slower than oxygen is delivered to your bag/mask then the bag will inflate assuming you have a tight seal of the mask against your face. That's unlikely given the single elastic strap and the fact that the system should be designed to allow you to breathe in room air even with the mask fully secured to your face. That'll keep you from suffocating once the oxygen generator runs out.

Since you're likely going to be breathing much faster due to the adrenergic surge from simply having emergency equipment deployed it's unlikely you'll be breathing slower than the rate of delivered oxygen. A normal adult inhales/exhales 6-8 liters of air a minute and can easily double that rate. I don't believe for a minute you're getting that much oxygen from your mask although I don't have the actual technical specs on hand.

I should add, while there's still 21% oxygen at altitude, the partial pressure of delivered oxygen (and hence what you're obsorbing) is lower at altitude. Hence you need more oxygen when you go higher up. The whole point of this is to keep your arterial blood oxygen level high enough to keep you from passing out. Aircraft are normally pressurized to 8,000 feet above sea level.

At sea level barometric pressure is 760mmHg, with the partial pressure of oxgen at 159mmHg. At 8,000 feet this drops to 564mmHg and 118mmHg respectively. At 30,000 feet it is 226mmHg and 47mmHg.

Now compare your sea level PaO2 (the amount of oxygen in your blood) to your levels at 8,000 feet and 30,000 feet. Let's say you have a "normal" range PaO2 of 90mmHg and no lung disease. At 8,000 feet you have a PaO2 of 56mmHg or just before the steep part of the oxyhemoglobin curve where your body gets really upset at you. At 30,000 feet you're estimated to have a PaO2 of 23mmHg which is pretty much guaranteed to make you pass out.