Security is normally pretty quick in OSL, especially in the Fast Track lane. The lounge (which is ok) is before immigration. Then, as others have mentioned, you need to go through to the non-Schengen area. The last time I passed through it took about 15 mins in the EU line.
I would arrive at the airport a minimum 90 mins before the flight, and check what the immigration line is like as soon as you get airside. If it looks long, then get queueing! There is pretty much nothing to do in the non-Schengen side, just a shop and cafe.
The OSL building is nice architecturally, but it really isn't fit for purpose. Even excluding the potential immigration lines, the very linear and narrow layout means that a lot of gates are a very long walk (and it's so cramped and busy, there is no room for travelators). I wonder if the new pier will be better? Really instead of the "two very long thin sticks" layout, they should have gone with a "central bubble with numerous short sticks"! Baggage reclaim area is also a mess....so busy and dark! I hate OSL!