We've been to both Riga and Stockholm and enjoyed both. Given that you haven't visited Stockholm and you will already be in Stockholm, I would stay. Otherwise, you chew up the better part of a day in moving. There's more than enough to keep you occupied in Stockholm for 4 days. It really is lovely in the longer days of spring and summer.