There is a lot of difference. You will like it.
I'm now 3 days in with my SE. Much better battery life and it is zippier. I'm not getting on Tmobile's band 12 with it yet at work, which is what I was hoping, so I need to look into that some more.
I purchased my 5s two years ago outright for $750. I traded in my old 4s end of February and my 5s when I picked up the SE, so the 64gb SE only cost me a bit under $300 with tax. If I get two years out of it, that seems like a good return.