I have no real learning experiences with the norwegian language. But as far as I know vocabulary and alphabet just differs a little bit from swedish and there are no big differences in grammar or "regularity" which is (in my opinion) much better than in danish (What's so irregular about English?). However, I wouldn't call pronounciation and listening as easy - it's something that you need to getting used to.
To put it in nutshell: There are more native Swedish speaker, but there shouldn't be any really big difference.