It would be more natural to say something like, "I'm attending a beginner's Swedish language class." In the U.S., at least, "elementary" is typically used as an age level for children. I is possible to use "elementary" to mean "basic level," but most of the time, it will be in the form of a predicate adjective such as "This class is very elementary." That would be clear to native speakers that you are talking about the subject level and not the age of the students.

Attending, yes, as far as I know. And if that's true, good for you! You're very lucky :)

Taking is more native. Attending is correct and more proper