Do they all mean the same thing? No.
1. This course is selective.
Hmm. This is an odd phrase, and it isn't clear what it would mean. If someone or something is 'selective', it means that that person or thing is careful about the choices they make. For example, if you're selective about your friends, this means that you're fussy about who you associate with.
If I saw this in a prospectus, for example, I'd interpret it as meaning that the course - or the people who are running it - can pick and choose which students can enrol for it. In effect, it would mean the opposite of what you think it means
2. This course is alternative.
This doesn't mean anything much. Alternative to what?
3. This course is an elective.
'Elective' is a noun here, and educational establishments use this for the type of course where students have a choice between several options. For most higher-education qualifications, some modules involve a compulsory course, while other modules give students the choice between two or more courses.
4. This course is optional.
This means that you aren't required to take this course. The implication is that this is an extra course which is on offer if students want or feel the need to take it: students can take it if they wish, but there is no obligation.