Two meanings:
m1) the sport or game, the one with the the holes in the ground.
m2) exactly what you said, both for men and women.
Sometimes we use also the form 'golfino' to mean a very light golf (the sweater)
Another name for 'golf' as a sweater is 'maglione', like in
-- questo maglione da sci e` quello che preferisco, mi tiene caldo e non e` troppo pesante.
Usually a 'maglione' is heavier than a 'golf'.
Other time we just say 'un pullover' instead of 'un golf'.