They each carry different meanings. The first- 'by saying goodbye' means that he said goodbye through the means of, or by using the expression 'see you'. So you can see why it doesn't make much sense as 'see you' is a way to say goodbye, not vice versa.
The second 'to say goodbye' just means that his intention was to say goodbye and he did it by saying see you... or He said goodbye by saying see you. This one makes actual sense as that's what one does by using the expression 'see you'.