With some slight modifications, one may easily see that the main sentence becomes, "Someone's studies were continued his studies".
This is certainly an illogical sentence, since
1) when "to continue" is used as a transitive verb, it should be in a form "someone continues something";
2) when "to continue" is used as an intransitive verb, it should be in a form "something (or someone) continues".
Another point is that, when using a gerund phrase in the subordinate clause, such as "after graduating", its subject MUST match with the subject in the main clause. In the example, "after graduating" refers to a subject of a person, whereas the subject in the main clause is "the studies of Prof. Smith", which is an impersonal noun. Otherwise, you need to have a complete sentence in the subordinate clause.
So, suggestions to the answers of this problem may be,
1) To use "to continue" as a transitive verb,
-- After graduating, Prof. Smith continued his studies. [as suggested by other respondents]
-- After Prof. Smith's students graduated, Prof. Smith's studies were continued by himself/his colleagues. [transitive verb in passive form]
2) To use "to continue" as an intransitive verb,
-- After Prof. Smith's students graduated, Prof. Smith's studies still continued.