When "which" is used to introduce additional information, we use a comma "," before and after its clause. Here if we remove the which part, the sentence is complete. I visited the castle. It only has less detail now but the meaning does not change if we remove the which part. Your use of the comma is correct ^^

I would do the same, two separate clauses, WHICH is often used in the non-defining relative clause (that which has a comma)! Good work

I agree with Radhia on the fact that we use the comma when we want to add additional information. But I don't agree as much with the use of comma here. Looking at the complete sentence, we can infer that there are multiple castles in the country, and you visited the one which is the oldest. Now, if we only say "I visited the castle.", it will mean that there's only one castle in the country, which is not what we want to say. The second part of the sentence is necessary to make it clear that you visited a particular castle. It is not additional, but rather required information. Hence, the correct usage would be without comma, "I visited the castle which is said to be the oldest in the country". The comma would be required, however, if you mention the name of the castle, because you are adding information, but we already know which castle exactly, so you can remove the second part of sentence and it will still mean the same. "I visited the Fuji castle, which is said to be the oldest in the country."