They are similar but there is a difference.
침묵하다 means "keep silent", or "do not say anything", to describe someone's behavior.
과묵하다 is used more often for someone's disposition, meaning he or she is taciturn, not given to talking by nature.
그는 그 일에 대해 침묵했다.: He kept silent about that matter.
그는 과묵한 성격이었다.: He was a man of few words. He was of a quiet nature.