These are mostly exclamations of surprise. By gosh and by golly are slightly stronger than gosh and golly. There's a saying "B'Gosh and B'Gorrah" that's a mild curse.

To me, they do seem the same. They are used in informal speaking situations.