They are not interchangeable, but you can say "in the morning of July 3". Here's an example where "in" is better than "on": "Jack had waited weeks for the letter to arrive. He could hardly stand the suspense. When would it come? What would it say? Then, late in the morning of July3 he heard a knock on his door. It was the postman!" Because of the word "late", "on" would not work at all.

You can do something "in the morning" once or habitually, but if you add a specific date you need to change the preposition to "on." The structure "in the morning" + "of" + a specific date just doesn't work in English.

You would say 'on the morning of July 3rd, 1863' :)