In the context of asking a person to stop talking they are, as the above answers say, the same. They both mean 'stop talking'. So, I don't really agree with some of the comments that suggest you could ask someone to be quieter...being quieter is not the same as stopping talking completely! 😀
However, it is worth adding that 'Shut your mouth' can also be used to tell someone (in a very informal and direct way) to close their mouth in other contexts. For example "Shut your mouth, we don't want to see what you are eating!"
'Shut up' can't be used that way. It's a phrasal verb that means 'be quiet' - for example "I realised I was talking too much, so I decided to shut up."