1 and 3 are correct. Both of the placements of "both" are acceptable. Another acceptable way of saying it would be "A and B both are larger than C and D." 2 and 4 are wrong, because they're using "or" instead of "and." To put the "and" into math terms like you did above, it would be {A,B}>{C,D}. I honestly don't think the "or" could be put into math terms, since there would probably have to be some dependency expressed. If you'd like me to really get into it, you can leave a comment asking me to and I'll try, but hopefully the "and" math is clear enough that I don't have to.

"Or" used to link alternatives Would you like coffee or tea? Its one or the other. This is a style point I'm going to make; so feel free to disregard. The use of "both" is not need in any of these sentences. Again not wrong but economy of words is often best in my opinion. A and B are larger than C and D. to quote Pascal "If I had more time, I would have written a shorter letter."

As a native English speaker, I would use (1). (3) also makes sense. (2) and (4) confuse me a little, but I would understand the meaning.