"to come over" means to pay a visit. So, "she came over and sat down next to him" means that she visited and sat next to him. On the other hand "to come up" means to approach or to come near. Hence, "she came up and sat down" means that she came near him and sat down. The difference is usage is that suppose you are at some market place and a friend sees you there and comes near to say hello, you would sat that: He/She came up and said hello. Whereas if you are at your home or office and some friend comes to your place to say hello, you would say that: He/She came over to say hello.

The correct phrase is "came over". It doesn't make a lot of sense to drag phrases out of context. "Came up" is right in a certain context. Go back to your original context and you'll understand it.