Straighten out means more on a horizontal level. Like say you lay down a tablecloth on a table and its all bucnhed up in the middle. You may be told to "Straighten that out"
Straighten out can also be used like this
"First we should straighten out this situation before worrying about that."
Straighten up means specifically up. But it can also mean like gather composure or to get a room in order.
"This room is so full of junk! There's nowhere for our guests."
"Let's straighten it up then" (tbasically like saying "Let's clean this up")