I think both phrases could be considered slang. You probably wouldn't use them in formal situations. I agree with the dictionary definition of "psych out." It means to do things that make someone else feel less confident. I think the idea behind the phrase is that you're putting ideas in someone's mind that make the person more likely to fail. It doesn't necessarily have to be aggressive. For example, let's imagine that Anna is an actress who wants to get a part in a play. Before Anna auditions, Betty, another actress competing for the part, tells her "don't forget your lines!" Anna then gets nervous and forgets her lines because of what Betty said. You could say Betty psyched out Anna. I've never heard "psych out" used to mean understanding another person's thoughts. "Psych up" is different. It means to do things that give someone the energy and focus to do something. It's almost the opposite of "psych out." For example, sports teams sometimes psych themselves up before a game by listening to a speech, or by doing a cheer of some sort. Does that make sense?