It can also refer to a person who is grumpy, surly and mean-tempered. "My boss is a crusty, entitled jerk most of the time." It is used sometimes to mean one is really dirty. Let's say, for example, you get home from a camping trip in the wilderness and you haven't showered for a week. You say to your camping buddies: "The first thing that I am going to do when I get home is shower. I'm feeling a bit crusty right now." Remember, a crust is the outer surface on breads and some desserts. So, yes, context is important. Good luck with your learning goals!

it's slang so it would heavily depend on the location of the person saying it, but I've seen it being used as either ugly, dirty or lousy