It doesn't mean "to be in big trouble," or "to be in a difficult situation." Denis is entirely correct: it means to put yourself in a dangerous (exposed) position.
Also, "to go out on a limb" is generally done for someone; or to serve higher (nobler) purpose. "He really went out on a limb there, speaking out for her the way he did."
You can't really use it when you simply put yourself in danger. Like if you're into hazardous sports, such as rifting, for instance, you wouldn't say: "I went out on a limb when my car almost tipped over." It's not grammatically incorrect, but simply not used that way. However, you would say, "He went out on a limb, trying to prevent his friend's car from tipping over."