In my opinion the most difficult thing is finding a good and effective method of learning. Obviously there is not one perfect method which allows you to speak foreign language fluently in few months or even in few years. Second thing is fear of making mistakes. You can't expect to speak 100% correctly. You don't even need to speak 100% correctly because even some native speakers don't speak 100% correctly. You have to speak clearly. Your statement has to be understandable. And the sooner we realize that, the better for us. The third thing is motivation to systematic learning. 15 minutes every day is better than one and half an hour once a week. And it realy works. But for me the biggest challenge is speaking with other people even with teachers...