I think so far, grammar and verbs have been the biggest hurdles for me. Its just so different from english. However in saying that, it does get easier the more you learn. I self studied for a couple of years in my spare time but now going to classes and having jp friends it really does boost your learning. I regret not studying this in High School.
In saying how hard it is, that really makes it rewarding to see progress in such a hard language. Every day is a new learning experience. Brining me a step closer to being at a level which I can speak and write effortlessly. I can't wait!! jaa. mina ganbare ne.
I really want to learn my mates local dialect more too. (Kansai Ben) to suppliment my text book tokyo language.
dialects make language learning hard too, kinda have to relearn what you already know with a twist on old words you know.