You just need to listen more.
It's as simple as that.
People may sound like they are speaking quickly, but that is only because you aren't used to listening to Japanese yet.
Keep on practising, and really really try to hear what people are saying.
If you are using youtube or something you can listen to specific parts of the video as much as you like until you pick out what words they are saying.
If you have an mp3 player of some kind, try putting Japanese music or sound clips on there and listen to them on repeat for a little while.
Try to find lyrics of songs, or try to find subtitles of dramas and films (For example there are transcripts of Japanese dramas on
http://www.dramanote.com/)
Basically if you can read what the person is saying as you listen to it, then definately do it.
Listening practice can be difficult to start with, and it's easy to feel like you aren't making any progress for a while, but keep on listening and eventually you will start hearing more and more.
Also keep in mind that when watching anime you are quite likely to encounter a lot of words which aren't even in your vocabulary yet, so naturally you won't be able to hear what they are saying properly.
So having said that perhaps you should look for a language exchange partner to speak with on skype, since daily conversation is a lot easier to practise to start with than listening to anime or dramas etc.