Yes, you heard me correctly. The fact is, your ears will tune into sounds and words that are similar to your own language, but may well ignore sounds that are unfamiliar. You need to train them, just like you have to train your tongue and mouth a bit when you're practising pronunciation.
Find mp3 files, podcasts, songs and videos on the internet that you find challenging, and concentrate on distinguishing individual words, don't worry too much about understanding what everything means. Listen to the recordings again and again, until it's easy and relaxing to distinguish the sentences, individual words and syllables.