To me, a "naturally gifted" listener would be in part someone with a great memory - if the first time you hear a 40 minute long symphony you notice that a motif from the opening movement has returned at the end of it, I envy the goll darng tar out of you.
There is probably also some individual difference in how well we can recognize intervals, not only in that some people can do so more accurately than others but that some people can do so much more easily (without extensive ear training exercises) than others.
I don't think we can easily contest those things. Of course musical education and experience can improve those things, but some people seem more talented than others. Individual variation is just a fact of life.
But whether there is any such thing as "getting" music in some romantic sense is open to question to me. I don't know. Sometimes I think so. Sometimes I don't .
In all questions like this, for me it comes back to the analogy with literature because I am much, much more confident in that field. There is such a thing as not "getting" something like, say, Shakespeare. Usually the main thing that means is a failure to understand the language itself, but in a lot of cases it also means a failure to appreciate the literary elements (you know, metaphors, puns, comparisons and contrasts between characters, irony, etc.) that he uses so well. Those guys, I'd say, don't "get it." Probably most of them could, if they were willing to put in the work. But there are also some people who in spite of being as aware (or more aware!) of such things as I am still don't enjoy Shakespeare - for instance, the vulgarity and cynicism turn some people off. Those guys, I'd say at least, "get it," but just don't enjoy it.
Edit: On the other hand, the skills that go into appreciating Shakespeare can make other authors or texts less enjoyable. If I were unaware of issues like symbolism or significant detail or whatever, I would probably enjoy Bradbury more than I do....
I think there is an analogy between literature and any other art: if you are educated in the field you can appreciate things that more casual observers can't. But I don't necessarily equate appreciation with enjoyment - there is a kind of legitimate variety of taste that exists independently of appreciation. Of course modernist art often aims (or seems to aim) to alienate a majority of its audience, but that is at worst only a complication.
In the field of music I've decided to enjoy it without working to further educate myself about it. I guess I will continue to learn more about it, but I'm going to do so at my haphazard pleasure, just having a good old time. Anyway, wow have I gotten off topic here. I'm going to leave this here because I'm too self-indulgent to delete it, but if you actually read this I should give you a "like" for getting here.
The original point of this example was supposed to be that some people pick up literary stuff more easily than others. There are naturally talented readers. Of course education can supplement or to some degree compensate for what nature gives. Edit: BUT "getting" that stuff is different than "enjoying" a work - and sometimes (or often?) "getting" that stuff means not enjoying a work as much as a more naive person might.