My opinion is the opposite of all of that. Then again, I am the son of an avid amateur classical musician who was listening to all classical music all the time by the age of 4 or earlier, and especially from 12 to 21. As a toddler I sat beneath the stands of my father's string quartet. I didn't turn my attention to other genres of music until I was 21.
It sounds like you started later and have listened to much less classical music than I have. I can play the Brahms violin concerto by ear, not because I have heard the recording, but because I have heard the recording literally hundreds of times.
Learning some formal theory is highly useful, even necessary, if you want to perform or compose. But being a good listener just takes lots of listening. "Lots" as in thousands of hours unless you are a one in a million prodigy.
Wow, you realize that you had a very very rare upbringing. Music comes far easier to you just by listening, concentrating, since you can remember fine details in the melody of a Brahms concerto.
Obviously my worries don't apply to a prodigy or a very self motivated young person, or a very intelligent older beginner, who understands human nature better than I do.
Yes, playing in a rock combo, I started late, late teens, finding my way through the piano works of the big CM names. Listening to orchestral works came later, but listening to them was never a high priority for me. Jazz, yes listening is crucial if you want to sound like traditional, bebop, modern jazz innovators, because so much of it is listening to their finest examples, playing freely, using relative pitch and jazz harmony.