I think that usually people regard music as something highly individual, something that is above social concerns and to be enjoyed in the privacy of one's own head. It might even be seen as an indicator of strongess: to be able to put those irrational feelings aside and just enjoy the thing in itself.
How I see this is that this view that stresses individuality is at odds with the - I think ecologically valid - view that music is at it's core social, shared, synchronization with other people. Now there's this conflict: on the other hand our culture strongly emphasizes the individuality of musical experience; on the other hand we can't escape the fact that musical experiences are social, shared, a kind of connection to the composer and performer. When someone presents the question whether you would listen to a string quartet by Adolf Hitler, these two views are in a conflict: we would like to act in a culturally approved way and show the strongness and individuality everyone expects from us, but at the same time we don't feel comfortable with having a emotional connection to Adolf Hitler.
This connection, if one exist, has to be in a sense "virtual". Obviously when one listens to, say, Schubert, one can't directly be connecting to Schubert. I would think it is something akin to looking at a movie: we are not dealing with real objects but representations of those objects: the actors are not really present, the actions are not really happening, it's all just dots on a computer screen, but still the emotional connection can be almost as strong as with real people. A CD is just a mechanical reproduction of something that has happened in the past: we don't, at least usually, think that we would be listening to our speaker elements - although their transduction of electrical activity into air pressure waves is the direct acoustical reason for our experience. We would say that we are listening to Schubert (performed by someone).
These mental objects are obviously abstractions. The most obvious reason is that no one here - I'd guess - has ever met Schubert, so no one can concretely know what kind of a person Schubert was. But even if one had a time machine and could go back in time and meet Schubert - would that really change the situation? Isn't every shared experience always in some way abstraction, a part of the whole experience? If someone could share her WHOLE experience with someone, I guess the one with whom she would be sharing the experience should be the exact same person who is sharing the experience - herself. Only parts of our experience can be shared and in that way we are always connecting with abstractions.