Personally, it is hard for me to accept the idea that some people are incapable of enjoying music that is dissonant or atonal or highly chromatic or with irregular rhythms...
Rather pretentious, don't you think? Others might suggest that it is hard to accept the idea of someone who cannot fully appreciate Mozart.
I don't think it is insurmountable for anybody else to come to a true understanding of this kind of music and derive the kind of passion they do from other music. I can listen to this sort of music and enjoy just as much as I enjoy rock music and tonal classical and jazz music, and that is because I gradually grew to understand the music and what it had to say. I am not without sympathy for those who have difficulty with some of the music. Total serialism and indeterminate and aleatoric music are extremely difficult to really get into. The syntax of that kind of music is extremely complex and difficult to understand, but that doesn't mean it is terrible music, it is just very challenging music.
Some undoubtedly dislike atonal music because they don't understand it... but seriously, it is no more complex or challenging than any number of other works of music. I suspect that there are more than a few who dismiss even trying to come to terms with atonal music because they are put off by this sort of attitude: "If you were willing to put forth the effort... if you weren't afraid of something just because it is intellectually challenging... if you were like me..." The reality is that some fully understand atonal music and still dislike it.
As far as music like that of Stravinsky, Schoenberg, Messiaen, Cage's work from the 40's, and even to an extent composers like Crumb, Varese and Ligeti, I don't have much sympathy for people who can't grasp that music. It may be difficult at first, but there are so many things these composers have in common with more traditionally tonal composers that it doesn't take terribly long for one's listening vocabulary to grow to accept what occurs in their works, and enjoy them.
Contrary to your suggestion, I have little doubt that there are many good... even "great" works of music that you don't like/can't grasp... Ummm... Mozart? Honestly, I appreciate Schoenberg, Crumb, Feldman, Webern, Ligeti, etc... but I don't really love them. I need to be in the right mood for them... otherwise they strike me as nothing more irritating noise.