Nothing is wrong with Karajan; he is clearly one of the greats. I find his Richard Strauss to be almost without peer, and I think all of his Bruckner is excellent to superb. I am a fan of the 1962 Beethoven symphony cycle as well, and in one or two Sibelius Symphonies (especially No. 4) his recordings are as good as anyone's. He was also enormously successful in opera, and sometimes really great in repertoire where, if you listened to the clichés, you might be surprised. His Hadyn, for example, or Dvořák 8. And there's an excellent Prokofiev 5, as well as very fine recordings of Brahms, Tchaikovsky, Schönberg, Berg, and Webern.
Like all conductors, there were a few duds, such as his Mozart Requiem, which for me is heavy and dull. Just one example.
I think the main "problem," such as it is, has been diagnosed well above. Namely, his mostly deserved high popularity, for a Classical musician, has spawned a certain backlash. The backlash has written a narrative about Karajan, that he was a one-dimensional conductor whose interpretations were always smooth, homogenized, and without edge, that has a very small kernel of truth. But anyone with a better than passing acquaintance with his catalog will realize quickly that that narrative is not born out in reality all that often.
Record collectors can be fetishists for the obscure and esoteric, as well. "Oh, well, you know, Karajan is fine, I guess, but only because you've never heard Pispott Q. Jakhasz with the Podunkton Symphony Radio Philharmonic in a 1947 semaphore broadcast, only released in Japan in a limited pressing on reel-to-reel, never released on CD. Believe me, that performance makes Karajan sound like moldy sodden rubbish!"
Karajan and the Berlin Philharmonic helped define what great modern orchestral music-making is. That's just a fact. In many ways, his legacy is a victim of its own success.
I posit that if Karajan's entire recorded legacy was magically forgotten, but then rediscovered after a long period when no one remembered anything about him, we would all be blown away by the quality of what he accomplished.