While some of the older Beethoven symphony cycles are great, and some of the new HIP recordings are intriguing, my recommendation is to go with Osmo Vanska's recent cycle on BIS. These are great recordings. I enjoy the Immerseel/Anima Eterna recordings, but Vanska's is so much more fulfilling.
And the symphonies are definitely a must in a Beethoven collection. I also highly enjoy Szell's recordings with the Cleveland Orchestra, and they have the added benefit of being pretty dirt cheap - they are on Sony, and I have been finding the discs at my local B&N for only ~$7.99 each (most containing two symphonies). Don't let the price fool you. His recording of the 3rd is generally praised.
Piano Sonatas - If you are interested, get all of them. If not, get the "highlights" - Moonlight, Appasionatta, Pathetique, Waldstein, Hammerklavier, among others. I love Kempff's recordings of the middle period sonatas. I'm told, though, that his late sonata performances don't have the same appeal. I have the cycle by Goode on Nonesuch, and it is quite good.
Missa Solemnis - If you enjoy the 9th symphony, this may interest you. It was written around the same time. Klemperer's recording is wonderful, but Gardiner also gives a good show. One of the great choral works, in my opinion.
Concertos - Triple Concerto, Violin Concerto, Piano Concertos - get them all.
String Quartets - I am not that enthralled by the early quartets, but the middle period quartets (esp. the Razumovsky and Harp Quartets) and the late quartets are very nice. For the middle quartets, I have a recording by the Takacs Quartet on Decca that I have very much enjoyed. In addition, the Brodsky Quartet does a good job. For the late quartets, I have only the Emerson String Quartet's recording, which I enjoy, but have been looking to add another recording to my collection - possibly the Takacs Quartet's recording. Takacs does a wonderful job - I also have their recording of the early quartets, as well as other chamber music from Brahms and Schubert and Haydn. All are wonderful, with great sound.
In my opinion, Beethoven is at the top with Bach. It has nothing to do with volume, rather how much their music moves me. They battle for the top spot. Yes, some of their works (the 9th symphony) do get over-played, but that does not mean that they still aren't incredible works. And even with the 1st and 2nd symphonies, which probably only get less recognition because of what followed, and not for any flaws they possess, his symphonic cycle is the most perfect of symphonic cycles (sorry, Mahler, Beethoven beats you out, in my book). For both of these masters, it is hard to find an area in which they do not hold a commanding presence, if not the top spot. (By the way, that is all my opinion)