For me, two categories make sense: instruments that take seriously good breathing (circular breathing, too) and tongue tricks, and instruments that take serious manual dexterity.
The hardest in the first category has to be the bagpipes, or something similar. Man does that take air. If you can't pull off circular breathing, expect to just pass out. I've passed out myself during practice.
Now, the hardest in the second category, I would say is the organ. Especially if you're playing music where you are required to play nasty little ornaments with your feet. In many musical situations it was considered obligatory to do so in order to recite the theme correctly as the feet have their go at it. Add some of the toughest examples of counterpoint to that, and virtuoso literature very similar in technical demands to the most difficult of romantic piano music. The only argument I could think of for the piano being more difficult, is that the organ lacks the action of a piano.