I agree that ultimately it will be subjective for 95% of listeners and pieces. Depending on the composer and era, though, there is justification for more subjective judgement of a work. Gunther Schuller makes an excellent case for actually following what composers wrote in his book The Compleat Conductor. That composers like Brahms, Beethoven, Schumann, Ravel, Tchaikovsky, R. Strauss knew exactly what they wanted and are fairly explicit in their scores (especially to conductors who know how to delve and study), and that there is no reason for a conductor to alter this with their own ideas of what is "right" for the piece.
Of course, the lay listener (or even the pro who does not have the time or advantage of score study) is at a disadvantage and must trust their favorite conductors and orchestras and then judge by how the music moves them, and ultimately who is to judge that?
Baroque/Renaissance/early Classical has a lot of extra room for play.