If you’re roughly my age and vintage (or older), it’s possible one of the earliest information resources you used to start your whisky journey was Michael Jackson’s “Malt Whisky Companion”. First published in 1989, it was a book that took whisky writing to new heights for many reasons, but one of the more far-reaching elements it introduced was the concept of scoring whisky. Each entry in the book would be given a score out of 100 and, suddenly, whisky readers had a point of reference and a measuring stick to judge one whisky over another. So how do you score whisky ?
Jackson’s scoring system was interesting. He scored whiskies out of 100, and yet he explained that each whisky was actually given a starting score of 50, just for turning up! He’d then add points to reflect the quality of the dram, and one of the pages in the book then explained his scale so that you could interpret each score and calibrate it to a level of quality. However, being the slightly mathematical person I am, this system troubled me: If a whisky’s score starts at 50 just for turning up, and Jackson gave a whisky a score of 80, was his score actually 80/100, or was it 30/50. One suggests a mark of 80%, the other only 60%. Quite a difference in perception, wouldn’t you agree?
14 years later, Jim Murray came along with his Whisky Bible and his own scoring system. A whisky’s qualities were defined and critiqued by four characteristics (Nose, Taste, Finish & Balance), and each characteristic was given a score out of 25, thus arriving at a final score out of 100. However, whereas Jackson would start with 50 and then add marks for quality; Murray would effectively start with 100 and then deduct marks for faults or flaws. My point here is simply that a whisky that scored 80/100 in Jackson’s book couldn’t really be compared with a whisky that scored 80/100 in Murray’s. And that’s assuming that both writers had the same tastebuds, preferences, likes, and dislikes – which they didn’t!
One of the biggest problems with scoring whiskies is that it is so subjective. There’s no international standard or system in place for scoring, and – even if there was – we’ll never escape the fact that what tastes great to one palate might be awful to another’s. And if someone – be they a famous whisky writer, or just a humble punter – gives a whisky a score of 9/10, is that really the gospel indicator of the whisky’s quality? Should a whisky’s reputation live or die by a number that is assigned to it?
The other – and more complicated – issue with scoring is that everyone calibrates differently, i.e. we all work to different scales of value and relativity. Again, there is no international or standardised system or scale in place. I might work to a scale where anything over 75/100 is an exceptionally good whisky. However, if you’re used to reading Jim Murray’s Bible, you’re probably accustomed to thinking that a whisky has to get to 85/100 before it rates in the upper echelons. At least with Jim, you see a large and diverse range of scores that hit the full spectrum of quality. Conversely, I remain amused (if not confused) by one particular whisky blogger I’ve noticed who scores every single whisky he tastes somewhere between 87 and 92 out of 100. Since a mere (and ridiculous) 5% separates his best ever whisky from his worst ever whisky, what can you realistically take from his scoring?
Don’t get me wrong – I’m not against scoring whiskies and, indeed, I critique and score whiskies incredibly frequently in my various whisky roles and activities. I have my own system and scale, and I rely on scoring whiskies to distinguish the goodies from the baddies. It’s a good thing. But I readily acknowledge that my scoring really only relates to my palate; my preferences; and reflects my own personal whisky journey and experience. And I’d suggest that that’s not transferrable.
Following on from that very point, there are two great examples which illustrate why praising or condemning a whisky based on someone else’s score is fraught with danger:
1. You have to know the scorer’s journey and experience. Someone who’s been drinking and critiquing whiskies for 20 years will have a very different outlook on a particular whisky to someone who’s been at it for six months. That’s not at all being dismissive or disrespectful to the newbie; it’s just a fact that we judge and rate based on experience and by comparing back to what we’ve had previously. Someone who has only just been introduced to Macallan will probably score their latest releases very differently to someone who was enjoying Macallan 15 years ago when the likes of (the original) Macallan Gran Reserva were going around!
2. Almost every whisky drinker I know has had the experience where they’ve identified a whisky that scored extremely highly in Jim Murray’s Whisky Bible, and then gone off and sourced a bottle of same. They’re high with hope and expectation that they’re about to taste a gem, only to be underwhelmed or wonder what Jim was making all the fuss about when the whisky fails to impress them. “Did Jim get it wrong?” “Was his sample different to what’s in my bottle?” “Did my bottle get left out in the sun?” These are the doubts and questions we ask as we struggle to comprehend why something we perceive as being plain or nothing special was scored 95/100 by one of the great whisky writers. No, Jim didn’t get it wrong, and neither did you. The point is, Jim’s score was Jim’s score, and your score is yours.
My final observation of why scoring whisky can ultimately be unhelpful is demonstrated in most issues of Whisky Magazine. The new releases that get published are typically scored by two panellists, together with some short tasting notes. If the same whisky gets a score of 9.1/10 by Dave Broom, and 5.9/10 by Martine Nouet, how can you possibly gauge if it’s a good whisky or not? All you know is that one person loved it and the other didn’t.
My advice instead is to focus not on the score, but on the tasting notes. Look for aroma and flavour descriptors that appeal and that you know you like, and you’re far more likely to be satisfied. ‘Coz I don’t know about you, but I drink whisky for its flavour…not for its score.
Cheers,
AD
Got any thoughts on this issue yourself? Scroll down to the comments section below and add your two cents worth…
Fantastic post! You’ve managed to articulate the point (and indeed my own thoughts) incredibly well.
Great article Andrew! I sometimes score whiskies, and like you it’s not about the number – it is about the relativities – which one(s) I prefer in relation to others tried on the night. We also know that the selection of companions, and the order of tasting, can markedly affect our aroma, flavour, mouthfeel and aftertaste perceptions, for a given whisky. Hence I have a preference to keep a small sample in the glass for later (comparative) reference. Like you, the nosing and tasting notes are most important for me in other peoples reviews. The Australian SMWS tasting panel does well in this regard.
Chris Middleton of Whisky Academy wrote a fantastic reply to me via email, and has given permission for me to reproduce it here. Chris wrote:
“Andrew, you’ve make a good point about the need to be conscious of the subjectivity whiskies judges pronounce in their sensory evaluations.
It’s a fascinating field of study from olfactory and organoleptic clinical studies, sometimes they can even be hilarious, if not disturbing, such as Fredric Brochet’s 2001 wine tasting experiments at the University of Bordeaux. Beauty, as with taste, is in the oral cavity on the beholder, along with some sight and even sound too.
You make an insightful point on the judging dichotomy between Jackson’s perspectives of adding points, to Murray deducting them. These are two opposing methodologies who are seeking to quantify whisky samples by variations against similar criteria and comparable taste training (i.e. SWRI). Some years back I back-analysed 5 years of the leading judges verdicts and found rarely did any of their numerical points concur. There was even greater viability in their use of flavour descriptors. The only observation I could determine was they did numerically cluster, evidencing similar opinions on how they perceived ‘quality’. Amongst this learned cohort they were vindicated by the law of statistical averages, much less so with their verbal statements.
At the other end of the scale, the macro level, I am also very guarded about judging competitions. Small distortions get amplified. Many competitions are merely commercial enterprises designed as marketing platforms or by organisations seeking a profitable venture from holding a whisky event, generating revenue and profits from entry fees and other merchandising expenses. Even distilleries have to allocate part of their annual A & P budget to how many they can afford to enter, if at all any. Many don’t, so it becomes the best who appeared on the day.
My observations, sometimes pet peeves, concern the reliability and robustness of competitions.
– Which events attract what entries? (Two dozen whisky competitions and counting, thousands of awards on offer i.e. Whisky Magazine has 150 whisky award levels)
– Who enters and how many entries? (Sometimes only a couple of samples are entered for a category – everyone wins a prize)
– Who judges? (How credible are the panel, e.g. bartenders, media celebrities, etc.)
– What’s the criterion for value? (Is a $70 malt matched against a $700 malt)
– What’s the scoring system? (Some have specific checklists, others are open-ended)
– Has the deck been stacked? (Craft distillers pick their best single cask, professional distilleries enter brands – one produces only 250 bottles, the other 2 million bottles, consistently each year)
Being aware of the potential faults and the personalities in the judging game does give the insider another filter, albeit a subjective filter. While winners can use awards to their PR and marketing advantage, for many whisky drinkers we get to vicariously consume labels we can eeither not afford, nor find.
Chris Middleton is totally correct.
To that list, other influential areas I would add, are: bias – most tasters will inevitably, and understandably, give disproportionate results to their preferred style, whoever the taster is; commercial connections (direct and indirect) – knowing which side the taster’s bread is buttered, whether amateur blogger or professional journalist; tasting ability – technical tasting is a great skill and not the same as just drinking thoughtfully. There is a sterile process to follow that should be identical for each and every whisky. One cannot deal fairly with an assault of 10 or more drams, nor when half cut or in varying environments.