Notes · · 7 min read
Ranking beats rating: why comparing two tasks works better than scoring ten
Scoring tasks out of ten feels precise and produces ties. Comparing two at a time feels crude and produces a decision. Here's why the crude method wins.
Absolute judgment is the hard version of the question
Rating a task out of ten asks you to hold an invisible reference scale in your head — ten compared to what? Your own scale drifts with mood, recency, and whatever you looked at last. This is why the same list scored on Monday and Thursday gives different numbers.
Comparing two tasks removes the invisible scale. The reference is right there: the other option. Everything you need to answer the question is in front of you.
Ties are the failure mode, not the kindness
Rating scales generate ties by design: with ten items and ten possible scores, collisions are guaranteed, and in practice everything clusters in the top third.
A tie preserves the exact ambiguity you were trying to resolve. Forced choice — no ties permitted — is uncomfortable precisely because it is doing the work. If a comparison feels close, that is information worth recording, not an excuse to skip it.
You don't need to compare everything
The objection to pairwise comparison is combinatorial: ten items is forty-five pairs, and nobody wants to answer forty-five questions.
You don't have to. Ranking algorithms exploit transitivity — if A beats B and B beats C, A vs C is usually settled. A well-built comparison flow needs roughly n log n judgments, which for ten items is around twenty-five, and most take a couple of seconds.
Close calls are a feature
When two items are genuinely near-equal, the comparison that separates them is low-confidence. Recording that — clear win versus close call — tells you which parts of your ranking are stable and which are worth revisiting mid-week.
A rating scale hides this entirely. Two items both scored eight look equally settled, when one might be a firm conviction and the other a coin flip.
Where rating still makes sense
Rating is fine when you need an absolute threshold rather than an order: is this bug severe enough to page someone, is this candidate above the hiring bar. Those are calibrated judgments against an external standard.
For 'what do I do first', there is no external standard. There is only your list, and the only meaningful question is which item beats which.