clement13.net
Original photo: Un cadran gradué éclairé, dont l'aiguille se lit sur l'échelle
Description
Un cadran gradué éclairé, dont l'aiguille se lit sur l'échelle
Source
freestocks.org · original page
Licence
CC0 Public domain
Original photo, unretouched. The background is a version of it stylised by the site.

Rating: How to Read Ratings, Scores, and the Legend

This section uses three measurement tools: criterion-by-criterion /5 ratings (method, tool, and architecture comparisons), maturity scores (five-level matrices), and…

Guide≈ 7 min readCheckedCheck doneFacts, figures and references were checked against the sources cited in the page.Written with AI assistance; sources are cited at the bottom of the page. An error? Report it via clement13.net.Reading
In brief

This section uses three measurement tools: criterion-by-criterion /5 ratings (method, tool, and architecture comparisons), maturity scores (five-level matrices), and reading-level markers (🟢🟡🔴). None of these is a physical measurement. This file explains what each one means, how to read it without misinterpretation, and how to rate your own organization honestly.

On this page
  1. 🟢 Novice — /5 Ratings in Comparisons
  2. 🟢 Novice — The 🟢🟡🔴 Scale and Callout Legend
  3. 🟡 Practitioner — Aspects and Their Subjectivity
  4. 🟡 Practitioner — Maturity Matrices
  5. 🔴 Expert — The Matrix Rating Rule
  6. 🔴 Expert — Gaming a Measure: Goodhart's Law
  7. See also

Level: 🟢→🔴

🟢 Novice — /5 Ratings in Comparisons#

In comparisons (for example Comparison of Project Management Methods, Comparison of Planning Tools, reporting architectures), each solution receives a 1-to-5 rating per criterion, written 4/5 or ★★★★☆ 4/5.

Rating Meaning How to read it
1/5 Very weak on this criterion A blocking point or nearly so, unless expensive workarounds are used
2/5 Weak Possible but painful, or with major tradeoffs
3/5 Adequate, average Gets the job done, no more
4/5 Good A clear strength
5/5 Excellent Best-in-class on this criterion in the given context

Three reading rules:

  1. A rating only has meaning in its context. Each table states the context (team size, project type). A 5/5 for a 5-person team may be a 3/5 for 200 people.
  2. A 3/5 is not criticism: it's the middle of the scale. Tables are designed to discriminate; if they were all 4s, they would teach nothing.
  3. Do not compute blind averages. A solution rated 5/5 on ten criteria and 1/5 on “fit for your case” is still bad for you. Read the one-sentence justification, and weight according to your priorities.
🧪 Example

Two options rated on three criteria: A = {cost 5, freshness 2, governance 3}, B = {cost 3, freshness 4, governance 3}. Averages: 3.3 each. If your need is an operational dashboard refreshed every 15 minutes, freshness carries significant weight (for example ×3): A = 2.8 · B = 3.6 — B wins. The math is trivial; what matters is making weights explicit before looking at ratings.

🟢 Novice — The 🟢🟡🔴 Scale and Callout Legend#

Level markers. Placed in headings or at the start of a section:

Marker Audience Content
🟢 Novice Discovery Definitions, vocabulary, principles, simple examples
🟡 Practitioner Day-to-day practice Implementation, tradeoffs, tooling, concrete cases
🔴 Expert Mastery Internal mechanisms, edge cases, scaling, fine-grained tradeoffs

The meta line Level: 🟢→🔴 in a file indicates the covered range. You can stop reading at the end of your level without losing the thread.

Recurring callouts.

✅ Good Practice

A proven practice, with the reason it works.

⚠️ Pitfall

A frequent error, with its symptom, cause, and remedy.

💡 Expert Tip

A detail you generally only learn through experience.

🧪 Example

A concrete case, often with numbers, based on the running example of a SQL Server datamart fed by a field sales force automation (SFA) application.

🤖 AI

What AI brings at this point, and its limits.

🟡 Practitioner — Aspects and Their Subjectivity#

“Aspects” are the comparison criteria. To stay consistent within a dossier, this section groups them by family:

Family Typical aspects
Project methods predictability · adaptation to change · risk control · lightness · fit for data projects · ease of adoption · scalability · visibility and governance · deliverable quality
Version control system learning curve · performance on large repositories · branching and merging · traceability · permissions granularity · large files · CI/CD integration · SQL/BI fit · long-term viability
Planning flexibility · reporting · code/CI integration · cost · adoption · governance · scalability · customization · extensibility
Reporting architectures freshness · performance · cost · complexity · governance and security · scalability · required skills · technology maturity · lock-in

Why subjectivity is irreducible.

  • The choice of aspects already steers the conclusion: adding “cost” or removing “lock-in” changes the ranking.
  • The rating depends on the evaluator's experience, the state of the offering, and team context.
  • 1-point gaps are often within the margin of error; a 2-point or greater gap is a signal.
  • AI-generated writeups have not used your tools: the ratings summarize public knowledge and a typical expert judgment, not a measurement.
✅ Good Practice

Redo the table with your own weights and ratings in a 90-minute workshop. The gap between your version and the site's version is more instructive than the table itself.

⚠️ Pitfall

Quoting “4/5” out of context in a committee (“this section says X is worth 4/5”). Symptom: the decision relies on a number without criterion. Cause: losing the one-sentence justification. Remedy: quote the full row with context.

🟡 Practitioner — Maturity Matrices#

Maturity Matrices — Principle, Anatomy, and Comparison measure a practice. A matrix is defined over a domain, themes, and for each theme, five levels. Each level has:

  • a promise: the observable result achieved (never the means);
  • milestones: three to five precise, verifiable criteria, with a stable identifier <CODE>-<THEME>-<level>.<n> (for example PI-RSK-3.2).

Matrix codes: PI = IT project management, DL = software development, PD = data project management, BI = BI development.

🧪 Example

“Open a door.” Performance: level 1 = under one hour, level 5 = under one second. Noise: level 1 = very noisy, level 5 = silent. Locking: level 1 = none, level 3 = locked, level 5 = restored to initial state. We describe outcomes, not the lock.

By design, level 5 is very difficult to reach fully. It serves as a promise to drive progress, even if its milestones are only partially met. We assess substance only (practices, outcomes), not a real interface.

🔴 Expert — The Matrix Rating Rule#

Common rule, identical in each matrix README:

Each milestone has progress 0 / 0.5 / 1. A level N is achieved if the average progress of all milestones from levels 1 to N is ≥ 0.8 and no milestone is at 0. Theme score = N_achieved + average progress of milestones at level N_achieved+1 × 0.99 (e.g., 2.4). Matrix score = weighted average of themes (default weight 1, editable). A full 5.0 score is exceptional and may never be reached: that is not failure.

Step-by-step reading.

  1. Rate each milestone: 0 = absent, 0.5 = partial or uneven, 1 = present and verifiable with evidence.
  2. Find the highest level N such that the average of milestones from levels 1 to N is ≥ 0.8 and none of those milestones is 0.
  3. Add the average of milestones at level N+1 multiplied by 0.99 (the cap prevents reaching the next level by rounding).
  4. Weight themes according to your stakes.
🧪 Numeric example

Theme with 4 milestones per level. Level 1: 1, 1, 1, 1 (average 1). Level 2: 1, 1, 0.5, 1; average of levels 1-2 = (4 + 3.5) / 8 = 0.94 → level 2 achieved. Level 3: 1, 0.5, 0, 0.5; average of levels 1-3 = (4 + 3.5 + 2) / 12 = 0.79 < 0.8 and one milestone at 0 → level 3 not achieved. Score = 2 + (2 / 4) × 0.99 = 2.495, i.e., 2.5.

What a “partial 5” means. A theme may display 4.6: level 4 is achieved, and more than half of level 5 milestones are already partially visible. That's a very good result: level 5 measures the horizon, not an obligation.

How to read matrix scores.

Range What the score says
0–1 Practices are absent or individual; outcomes are unpredictable
1–2 Practices exist but are uneven across people or projects
2–3 Practices are defined and applied; outcomes are generally predictable
3–4 Practices are measured; decisions are based on indicators
4–5 Tool-supported continuous improvement; the organization corrects its own gaps

No range is a moral judgment: a score of 2 is suitable for a small internal project, and higher requirements there could be counterproductive.

🔴 Expert — Gaming a Measure: Goodhart's Law#

“When a measure becomes a target, it ceases to be a good measure.” A poorly used maturity matrix becomes a team ranking, and milestones get checked without changing outcomes.

Drift Symptom Countermeasure
Milestone checked with no evidence Score rises, outcomes stagnate Require the listed verifiable evidence for each milestone
Score compared across teams Polishing, information hoarding Scores reserved for the team that produces them
Aiming for 5 everywhere Overinvestment, cost without benefit Set a target per theme based on stakes
Assessment by management alone Gap from field reality Cross self-assessment, see Self-Assessment with Maturity Matrices
🤖 AI

An AI can prefill a grid from documents (tickets, repository history), which speeds up evidence collection. It cannot judge whether a practice is truly lived. Treat its suggestions as hypotheses to confirm by sampling.

See also#

✻ Written by Claude Sonnet 5.5