Quality and UAT (QUA)

This theme addresses quality from a steering perspective: is the delivery accepted, based on what criteria, by whom, and what defect rate reaches users? The technical quality…

Guide≈ 5 min readCheckedReading

On this page

Why this theme
What the theme measures
The five levels
Milestones by level
Detailed reading by level
Partial achievement and rating
Move to the next level
See also

Level: 🟢→🔴 · Code: QUA · Matrix: PI

Why this theme#

This theme addresses quality from a steering perspective: is the delivery accepted, based on what criteria, by whom, and what defect rate reaches users? The technical quality of code and automated testing belongs to the Software Development matrix. Here, we look at UAT: UAT that is late, improvised, or lacks criteria turns production release into a gamble and the relationship with the business into a trial. A defect found in production costs, by commonly accepted orders of magnitude, several times to several dozen times more than one found in design (order of magnitude, varies by context).

What the theme measures#

Axis: from last-minute UAT to quality driven by prevention.
Key indicator: escaped defect rate (found after delivery / total) and acceptance decision lead time.
Unit: % of escaped defects; UAT days; % of acceptance criteria verified before delivery.

The five levels#

Level	Short name	Promise	Typical indicator (order of magnitude)
1	UAT at the end	UAT happens just before delivery, at best; defects are discovered by users.	Escaped: > 30 %
2	UAT script	A set of UAT cases exists and is used to validate each delivery; defects are tracked.	Escaped: 15 to 30 %
3	Upfront criteria	Acceptance criteria are known before development; UAT is planned and shared between team and business; the acceptance decision is tracked.	Escaped: 8 to 15 %; 100% of deliveries with criteria
4	Measured defects	Escaped defects are analyzed by cause; UAT relies on versioned reference datasets; trends are monitored.	Escaped: 3 to 8 %; cause analysis on 100% of major defects
5	Prevention	Most defects are prevented upstream; UAT confirms instead of discovering; users trust the numbers without recounting them.	Escaped: < 3 %; critical = zero

Milestones by level#

ID	Milestone	Verifiable evidence
PI-QUA-1.1	A UAT phase is planned before delivery.	UAT date in the schedule.
PI-QUA-1.2	Issues are logged with an identifier.	Issue list from the last 3 deliveries.
PI-QUA-1.3	An owner accepts or rejects the delivery.	Name of the acceptance decision-maker.
PI-QUA-2.1	A UAT scenario script is maintained.	Script version; numbered scenarios.
PI-QUA-2.2	Each issue has a status, severity, and owner.	Severity/status column filled in.
PI-QUA-2.3	The script is replayed for each delivery (at least the core).	Execution report by delivery.
PI-QUA-3.1	Acceptance criteria are written before development of each requirement.	10 randomly selected requirements: criteria present.
PI-QUA-3.2	Business users participate in UAT (definition and execution).	Names and participation dates.
PI-QUA-3.3	An exit threshold is written (for example zero critical defects, majors < 3, action plan for minors).	Documented thresholds; compliant decision.
PI-QUA-3.4	The acceptance decision is tracked with any reservations.	Signed minutes or report.
PI-QUA-3.5	UAT environments and datasets are described and reproducible.	Procedure for setting up an environment.
PI-QUA-4.1	The escaped defect rate is calculated per delivery.	Indicator across ≥ 6 deliveries.
PI-QUA-4.2	Each escaped major defect is subject to cause analysis and preventive action.	Analysis sheets; closed actions.
PI-QUA-4.3	Reference datasets (expected figures, typical cases) are versioned and replayed.	Repository of reference datasets with history.
PI-QUA-4.4	UAT lead time is measured and does not recurrently block delivery.	Lead times by delivery; stable or downward trend.
PI-QUA-5.1	Recurring causes are eliminated at the root (tooling, rule, training).	Rate decrease on a treated cause.
PI-QUA-5.2	A significant part of UAT is automated and replayable on demand.	Measured automated share (> 60% of core scenarios).
PI-QUA-5.3	Users trust delivered figures: few requests for manual verification.	Number of figure disputes declining over 12 months.
PI-QUA-5.4	Cost of quality (prevention, detection, failure) is tracked.	Annual report of the three components.

Detailed reading by level#

Level 1. Signs: “we’ll see in production”; users become testers. Gain: speed. Does not guarantee: business trust. Difficulty: 1/5. Effort: 1/5.

Level 2. Signs: a script exists, replayed under pressure. Gain: a minimal safety net. Does not guarantee that cases reflect business needs. Difficulty: 2/5. Effort: 2/5.

Level 3. Signs: “we know what done means”; UAT is a shared moment, not a trial. Gain: fewer end-of-project discussions, clear decisions. Does not guarantee absence of defects. Difficulty: 3/5. Effort: 3/5.

Level 4. Signs: escaped defect rate tracked, cause analyses, reference datasets. Gain: the same mistakes do not come back. Does not guarantee perceived quality (it may not align with counted defects). Difficulty: 4/5. Effort: 3/5.

Level 5. Signs: mostly automated UAT, exceptional critical defects, cost of quality managed. Gain: trust and speed. Does not guarantee that criteria are the right ones. Difficulty: 5/5. Effort: 5/5.

Partial achievement and rating#

Common rule: see README.

0.5: criteria written but only for major requirements; escaped defects counted without cause analysis.
1: a sample of deliveries and defects confirms the practice.
A partial 5: upfront criteria everywhere, systematic cause analysis, UAT automated at 40%; cost of quality never calculated. Score ≈ 4.5.

🧪 Example

Sales datamart: business UAT compares figures from a sample of 20 regions to the reference figure calculated manually. A versioned reference dataset (20 regions × 5 indicators, expected values) replayed at each delivery turns half a day of checking into 10 minutes. See UAT and Business Validation and Testing and Metrics Reliability.

⚠️ Pitfall

Formalistic UAT. Symptom: UAT is “OK” in 30 minutes, defects show up in production. Cause: testers without representative scenarios, or schedule pressure. Remedy: realistic data, business as an active participant, respected exit threshold.

⚠️ Pitfall

Goodhart: a “zero critical defects” threshold is bypassed by reclassifying as “major.” Have severity arbitrated by a written rule and review a sample.

✅ Good practice

Acceptance criteria in the form of concrete examples (“given X, when Y, then total for region Z equals 12,430”) are both readable by the business and directly automatable.

Move to the next level#

Possible means (optional)

1 → 2: UAT script, issue tool, minimal replay.
2 → 3: acceptance criteria before development, business presence, exit thresholds; see Scrum in Practice (Definition of Done) and fundamentals.
3 → 4: escaped defect indicator, cause analysis, versioned reference datasets (see Azure DevOps / TFS for test plans).
4 → 5: automation, prevention at the source, cost of quality; see Testing (SD).

🤖 AI

Can generate test cases and datasets, compare gaps between report versions, and rephrase criteria; does not judge whether a number is “good” for the business, and generated tests without review create fake coverage.