Matrix: Operations

How the product is run in production: service objectives, monitoring, incidents, root cause analysis, on-call, capacity, costs, continuity.

≈ 11 min readReading

In brief

8 dimensions, levels 0 to 5, 112 criteria.
Tool: self-assessment, choosing "Operations".
Levels and criteria: original synthesis, references cited at the bottom of the page.

On this page

Overview
Service delivered and incidents
Observability and organization
1. Monitoring and alerts
2. On-call and escalation
Capacity, costs and continuity
Sources and verification

Scope

The product's life in production: service objectives, monitoring and alerts, incident management, post-incident analysis, organization of support and on-call, capacity, operating costs, business continuity. Going live falls under the "Deployments" theme, and functional administration under the "Backoffice and tools" theme.

8 dimensions, six levels (0 to 5) and 112 observable criteria. To place your product: open the self-assessment for this theme. The calculation rules are here.

Overview#

Each cell is the level's signature for the dimension.

Dimension	0 · Handcrafted	1 · Locally tooled	2 · Shared	3 · Industrialized	4 · Integrated and governed	5 · Autonomous
Service objectives	No commitment: the service is "best effort".	Expectations expressed.	Quantified objectives.	Objectives tracked.	Objectives managed.	Objectives adapted to usage.
Monitoring and alerts	Problems are reported by users.	Manual monitoring.	Automatic monitoring.	Useful alerts.	End-to-end observability.	Predictive monitoring.
Incident management	Each incident is handled in an improvised way.	Incidents noted.	Incident procedure.	Roles and communication.	Incidents measured.	Accelerated recovery.
Root cause analysis and improvement	The incident is closed as soon as service is restored.	Cause noted.	Post-incident analysis.	Actions tracked.	Continuous improvement.	Systematic prevention.
On-call and escalation	Support rests on a single person, reachable when they can be.	Support identified.	Organized team.	On-call planned.	Support tiers.	Assisted support.
Capacity and forecasting	Capacity is discovered when the product saturates.	Resources known.	Utilization measured.	Forecasting.	Sizing under control.	Elastic capacity.
Costs and FinOps	Cost is discovered on the invoice.	Overall cost known.	Cost broken down.	Cost allocated.	Cost optimized.	Cost steered by value.
Business continuity and recovery	No plan: we'll deal with it when it happens.	Risks identified.	Written plan.	Plan tested.	Recovery proven.	Recovery automated.

Link with the global model

Dimensions of the global model most concerned: Availability and performance and Support and observability. For the maturity of cloud costs on the organization side, see the FinOps matrix.

Service delivered and incidents#

Service objectives#

Is the expected service level defined, measured and shared?

Level	Signature	Criteria to meet
0 · Handcrafted	No commitment: the service is "best effort".	Starting point: no criteria.
1 · Locally tooled	Expectations expressed.	Users' expectations (time of data availability, display time) are gathered. The product's operating hours are known.
2 · Shared	Quantified objectives.	A few service indicators (for example availability, freshness, display time) are defined with a quantified target. The target of each indicator is shared with users. The indicators are measured, at least after the fact.
3 · Industrialized	Objectives tracked.	The indicators are measured continuously and published. A distinction is made between the internal objective and the contractual commitment to the customer. A deviation from the objective triggers a review.
4 · Integrated and governed	Objectives managed.	An error budget (acceptable breach of the objective) arbitrates between enhancements and reliability. The availability of a cloud provider's service is distinguished from that of the product in the measurements. The objectives are reviewed with users at regular intervals.
5 · Autonomous	Objectives adapted to usage.	Objectives are differentiated according to the criticality of uses and customers. The indicators are calculated from the user's point of view, end to end. The attainment of objectives is tracked over several periods to decide on investments.

Effect on the user

Users and team know what is promised, and everyone can see whether the promise is kept.

Incident management#

How is an incident detected, qualified, handled and closed?

Level	Signature	Criteria to meet
0 · Handcrafted	Each incident is handled in an improvised way.	Starting point: no criteria.
1 · Locally tooled	Incidents noted.	Incidents are noted, with their date and effect. A person is designated for each incident.
2 · Shared	Incident procedure.	A procedure describes the steps: detection, qualification, handling, closure. The severity of an incident is defined by its effect on users. The users concerned are informed of the incident and of its expected duration.
3 · Industrialized	Roles and communication.	Roles are defined: incident lead, communication, resolution. A coordination channel and a message template are ready in advance. Incidents are recorded in a tool with their timeline.
4 · Integrated and governed	Incidents measured.	The time to detect and the time to recover from each incident are measured. Frequent cases have a written and proven runbook. Incident management exercises are run at regular intervals.
5 · Autonomous	Accelerated recovery.	Frequent incidents are resolved automatically, without intervention. The time to recover decreases from one period to the next, and this is tracked. The users concerned are notified automatically as soon as an incident is detected.

Effect on the user

An incident costs a limited, known duration, with someone steering and someone informing.

Root cause analysis and improvement#

Are lessons drawn from an incident so that it does not happen again?

Level	Signature	Criteria to meet
0 · Handcrafted	The incident is closed as soon as service is restored.	Starting point: no criteria.
1 · Locally tooled	Cause noted.	The apparent cause of each incident is noted. Incidents that recur are identified.
2 · Shared	Post-incident analysis.	Major incidents are the subject of a written analysis: timeline, cause, effect. Each analysis leads to actions with an owner and a due date. The analysis looks for the cause in the system, not the fault of a person.
3 · Industrialized	Actions tracked.	The actions decided on are tracked through to completion. Analyses are shared with the team and kept. Causes are classified in order to identify families of problems.
4 · Integrated and governed	Continuous improvement.	Families of causes feed the list of product enhancements. The effectiveness of actions is verified by the drop in incidents of the same nature. Analyses are reviewed at regular intervals to detect trends.
5 · Autonomous	Systematic prevention.	Each identified cause gives rise to an automated check to detect it in the future. Deliberate failure tests prove the product's resilience. Lessons learned are reused when designing new features.

Effect on the user

An incident does not happen twice for the same reason.

Observability and organization#

Monitoring and alerts#

Do you know there is a problem before the users do?

Level	Signature	Criteria to meet
0 · Handcrafted	Problems are reported by users.	Starting point: no criteria.
1 · Locally tooled	Manual monitoring.	A person regularly checks the state of the product. The list of items to check is written down.
2 · Shared	Automatic monitoring.	Automatic probes check the availability of the main components. An alert is sent to a designated person or team. Each alert specifies the component, the time and the severity.
3 · Industrialized	Useful alerts.	Alerts are classified by severity, with an associated response time. Useless or repetitive alerts are reviewed to keep a small number of alerts that matter. A dashboard shows the state of the product in real time.
4 · Integrated and governed	End-to-end observability.	Logs, metrics and traces are centralized and correlatable. A probe runs end-to-end journeys (load, calculation, display) like a user. A health model links components to their effect on the service.
5 · Autonomous	Predictive monitoring.	A drift is detected before the incident (saturation, slowdown, delay). A first automatic response is triggered for known incidents. Alerts are deduplicated and grouped automatically by probable cause.

Effect on the user

The problem is detected, qualified and assigned without a user having to report it.

On-call and escalation#

Who responds, at what time, with what authority, and to whom do you escalate?

Level	Signature	Criteria to meet
0 · Handcrafted	Support rests on a single person, reachable when they can be.	Starting point: no criteria.
1 · Locally tooled	Support identified.	A single address or number receives support requests. Response hours are announced to users.
2 · Shared	Organized team.	Several people can handle a request. Requests are recorded with their priority. An escalation rule says whom to turn to when the deadline is exceeded.
3 · Industrialized	On-call planned.	On-call is planned for critical periods, with a known calendar. The on-call person has the access and procedures needed. The on-call load is spread out and tracked so that it is sustainable.
4 · Integrated and governed	Support tiers.	Support is organized in tiers, with documentation for each. Response and resolution times are measured by priority. A contact is designated for each important customer.
5 · Autonomous	Assisted support.	Common requests receive an automatic response based on the knowledge base. Requests are classified and distributed automatically. Support indicators feed staffing and tooling decisions.

Effect on the user

A problem outside office hours has someone to turn to, without relying on the goodwill of a single person.

Capacity, costs and continuity#

Capacity and forecasting#

Does the product have the resources needed for its current and future load?

Level	Signature	Criteria to meet
0 · Handcrafted	Capacity is discovered when the product saturates.	Starting point: no criteria.
1 · Locally tooled	Resources known.	The resources used (compute, memory, storage, licenses) are listed. The utilization level is consulted from time to time.
2 · Shared	Utilization measured.	Resource utilization is measured continuously. An alert is triggered before saturation. Peak periods (closings, month-ends) are identified.
3 · Industrialized	Forecasting.	A twelve-month forecast of needs is drawn up from observed growth. The number of users and the volume of data are tracked as drivers of the load. The forecast is reviewed at every major change (new customer, new source).
4 · Integrated and governed	Sizing under control.	The sizing of each environment results from load tests. Safety margins are defined and known to the budget. Resources of non-production environments are adjusted or shut down when not in use.
5 · Autonomous	Elastic capacity.	Capacity adjusts automatically to the load within defined limits. Peaks are absorbed without manual intervention. The gap between forecast capacity and capacity used is tracked to refine the forecast.

Effect on the user

Growth in usage does not degrade the service, and needs are known before they become urgent.

Costs and FinOps#

Do you know what the product costs, to whom to allocate it and how to optimize it?

Level	Signature	Criteria to meet
0 · Handcrafted	Cost is discovered on the invoice.	Starting point: no criteria.
1 · Locally tooled	Overall cost known.	The total monthly cost of the product is known. The main expense items are identified.
2 · Shared	Cost broken down.	Resources are tagged by environment and by customer or use. The cost per environment is known every month. A budget alert is triggered as a threshold approaches.
3 · Industrialized	Cost allocated.	Cost is allocated to owners, who see it regularly. A unit cost is calculated (per user, per customer, per report). Unused or oversized resources are identified.
4 · Integrated and governed	Cost optimized.	Regular optimizations are carried out (shutdown when not in use, rightsizing, term commitments). Architecture decisions take their operating cost into account. Technical, finance and business teams meet at a fixed interval about costs.
5 · Autonomous	Cost steered by value.	Cost anomalies are detected automatically. Cost is set against the value delivered in order to decide on investments. Validated optimizations are applied automatically within defined limits.

Effect on the user

The cost of the product is known, discussed with those who trigger it, and optimized without degrading the service.

Business continuity and recovery#

If part of the product goes down, how do you recover, and how quickly?

Level	Signature	Criteria to meet
0 · Handcrafted	No plan: we'll deal with it when it happens.	Starting point: no criteria.
1 · Locally tooled	Risks identified.	The possible major outages (site, platform, data, account) are listed. The indispensable components are identified.
2 · Shared	Written plan.	A recovery plan describes the steps, the roles and the order of restoration. Recovery time objectives and acceptable data loss are set. The plan is accessible even if the production platform is unavailable.
3 · Industrialized	Plan tested.	Part of the plan is tested at regular intervals, outside production. Test results are recorded and the plan is corrected accordingly. External dependencies (identity, network, suppliers) are taken into account in the plan.
4 · Integrated and governed	Recovery proven.	A full exercise measures the actual recovery time against the objective. A partial recovery (one component, one region) is possible without declaring the major outage. The plan is reviewed at least twice a year and at every architecture change.
5 · Autonomous	Recovery automated.	Failover to a backup site is automated or triggered by a single action. Surprise exercises or failure tests prove the plan without advance notice. The return to normal is prepared as carefully as the failover.

Effect on the user

A major outage has a plan, and the plan has been tried out.

Origin of the criteria

The dimensions are inspired by the references cited below; the levels and criteria are an original synthesis, not a reproduction of a framework. They are offered for discussion and adaptation, not as authoritative.

Sources and verification#

Google — Site Reliability Engineering: Service Level Objectives — indicator (SLI), objective (SLO), contractual commitment (SLA); few, well-chosen indicators; error budget rather than 100%
Microsoft Learn — Operational excellence (Azure Well-Architected Framework) — principles: DevOps culture, development standards, observability, automation, safe deployment; incident management
Microsoft Learn — Architecture strategies for disaster recovery (Azure Well-Architected Framework, RE:09) — disaster thresholds, tested and documented plan, scheduled and surprise exercises, plan review about every six months
FinOps Foundation — The FinOps Framework — three phases (inform, optimize, operate) and six principles; progressive "crawl, walk, run" approach
DORA — Software delivery performance metrics — time to restore service after a failed deployment and change failure rate, among the five metrics