Matrix: Deployments

How the product reaches production and customers: environments, build, delivery pipeline, schema changes, configuration, strategies, checks, delivery to customers.

≈ 11 min readReading

In brief

8 dimensions, levels 0 to 5, 112 criteria.
Tool: run an assessment by choosing "Deployments".
Levels and criteria: original synthesis, references cited at the bottom of the page.

On this page

Overview
Delivery pipeline
Environments and configuration
1. Environments
2. Configuration and secrets per environment
Deployment safety
Sources and verification

Scope

The routing of the product to production and to customers: environments, build and continuous integration, delivery pipeline, schema and data changes, configuration and secrets per environment, deployment and rollback strategies, post-deployment checks, delivery to customers. The life of the installed product belongs to the "Application maintenance" theme, day-to-day production to the "Operations" theme.

8 dimensions, six levels (0 to 5) and 112 observable criteria. To place your product: open the assessment for this theme. The scoring rules are here.

Overview#

Each cell is the level's signature for the dimension.

Dimension	0 · Handcrafted	1 · Locally tooled	2 · Shared	3 · Industrialized	4 · Integrated and governed	5 · Autonomous
Environments	A single environment: development and testing happen in production.	Test environment.	Staged environments.	Representative environments.	On-demand environments.	Ephemeral environments.
Build and continuous integration	The build is done on a developer's workstation.	Scripted build.	Automatic build.	Verified continuous integration.	Extended checks.	Optimized build.
Delivery pipeline and promotion	Deliverables are copied by hand from one environment to another.	Written procedure.	Scripted deployment.	Automated pipeline.	Complete pipeline.	Continuous delivery.
Schema and data changes	Structure changes are applied by hand in production.	Scripts kept.	Scripts versioned.	Migrations tested.	Migrations compatible.	Zero-downtime migrations.
Configuration and secrets per environment	Environment-specific values written in the code, passwords in scripts.	Values identified.	Configuration separated.	Configuration versioned.	Secrets managed.	Configuration steered.
Deployment strategies and rollback	A failed deployment is repaired by moving forward, under pressure.	Backup before deployment.	Rollback described.	Rollback proven.	Progressive deployment.	Automatic rollback.
Post-deployment checks	A deployment is known to have succeeded when nobody complains.	Manual verification.	Basic checks.	Automatic checks.	End-to-end checks.	Enhanced monitoring.
Delivery and installation at customers	Each installation is manual work, specific to the customer.	Installation procedure.	Installation kit.	Reliable installation.	Delivery managed.	Delivery automated.

Link with the overall model

Dimensions of the overall model that are most relevant: Industrialization and deployment and Testing and fixes; also Availability and performance and Security and compliance.

Delivery pipeline#

Build and continuous integration#

Is each change built and verified automatically, the same way every time?

Level	Signature	Criteria to meet
0 · Handcrafted	The build is done on a developer's workstation.	Starting point: no criteria.
1 · Locally tooled	Scripted build.	The build is described by a script. The source code is kept in a shared repository.
2 · Shared	Automatic build.	Each change triggers a build on a dedicated server. A failed build is reported to the team. The result of the build is a versioned deliverable, which is kept.
3 · Industrialized	Verified continuous integration.	The build runs the automatic tests (unit, integration). A change that fails the tests cannot be integrated. The product documentation is built along with the rest.
4 · Integrated and governed	Extended checks.	The build also runs style, security and dependency checks. End-to-end tests, including user journeys, run automatically. Build duration is tracked and kept short enough to be waited for.
5 · Autonomous	Optimized build.	Only the elements affected by a change are rebuilt and retested. An unstable test is detected and isolated automatically. The provenance of each deliverable is verifiable: who, when, from which code.

Effect on the user

A broken change is detected within the hour, not at delivery.

Delivery pipeline and promotion#

Is moving from one environment to another automated, traced and validated?

Level	Signature	Criteria to meet
0 · Handcrafted	Deliverables are copied by hand from one environment to another.	Starting point: no criteria.
1 · Locally tooled	Written procedure.	The procedure for going to production is written down. Each production release is recorded with its date.
2 · Shared	Scripted deployment.	Deployment is run by a script or a pipeline, identical for all environments. The version deployed in each environment is known. The same version that was tested is promoted to the next environment.
3 · Industrialized	Automated pipeline.	A delivery pipeline chains build, tests and deployment across environments. Validation steps (tests, human approval) are conditions for moving to production. Each deployment is traced: who, what, when, with what result.
4 · Integrated and governed	Complete pipeline.	All elements of the product (code, database structure, reports, configuration) are deployed by the same pipeline. Differences between two environments can be compared before a deployment. A deployment can be triggered at any time by an authorized person, without waiting for a special window.
5 · Autonomous	Continuous delivery.	A validated change can reach production automatically. The lead time between the validation of a change and its release to production is a few hours. The delivery pipeline is itself versioned, tested and fixed like code.

Effect on the user

What reaches production is the very thing that was tested, with no handling in between.

Delivery and installation at customers#

How is the product delivered, installed and updated at each customer?

Level	Signature	Criteria to meet
0 · Handcrafted	Each installation is manual work, specific to the customer.	Starting point: no criteria.
1 · Locally tooled	Installation procedure.	An installation procedure is written. The prerequisites of the customer environment are listed.
2 · Shared	Installation kit.	A single installation kit delivers all elements of the product. Installation is scripted and parameterized, with no modification of the kit. Installation checks the prerequisites before starting and stops with a clear message if they are missing.
3 · Industrialized	Reliable installation.	An interrupted installation can be restarted without corrupting what exists. The result of the installation is checked and recorded. The same kit serves for a first installation and for an update.
4 · Integrated and governed	Delivery managed.	Each delivered kit is tested by a complete installation on a reference environment before publication. Customer-specific customizations are replayed after each installation. The version installed at each customer and the date of the last installation are known.
5 · Autonomous	Delivery automated.	Installations and updates at customers are triggered remotely, with their agreement, by the same pipeline. A failed installation is rolled back automatically and reported to support. The delay between the publication of a version and its installation at customers is measured.

Effect on the user

Installing or updating the product is a predictable operation, which the customer or support can carry out without surprises.

Environments and configuration#

Environments#

Are there separate environments, similar to production, for development and testing?

Level	Signature	Criteria to meet
0 · Handcrafted	A single environment: development and testing happen in production.	Starting point: no criteria.
1 · Locally tooled	Test environment.	An environment separate from production exists for trying out changes. Access to production and to test is separated.
2 · Shared	Staged environments.	Environments are staged (development, test, production) with a defined role for each. The structure of environments is identical: same components, same versions. Each environment has an owner.
3 · Industrialized	Representative environments.	A test environment contains representative data, masked if sensitive. Environments are described as code and can be recreated. Configuration differences between environments are listed and justified.
4 · Integrated and governed	On-demand environments.	A complete environment can be created in less than a day for a trial or a reproduction. Temporary environments are deleted automatically after use. An environment reproduces a given customer installation to diagnose a problem.
5 · Autonomous	Ephemeral environments.	Each significant change has its own environment, created and destroyed automatically. The cost of environments is tracked and limited automatically. The conformity of each environment to its description is verified continuously.

Effect on the user

What works in test works in production, because the two resemble each other and nobody tests in production.

Configuration and secrets per environment#

Is the configuration specific to each environment separated from the code and protected?

Level	Signature	Criteria to meet
0 · Handcrafted	Environment-specific values written in the code, passwords in scripts.	Starting point: no criteria.
1 · Locally tooled	Values identified.	The values that differ from one environment to another are listed. Passwords do not appear in shared documents.
2 · Shared	Configuration separated.	The configuration of each environment is separate from the deliverable. The same deliverable is deployed in all environments. Secrets are stored in a vault, never in the code repository.
3 · Industrialized	Configuration versioned.	The configuration of each environment is versioned, without the secrets. Missing or invalid settings block the deployment with a clear message. A configuration change follows the same validation process as a code change.
4 · Integrated and governed	Secrets managed.	Secrets are supplied to the deployment by the vault, with rights limited to what is needed. Secrets are rotated on a schedule, without service interruption. Access to secrets is logged.
5 · Autonomous	Configuration steered.	The actual configuration of each environment is continuously compared with the intended configuration. A drift is corrected or flagged automatically. The effect of a configuration change is simulated before it is applied.

Effect on the user

The same deliverable is deployed everywhere; only the configuration and the secrets change, and they do not leak.

Deployment safety#

Schema and data changes#

Are changes to the database structure and to data applied in a safe, replayable and reversible way?

Level	Signature	Criteria to meet
0 · Handcrafted	Structure changes are applied by hand in production.	Starting point: no criteria.
1 · Locally tooled	Scripts kept.	Each structure change is written in a script that is kept. Scripts are run in the intended order.
2 · Shared	Scripts versioned.	Structure change scripts are versioned with the code. The structure version of each database is recorded in the database. A script runs only once, or can be replayed with no harmful effect.
3 · Industrialized	Migrations tested.	Migrations are tested on a copy of real data before production. The duration and locking of each migration are measured. A long migration is split up to limit the interruption.
4 · Integrated and governed	Migrations compatible.	Incompatible changes follow the sequence: add, migrate the usages, remove. The old and the new version of the product work with the structure during the transition. Each migration has a tested rollback plan, or a verified backup taken just before.
5 · Autonomous	Zero-downtime migrations.	Migrations are applied without interrupting the service. Differences in structure between the expected and the actual are detected automatically. Migrated data is compared automatically before and after.

Effect on the user

A structure change loses no data and blocks no report along the way.

Deployment strategies and rollback#

Can a deployment be progressive, and can it be rolled back quickly?

Level	Signature	Criteria to meet
0 · Handcrafted	A failed deployment is repaired by moving forward, under pressure.	Starting point: no criteria.
1 · Locally tooled	Backup before deployment.	A backup is made before each deployment to production. Deployment slots avoid moments that are critical for users.
2 · Shared	Rollback described.	Each deployment has a written rollback procedure. The previous version remains available after a deployment. Users are warned of a planned interruption.
3 · Industrialized	Rollback proven.	Rollback is tested at least once per version. The conditions that trigger a rollback are defined in advance. A deployment can be applied to a single customer or a single group before the others.
4 · Integrated and governed	Progressive deployment.	A new version is first exposed to a small group, then widened according to the results. The old and the new version coexist for the duration of a switchover (for example two environments in parallel). New features can be switched on or off without a new deployment.
5 · Autonomous	Automatic rollback.	A deployment that degrades service indicators is rolled back automatically. The time to restore service after a failed deployment is measured and remains very short. Routine deployments require no visible interruption.

Effect on the user

A failed deployment is undone in minutes, with no lasting consequence for users.

Post-deployment checks#

Is it verified, right after a deployment, that the product really works?

Level	Signature	Criteria to meet
0 · Handcrafted	A deployment is known to have succeeded when nobody complains.	Starting point: no criteria.
1 · Locally tooled	Manual verification.	A few checks are done by hand after a deployment. The list of checks is written down.
2 · Shared	Basic checks.	A set of quick checks (availability, login, a key report, a data load) runs after each deployment. The result of the checks is recorded with the deployment. A failed check triggers an immediate alert.
3 · Industrialized	Automatic checks.	Post-deployment checks are fully automatic. They compare reference figures before and after the deployment. A failure of the checks blocks or cancels promotion to the next stage.
4 · Integrated and governed	End-to-end checks.	Complete journeys (from source to report) are replayed after each deployment. Access rights are verified by role after each deployment. Checks are enriched after each incident caused by a deployment.
5 · Autonomous	Enhanced monitoring.	A period of enhanced monitoring follows each deployment, with stricter thresholds. Checks also run continuously in production. The change failure rate is tracked and decreasing.

Effect on the user

A deployment that does not work is noticed right away, by the team, before the users.

Origin of the criteria

The dimensions are inspired by the references cited below; the levels and criteria are an original synthesis, not a reproduction of a framework. They are offered for discussion and adaptation, not as authoritative.

Sources and verification#

DORA — Software delivery performance metrics — lead time for changes, deployment frequency, time to restore service after a failed deployment, change failure rate; speed and stability are not opposed
Microsoft Learn — Overview of Fabric deployment pipelines — development, test, production stages; comparing content between stages; pairing of items; automation through programming interfaces
Martin Fowler — Parallel Change — expand, migrate, then contract, for interface or database schema changes that are compatible at every step
Microsoft Learn — Operational excellence (Azure Well-Architected Framework) — principles: automate, adopt safe deployment practices, infrastructure described as code, reliable supply chain
The Twelve-Factor App — III. Config — strictly separate configuration from code