Matrix: Ingestion

How data from the source systems enters the product: knowledge of the sources, acquisition modes, change capture, contracts, schema evolution, security, monitoring.

≈ 12 min readReading

In brief

8 dimensions, levels 0 to 5, 112 criteria.
Tool: self-assessment, choosing "Ingestion".
Levels and criteria: original synthesis, references cited at the bottom of the page.

On this page

Overview
Knowing and reaching the sources
1. Source inventory
2. Acquisition modes
Flows and reliability
Contract and security
Sources and verification

Scope

The arrival of data: inventory of sources, acquisition modes (file, query, interface, change capture, stream), extraction windows, contract with the data provider, evolution of source schemas, secure access, monitoring of arrivals. What is done with the data once it has arrived falls under the "Storage" and "Processing" themes.

8 dimensions, six levels (0 to 5) and 112 observable criteria. To place your product: open the self-assessment for this theme. The calculation rules are here.

Overview#

Each cell is the level's signature for the dimension.

Dimension	0 · Handcrafted	1 · Locally tooled	2 · Shared	3 · Industrialized	4 · Integrated and governed	5 · Autonomous
Source inventory	Sources known from memory, by the person who connected them.	List of sources.	Sheet per source.	Inventory kept up to date.	Governed inventory.	Inventory fed automatically.
Acquisition modes	Files received or retrieved by hand.	Scheduled retrieval.	Direct access to sources.	Mode chosen according to the source.	Heterogeneous sources unified.	Continuous acquisition.
Change capture at the source	Full copy of the source every time.	Detection by date.	Detection mechanism per source.	Reliable capture.	Complete, verified capture.	Near-continuous capture.
Scheduling and extraction windows	Extractions launched whenever convenient, sometimes in the middle of the source's peak activity.	Fixed times.	Agreed window.	Managed window.	Consistent extractions.	Adaptive extraction.
Data contracts	No agreement: you take whatever arrives.	Written expectations.	Delivery agreement.	Verified contract.	Negotiated and monitored contract.	Executable contract.
Evolution of source schemas	A structural change is discovered through a processing error.	Changes noticed.	Changes detected.	Changes absorbed.	Changes anticipated.	Changes steered.
Access and connection security	Connections with personal or over-privileged accounts, passwords in scripts.	Dedicated accounts.	Minimal rights.	Managed connections.	Managed identities.	Monitored connections.
Arrival tracking and freshness	The absence of data is noticed when a report is empty or wrong.	Manual check.	Arrival recorded.	Freshness measured.	Freshness published.	Delays anticipated.

Link with the global model

Dimensions of the global model most concerned: Data sources and flows, Data quality, Security and compliance and Support and observability.

Knowing and reaching the sources#

Source inventory#

Do you know which sources feed the product, where they come from and who answers for them?

Level	Signature	Criteria to meet
0 · Handcrafted	Sources known from memory, by the person who connected them.	Starting point: no criteria.
1 · Locally tooled	List of sources.	The data sources are listed in a document. Each source has a name and a system of origin.
2 · Shared	Sheet per source.	Each source has a sheet: content, volume, frequency, access mode. Each source has an owner on the provider side and on the product side. The list is reviewed whenever a source is added or removed.
3 · Industrialized	Inventory kept up to date.	The inventory is kept in a shared tool, not in an isolated document. Each source is linked to the tables and reports it feeds. Inactive sources are identified and dealt with (removal or justification).
4 · Integrated and governed	Governed inventory.	Each source is classified (sensitivity, criticality, personal data). The impact of a source's unavailability or change is known before it happens. New access to a source goes through a request and approval process.
5 · Autonomous	Inventory fed automatically.	Sources and their structures are discovered and inventoried automatically. A gap between the inventory and reality (undeclared source, vanished source) is detected automatically. The criticality of each source is calculated from its actual usage.

Effect on the user

A question about the origin of a figure is answered in minutes, not days.

Acquisition modes#

By what means does the data arrive, and are they suited to each source?

Level	Signature	Criteria to meet
0 · Handcrafted	Files received or retrieved by hand.	Starting point: no criteria.
1 · Locally tooled	Scheduled retrieval.	File retrieval is scheduled rather than manual. The expected format and location of each file are defined.
2 · Shared	Direct access to sources.	Database sources are queried directly through a dedicated connection. Application interfaces (APIs) are used when they exist. Each acquisition mode is documented with its limits (volume, throughput, window).
3 · Industrialized	Mode chosen according to the source.	The acquisition mode is chosen according to volume and the desired freshness, by a written rule. The impact of extraction on the source system is measured and accepted by its owner. Connectors are standard and reused rather than rewritten for each source.
4 · Integrated and governed	Heterogeneous sources unified.	Sources of different kinds (databases, files, interfaces, streams) arrive through the same acquisition framework. Adding a source of a known type requires a description, not new code. Sources external to the organization follow specific rules (secure exchange, acknowledgment of receipt).
5 · Autonomous	Continuous acquisition.	Sources that warrant it feed the product as a continuous stream. The acquisition mode adapts to the availability of the source (automatic retry, failover). The cost and lead time of each mode are tracked and compared to make decisions.

Effect on the user

Each source is reached by the most reliable means, without improvisation specific to one person.

Flows and reliability#

Change capture at the source#

Can you tell what has changed in the source since last time?

Level	Signature	Criteria to meet
0 · Handcrafted	Full copy of the source every time.	Starting point: no criteria.
1 · Locally tooled	Detection by date.	Modified rows are identified by a modification date, when the source provides one. Sources without a modification date are identified.
2 · Shared	Detection mechanism per source.	A detection mechanism is chosen for each source (date, version, hash, log). Deletions in the source are detected or, failing that, identified as a gap. A full reload remains possible for verification.
3 · Industrialized	Reliable capture.	Native change capture (transaction log, change tracking) is used when the source offers it. The period covered by the available changes is known, and a missing period is detected. The load added to the source by the capture is measured.
4 · Integrated and governed	Complete, verified capture.	Inserts, updates and deletes are all carried into the product. A check regularly compares the content of the product with that of the source. Structural changes in the source are captured along with data changes.
5 · Autonomous	Near-continuous capture.	Changes arrive in the product with a measured and guaranteed delay. A capture delay is detected and flagged automatically. The end-to-end delay, from the source to the report, is published.

Effect on the user

Only what has changed is copied: the data is fresh and the source is not penalized.

Scheduling and extraction windows#

When do you extract, and does it get in the way of the source's users?

Level	Signature	Criteria to meet
0 · Handcrafted	Extractions launched whenever convenient, sometimes in the middle of the source's peak activity.	Starting point: no criteria.
1 · Locally tooled	Fixed times.	Extractions have fixed times. The times avoid the source's known activity peaks.
2 · Shared	Agreed window.	An extraction window is agreed with the owner of each source. Closing days and planned shutdowns of the source are taken into account. An extraction that exceeds its window is flagged.
3 · Industrialized	Managed window.	The duration of extractions is tracked and compared with the agreed window. Extractions from related sources are coordinated (same reference point in time). An unavailability of the source is handled by a planned retry.
4 · Integrated and governed	Consistent extractions.	Data from several sources is extracted at points in time that are consistent with one another. The load on the sources is spread out, and a ceiling is defined. The sources' calendar (closings, migrations, shutdowns) automatically feeds the scheduling.
5 · Autonomous	Adaptive extraction.	Launch adapts to the source's activity (extraction during detected quiet periods). A change in the source's habits is detected and suggests an adjustment. Windows are reviewed regularly based on their history.

Effect on the user

Extractions disturb neither the source nor its users, and the data arrives when it is expected.

Arrival tracking and freshness#

Do you know, source by source, whether the data has arrived and whether it is up to date?

Level	Signature	Criteria to meet
0 · Handcrafted	The absence of data is noticed when a report is empty or wrong.	Starting point: no criteria.
1 · Locally tooled	Manual check.	The arrival of data is checked by hand after each major load. The date of the latest data received is known.
2 · Shared	Arrival recorded.	Each arrival is recorded with its date, source and volume. The absence of a delivery at the expected time triggers an alert. An empty or abnormally small delivery is flagged.
3 · Industrialized	Freshness measured.	The freshness of each source is calculated against its expected rhythm. A delay threshold is defined for each source, with an owner for chasing it up. A delay in one source is passed on to the processes that depend on it.
4 · Integrated and governed	Freshness published.	The freshness of each dataset is visible to users. Delays are tracked over time, by provider. Delivery commitments are recalled to providers along with the gaps.
5 · Autonomous	Delays anticipated.	A probable delay is announced based on delivery habits. The users concerned are notified automatically of a data delay. Recurring delays from a source trigger an action with its provider.

Effect on the user

A delay or an absence of data is known before users notice it.

Contract and security#

Data contracts#

Have the provider and the consumer agreed on what is delivered, when and with what quality?

Level	Signature	Criteria to meet
0 · Handcrafted	No agreement: you take whatever arrives.	Starting point: no criteria.
1 · Locally tooled	Written expectations.	What is expected from each source (content, format, frequency) is written down. A contact is known on the provider side.
2 · Shared	Delivery agreement.	The expected structure (columns, types, keys) of each delivery is described. The frequency and the delivery deadline are agreed. A deviation from the agreement is reported to the provider through a defined process.
3 · Industrialized	Verified contract.	A data contract describes structure, quality rules, deadlines and owners. Each delivery is automatically checked against the contract. The contract is versioned and dated.
4 · Integrated and governed	Negotiated and monitored contract.	A contract change follows a procedure with advance notice and consumer approval. Compliance with the contract is measured and shared with the provider. The contract also specifies the security and sensitivity of the data.
5 · Autonomous	Executable contract.	The contract is a machine-readable definition, enforced by the processes. A non-compliant delivery is automatically isolated, without blocking the others. Contracts are published in a catalog that consumers can browse.

Effect on the user

A non-compliant delivery is a deviation from the contract, which is noticed and dealt with, not a surprise.

Evolution of source schemas#

What happens to the product when a source changes structure?

Level	Signature	Criteria to meet
0 · Handcrafted	A structural change is discovered through a processing error.	Starting point: no criteria.
1 · Locally tooled	Changes noticed.	A change in a source's structure is noticed at the next load. The fix is made by hand, case by case.
2 · Shared	Changes detected.	The expected structure is compared with the structure received at each load. A structural deviation blocks the load or puts it in quarantine, with a clear message. Added columns are distinguished from deletions and type changes.
3 · Industrialized	Changes absorbed.	A new column does not interrupt the loading of the expected columns. Unexpected columns are kept in the raw layer. Structural changes are recorded with their date.
4 · Integrated and governed	Changes anticipated.	Providers warn of changes, and a calendar lists them. The impact of a source change on tables and reports is known before it is applied. Both structures, old and new, can be processed during a transition period.
5 · Autonomous	Changes steered.	A routine structural adaptation propagates automatically through the product's layers. An undeclared change is detected and flagged before the next load. Past changes feed a stability measure for each source.

Effect on the user

A change on the provider's side is detected before it breaks a report.

Access and connection security#

How does the product connect to the sources, with what rights and what protection?

Level	Signature	Criteria to meet
0 · Handcrafted	Connections with personal or over-privileged accounts, passwords in scripts.	Starting point: no criteria.
1 · Locally tooled	Dedicated accounts.	Each connection uses an account dedicated to the product, not a person's account. The list of connection accounts is kept.
2 · Shared	Minimal rights.	Connection accounts have read-only rights, limited to the objects needed. Passwords and keys are kept out of scripts and versioned configuration files. Exchanges with the sources are encrypted.
3 · Industrialized	Managed connections.	Secrets are kept in a vault, with logged access. Secrets are rotated at a defined interval, without service interruption. Connection to internal sources goes through a private network or an allowlist of addresses.
4 · Integrated and governed	Managed identities.	Connections use managed identities, with no stored secret, when the source allows it. The rights of each connection are reviewed at regular intervals. Access to sensitive sources is logged and monitored.
5 · Autonomous	Monitored connections.	Abnormal use of a connection (volume, time, origin) is detected automatically. Excessive or unused rights are detected and proposed for removal. A compromised secret can be replaced automatically everywhere it is used.

Effect on the user

The product reads only what it needs, and the compromise of a connection remains limited.

Origin of the criteria

The dimensions are inspired by the references cited below; the levels and criteria are an original synthesis, not a reproduction of a framework. They are offered for discussion and adaptation, not as authoritative.

Sources and verification#

Kimball Group — ETL Architecture's 34 Subsystems — three extraction subsystems out of thirty-four, divided into extract, transform, load and manage
Microsoft Learn — What is change data capture (CDC)? (SQL Server) — capture of inserts, updates and deletes from the transaction log; validity window and cleanup of changes; handling of source structural changes
Bitol — Open Data Contract Standard (ODCS) — a data contract is an agreement between producer and consumers: structure, quality, service levels, owners, security
Microsoft Learn — What is the medallion lakehouse architecture? (Azure Databricks) — raw layer: original state preserved, incremental append, minimal validation; do not write to the validated layer directly from ingestion
DAMA International — DAMA-DMBOK2 — the "Data Integration and Interoperability" domain; paid framework, therefore not reviewed line by line