i

Description

Un microscope posé sur une paillasse de laboratoire

Source

Tomiwa94 · original page

Licence

CC0 Public domain

Original photo, unretouched. The background is a version of it stylised by the site.

Methods Specific to Data Projects

General methods (predictive, agile, and lean) say almost nothing about what makes a data project unique: data is an unstable material, its condition is only known once you…

In brief

• General methods (predictive, agile and lean) say almost nothing about what makes a data project unique: data is an unstable material, its condition is only known once you open it, and the “truth” of the result is negotiated with the business.
• You'll learn how to position and compare CRISP-DM, ASUM-DM, TDSP, the Kimball lifecycle, DataOps, data products / Data Mesh, MLOps/AIOps, DAMA-DMBOK, then the five project families (analytics/machine learning, BI/datamart, migration, governance, quality).
• You'll learn how to choose: which method for a BI/datamart project, which one for a machine learning project, which one for a quality initiative.
• Each method is rated /5 according to the file criteria (see RATING); ratings are contextual expert assessments.
• Methods are combined: the general case is “Kimball for modeling + Scrum/Kanban for steering + DataOps for industrialization”.

On this page

1. 1. Why a method specific to data
   1. 🟢 Novice — what changes when data is the material
   2. 🟡 Practical — the five families of data projects
   3. 🔴 Expert — the shifted “cone of uncertainty”
2. 2. CRISP-DM
   1. 🟢 Novice — the model
   2. 🟡 Practical — how to use it today
   3. 🔴 Expert — limits and extensions
3. 3. ASUM-DM
   1. 🟢 Novice — an IBM extension
   2. 🟡 Practical — when to cite it
   3. 🔴 Expert — critical reading
4. 4. TDSP (Team Data Science Process)
   1. 🟢 Novice — Microsoft's method
   2. 🟡 Practical — what you can reuse without adopting it
   3. 🔴 Expert — limits
5. 5. The Kimball Lifecycle
   1. 🟢 Novice — the data warehouse reference
   2. 🟡 Practical — ideas that still matter
   3. 🔴 Expert — Kimball vs. modern practices
6. 6. DataOps
   1. 🟢 Novice — applying the DevOps mindset to data
   2. 🟡 Practical — two pipelines, two loops
   3. 🔴 Expert — measuring DataOps
7. 7. Data products and Data Mesh
   1. 🟢 Novice — thinking in “product” terms
   2. 🟡 Practical — what this changes for project management
   3. 🔴 Expert — Data Mesh: when and when not
8. 8. MLOps and AIOps in brief
   1. 🟢 Novice — two acronyms, two meanings
   2. 🟡 Practical — what changes in project management
   3. 🔴 Expert — specific pitfalls
9. 9. DAMA-DMBOK
   1. 🟢 Novice — the data management framework
   2. 🟡 Practical — how to use it without getting lost in it
   3. 🔴 Expert — critical use
10. 10. Data governance managed as a project
    1. 🟢 Novice — what it is, what it is not
    2. 🟡 Practical — start it as a value-driven project
    3. 🔴 Expert — why it fails, how we know
11. 11. The data migration project
    1. 🟢 Novice — why it is a project in its own right
    2. 🟡 Practical — phases and controls
    3. 🔴 Expert — reconciliation and idempotence
12. 12. The BI / datamart project
    1. 🟢 Novice — the most common form
    2. 🟡 Practical — a recommended composite method
    3. 🔴 Expert — structuring trade-offs
13. 13. Data quality projects
    1. 🟢 Novice — fix the cause, not just the symptom
    2. 🟡 Practical — the quality loop
    3. 🔴 Expert — trade-offs
14. 14. Comparison Table and Ratings
15. 15. Pitfalls and good practices
16. 16. AI Contribution
17. 17. See also

Level: 🟢→🔴

1. Why a method specific to data#

🟢 Novice — what changes when data is the material#

In a traditional software project, you build a feature; you roughly know when it will be finished. In a data project, you build a result that depends on a material you don't control: source data. Three differences structure this entire file.

1. The material is unknown at the start. Until you've profiled the sources (NULL rate, duplicates, value domains, cross-system consistency), you don't know whether the request is feasible. Uncertainty is structural and concentrated at the beginning of the project.
2. The output is a number, not a feature. A feature is correct or not against a specification. A number (“1,284 visits in week 38”) is correct if the business recognizes it — which assumes reconciliation with a reference that may itself be debatable. Acceptance testing is therefore a negotiation of definitions, not just verification.
3. The product lives on after delivery. Sources drift, volumes grow, business definitions evolve. A “finished” delivery that is not operated and monitored degrades within a few months.

🟡 Practical — the five families of data projects#

🔴 Expert — the shifted “cone of uncertainty”#

Predictive methods assume uncertainty decreases with analysis. In data, a large share of uncertainty is only reduced through contact with real data, often after the official “analysis” phase. Consequences: (a) a profiling spike (1 to 5 days) must precede any date commitment; (b) estimation is done in ranges and revised after profiling; (c) sprint reviews on a data project must show reconciled numbers, not mockups. This point is developed in Scrum and data and measured by the 03 Data Project Management matrix.

2. CRISP-DM#

🟢 Novice — the model#

CRISP-DM (CRoss-Industry Standard Process for Data Mining) is a process model published as version 1.0 in 1999-2000 by an industry consortium (including SPSS, NCR, Daimler-Chrysler, OHRA). For years, it remained the most cited process model in data mining practitioner surveys; it has never had a single official successor.

It has six phases, iterative and not strictly sequential:

🟡 Practical — how to use it today#

CRISP-DM is a thinking framework, not a management framework: it doesn't say who, how much, or when. Its value is forcing you to start with the business and close the loop (the return arrows are the core of the model: you often go back from “data” to “business” because the objective was not realistic).

Effective use: make it the backbone of each cycle in an agile backlog — each analytics epic goes through the six phases, but in short slices. A sprint can cover “2→3” (profiling + preparation) on a limited scope.

🔴 Expert — limits and extensions#

• No post-deployment lifecycle: monitoring, retraining, and drift are barely mentioned → hence MLOps (section 8).
• No teamwork layer: no roles, no tooling, no version control system for data and models → hence TDSP and ASUM-DM.
• Documentation-heavy in version 1.0: useful for audits, heavy for short iterations. Adapt it: produce only deliverables that support a decision.
• Relevance beyond ML: phase 2-3 (understanding and preparation) is universal and appears in any BI or migration initiative; the rest is specific to predictive modeling.

Ratings (context: team of 4 to 8 people, analytics or ML project):

3. ASUM-DM#

🟢 Novice — an IBM extension#

ASUM-DM (Analytics Solutions Unified Method for Data Mining/Predictive Analytics) is a method published by IBM (2015) that extends CRISP-DM. It keeps CRISP-DM's analytical phases and adds a project management and operations layer. It is less widespread than CRISP-DM and mostly associated with the IBM ecosystem; to be verified regarding its current maintenance status, since its public documentation has become hard to locate.

What it brings, in spirit:

• additional phases upstream and downstream: design, implementation, operations (operations), beyond simple deployment;
• a project dimension: governance, risk management, roles;
• the idea that analytical phases (understand, prepare, model, evaluate) are carried out in an agile and iterative way, while the project layer remains managed in a more traditional way.

🟡 Practical — when to cite it#

In a response or scoping document for a large enterprise client, ASUM-DM is reassuring: it shows that the cycle does not stop at deployment. In practice, a team that combines CRISP-DM and an agile backlog with explicit operations phases gets the same result without formally adopting ASUM-DM.

🔴 Expert — critical reading#

The real value is conceptual: separating the analytical loop (short, iterative) from the project loop (long, milestone-driven). This is the same separation as in an “agile within predictive” hybrid (see Choosing Your Method and Hybridizing). If your context is not IBM, keep the separation, not the name.

4. TDSP (Team Data Science Process)#

🟢 Novice — Microsoft's method#

The Team Data Science Process is a methodology published by Microsoft for data science teams. It offers a life cycle similar to CRISP-DM, a standardized repository structure, roles, and integration with Azure tooling (Azure DevOps for task and code tracking). Its life cycle has five stages: business understanding, data acquisition and understanding, modeling, deployment, customer acceptance testing.

Its uniqueness is not the cycle, but team industrialization: a project template (folder structure, report templates), a backlog linked to the repository, naming conventions. The maintenance status of TDSP documentation is to be verified: the reference page exists on Microsoft Learn, but Microsoft's focus has shifted toward MLOps and Fabric/Azure Machine Learning approaches (status as of 2026-10, not re-verified in detail).

🟡 Practical — what you can reuse without adopting it#

• A standard project tree: data/ (raw, cleaned), code/, models/, reports/, docs/. It avoids each project reinventing its organization.
• A standardized data quality report at stage 2 (reproducible profiling, not copy-pasted screenshots).
• A definition of “done” that includes customer acceptance testing: the model is delivered only when the customer has accepted it based on written criteria.

🔴 Expert — limits#

TDSP assumes a data science team with notebooks, experiments, and compute infrastructure: a framework heavier than necessary for a team of 2 BI developers. The acceptance cycle is valuable but does not address drift after delivery — MLOps must be added.

5. The Kimball Lifecycle#

🟢 Novice — the data warehouse reference#

Ralph Kimball and his coauthors described in The Data Warehouse Lifecycle Toolkit (“the Kimball Toolkit,” 2nd ed. 2008) a dimensional data warehouse project lifecycle. It is the historical BI method: it is centered on the business, dimensional modeling (facts and dimensions), and incremental delivery by business process.

Three parallel paths (technology, data, BI applications) converge toward deployment; the whole effort is framed by planning and requirements gathering, and followed by a growth cycle.

🟡 Practical — ideas that still matter#

🔴 Expert — Kimball vs. modern practices#

• Agile and Kimball are compatible: a Kimball batch can be broken down into stories (one dimension, one fact, one report). The classic Agile risk is delivering only reports without the conformed dimension — model debt accumulates. Keep the bus matrix as your roadmap.
• ELT / lakehouse: Kimball's technical argument (model optimized for a relational engine) is still valid for the presentation layer; dimensional modeling appears again in the Power BI semantic model. What changes: storage (parquet, Delta) and when transformation happens.
• Data Vault and Inmon: competing or complementary approaches (Inmon: normalized warehouse first; Data Vault: modeling for auditability and source flexibility). A common pattern: Data Vault or normalized zone in integration, Kimball star in presentation.
• Historization: slowly changing dimensions (SCD type 1, 2, etc.), to be decided by attribute with the business — not by default. See the data matrix modeling theme.

6. DataOps#

🟢 Novice — applying the DevOps mindset to data#

DataOps refers to a set of practices that apply Agile, DevOps, and statistical process control (Lean manufacturing) ideas to the data production chain. The goal is to shorten lead time between a data request and its reliable delivery, while reducing production errors. The term was popularized around 2014-2015; a DataOps Manifesto was later published, with 18 principles.

The manifesto (read it at dataopsmanifesto.org) states values and principles. The most structuring ones, rephrased in my words:

The fact of versioning data or its samples and creating disposable environments distinguishes DataOps from classic DevOps: data, unlike code, has volume, sensitivity, and state.

🟡 Practical — two pipelines, two loops#

A productive mental model: a data chain contains two pipelines that intersect.

• The value pipeline (production): sources → transformations → model → reports, run every day. It must be robust, monitored, replayable.
• The innovation pipeline (development): idea → development → test → deployment. It must be short and automated.

The two control loops: production quality (data tests at each run, alerts) and development quality (automated tests on every change, continuous integration). Technical detail is in CI/CD for data: foundations, datamarts, pipelines, and reports.

🔴 Expert — measuring DataOps#

Relevant indicators (to adapt): production lead time for a change; deployment frequency; data incident rate per month; time to detect then to fix; share of time spent on unplanned work (rework, fixes). They extend DORA metrics (see CI/CD: Continuous Integration and Continuous Delivery) with data-specific measures, developed in data matrix theme 06.

7. Data products and Data Mesh#

🟢 Novice — thinking in “product” terms#

A data product (data product) is a dataset — or a service that provides one — designed, maintained, and offered as a product to consumers: with an owner, documentation, a service level, and measured quality. The core idea is to move from “a project that delivers tables” to “a team responsible for a product over time.”

Data Mesh is a paradigm described by Zhamak Dehghani (2019-2020 articles, Data Mesh book in 2022) that proposes, in response to the limits of a central warehouse, four principles:

1. Domain ownership: business teams that produce data are responsible for it;
2. Data as a product: each domain publishes reliable, discoverable products;
3. Self-service platform: a platform team provides shared tooling;
4. Federated computational governance: global rules, applied automatically, decided collectively.

🟡 Practical — what this changes for project management#

The data contract (schema, semantics, freshness, expected quality, evolution rules) is the object that formalizes the producer-consumer relationship. It is handled like an API interface: any contract break is an announced major version change.

🔴 Expert — Data Mesh: when and when not#

Data Mesh solves an organizational scale problem (dozens of domains, one central bottleneck). It fails when the organization lacks data skills in domains, lacks a platform, or lacks appetite for federated governance. For a single BI team of a few people, adopting the “product” vocabulary (owner, contract, SLO) makes sense; adopting the Data Mesh architecture does not. The ratings below apply to a mid-sized organization (200 to 2,000 people).

8. MLOps and AIOps in brief#

🟢 Novice — two acronyms, two meanings#

• MLOps: practices to industrialize a machine learning model: version data, code, and model; automate training, validation, deployment; monitor performance and drift; retrain. It is “DevOps for models.”
• AIOps: an ambiguous term. In its original meaning (Gartner), using AI/ML to operate IT (alert correlation, anomaly detection, incident automation). Do not confuse it with “operating AI systems” (which falls under MLOps, and for generative AI, labels such as LLMOps).

🟡 Practical — what changes in project management#

A model is never “finished”: its behavior depends on current data. Therefore:

• an explicit run budget (monitoring, retraining) from the framing stage;
• two-stage acceptance testing: offline performance, then production performance (“shadow” mode or pilot);
• a model registry (version, training data, metrics, owner);
• alert thresholds for drift (population, variable distribution, measured performance when real outcomes are known).

🔴 Expert — specific pitfalls#

For a BI developer, a reasonable entry point is modest: a score produced by a scheduled job, stored in a table, consumed in a report, with its version and training date exposed. That is already minimal MLOps. See also AI in Planning and Version Control Tools.

9. DAMA-DMBOK#

🟢 Novice — the data management framework#

The DMBOK (Data Management Body of Knowledge), published by DAMA International, is the framework for the data management discipline. Its 2nd edition (DMBOK2, 2017) structures the field into eleven knowledge areas, often represented as a “wheel” with data governance at the center:

1. Data governance
2. Data architecture
3. Data modeling and design
4. Data storage and operations
5. Data security
6. Data integration and interoperability
7. Document and content management
8. Reference data and master data
9. Data warehousing and business intelligence
10. Metadata management
11. Data quality

It is not a project method: it is a vocabulary and a map (who does what, which activities, which deliverables, which roles). It serves as an audit framework and common language, especially with leadership.

🟡 Practical — how to use it without getting lost in it#

• Assess the organization across each area (maturity grid) to decide where to invest, not to cover everything.
• Name roles (data owner, data lead, data steward, architect): the most concrete contribution.
• Align vocabulary: “reference data,” “master data,” and “metadata” are not the same.

🔴 Expert — critical use#

DMBOK is descriptive, not prescriptive: it does not say in what order, or with what budget. It is useful for a data manager or architect, but not very operational for a development team. The file's maturity matrices (Maturity Matrix — Data Project Management (PD), BI) operationalize part of these areas as verifiable milestones. Details on Data Governance and Organization.

10. Data governance managed as a project#

🟢 Novice — what it is, what it is not#

Data governance is the exercise of authority and control (planning, monitoring, enforcement) over the management of data assets. Put simply: who decides what, under which rules, and who is accountable. It is not a tool (the catalog is only a support) nor an additional committee.

🟡 Practical — start it as a value-driven project#

A governance program fails when it is “top-down, exhaustive, with no visible benefit.” It succeeds when it starts from a concrete pain point.

🔴 Expert — why it fails, how we know#

• No sponsor able to arbitrate between two departments: governance is nothing more than institutionalized arbitration.
• Roles without time allocation: “data steward” added to a job description without backfill is a fictional role. Indicator: how many hours per week, explicitly scheduled?
• Documentation without control: a glossary that nothing verifies becomes outdated in six months. Link each definition to an automated test or a dated review.
• Compliance as an afterthought: GDPR, regulated sectors — legitimate requirement, but to be treated as a design constraint, not as an afterthought (see the theme Security, Privacy, Compliance).

Ratings for governance as an approach (context: organization of 500+ people, identified sponsor): predictability 2/5, adaptability 3/5, risk control 4/5, lightweightness 2/5, adoption 2/5, visibility 5/5, deliverable quality 4/5. The main determinant is whether a sponsor is present: without one, everything drops by one point.

11. The data migration project#

🟢 Novice — why it is a project in its own right#

Migrating means moving data from one system to another while preserving meaning. Two errors are common: believing it is simple “technical copy-paste” and underestimating it; underestimating data cleansing and business validation. The cutover date is often fixed (contract end, shutdown of the old system): the project is more predictive than exploratory.

🟡 Practical — phases and controls#

Dry-run rehearsals are the #1 risk lever: they turn the unknown (“how long?”) into measured data and reveal edge cases (special characters, out-of-range dates, orphan keys).

🔴 Expert — reconciliation and idempotence#

Robust reconciliation combines three levels: counts (number of rows per table, per partition), aggregates (sums of numeric columns, date min/max), fingerprints (checksum of rows or of a subset). Comparable T-SQL checksum example across two databases:

SELECT COUNT(*)                         AS nb_lignes,
       SUM(CAST(montant AS decimal(19,4))) AS somme_montant,
       CHECKSUM_AGG(BINARY_CHECKSUM(*)) AS empreinte
FROM   dbo.Commande
WHERE  date_commande >= '2026-01-01';

(CHECKSUM_AGG/BINARY_CHECKSUM can have collisions: good for detecting a discrepancy, insufficient to prove equality; plan a targeted complementary check.)

The migration process must be idempotent: replaying it on the same input yields the same result with no duplicates. Technique: natural key + MERGE or INSERT … WHERE NOT EXISTS, processed-batch log. See CI/CD for data: foundations, datamarts, pipelines, and reports.

12. The BI / datamart project#

🟢 Novice — the most common form#

A BI project builds a datamart (dimensional model for a business domain) and reports that consume it. It follows: needs → model → loading → reports → acceptance testing → operations. It is a project that is half software engineering, half data investigation.

🟡 Practical — a recommended composite method#

Typical sequence for a batch (3 to 8 weeks):

1. Need and metrics: list of decisions to inform, metrics, and their written definitions.
2. Source profiling (spike): feasibility, quality, volume.
3. Dimensional model: grain, facts, dimensions, historization decided with the business.
4. Loading: replayable transformations, data tests.
5. Semantic model and reports: measures, security (RLS), translations.
6. Business acceptance testing: reconciliation with a reference.
7. Delivery: versioned kit, release notes, light training.
8. Operations: monitoring, support, feedback collection.

🔴 Expert — structuring trade-offs#

• Where to put logic: transformation in SQL (reusable by all tools) rather than in the semantic model (locked into one tool). Measures, however, live in the model. Details: Semantic Model and Data Layer and ETL.
• Deliver fast without model debt: a fast batch that bypasses a conformed dimension creates debt that will be paid in the next batch.
• Versions: a report consumed by external clients or subsidiaries requires not breaking column and measure names — treat the model as an API (see Versioning and Industrialization of Reports).
• Power BI reporting: target architectures are detailed in 04 Power BI Reporting.

13. Data quality projects#

🟢 Novice — fix the cause, not just the symptom#

A data quality project aims to measure and then improve data reliability. The common mistake: clean once (a “major cleanup”) and then find that defects come back. Quality is a process — trace back to the cause (entry, interface, business rule, integration) and put in a lasting control.

🟡 Practical — the quality loop#

A loop inspired by DMAIC (Define, Measure, Analyze, Improve, Control):

Common dimensions (vocabulary from data quality literature, to be defined locally): accuracy, completeness, consistency, uniqueness, freshness/currency, validity (format/domain compliance), referential integrity.

🔴 Expert — trade-offs#

• Block or let through: should poor-quality data prevent loading? Practical rule: block for defects that make aggregates wrong (null key, duplicate fact); let through with a signal for descriptive defects (missing label), with an “unknown” member in the dimension.
• Quarantine: write rejected rows to an exception table usable by the business, not to a technical log.
• Cost measurement: value calculation is approximate (correction time, delayed decisions); an honest, ordered estimate is better than an invented precise number.
• Link to the matrix: Data Quality topic for maturity levels.

14. Comparison Table and Ratings#

Summary (expert, contextual assessments: team of 3 to 10 people, data project of 3 to 12 months; see RATING). The approaches are not all of the same nature — some are processes, others are frameworks or philosophies — so the comparison is indicative. Full comparison with general methods: 08 Comparison of Methods.

(Ratings out of 5, the “Predict.” column = predictability, etc.)

How to read it:

• Kimball and DataOps are the two pillars of a BI project, and they complement each other (model + flow); you get the best of Kimball + Scrum/Kanban + DataOps.
• CRISP-DM is the best entry point for an analytics project; MLOps takes over after deployment.
• DAMA-DMBOK and Data Mesh are organizational in nature; they do not replace a project method.
• No single method covers modeling, flow, governance, and business acceptance testing all at once: the art is to assemble (Choosing Your Method and Hybridizing).

Quick decision tree:

15. Pitfalls and good practices#

The 40+ cross-functional pitfalls and 30+ good practices are grouped in 13 Pitfalls and good practices in project management (cross-functional summary). Those specific to this file:

16. AI Contribution#

Sample prompt (to run on an anonymized sample):

Here is the schema of a SQL Server table (columns, types, 20 anonymized sample rows).
1) Propose 10 quality checks in T-SQL (uniqueness, completeness, validity, date consistency).
2) For each, indicate severity (blocking/warning) and the question to ask the business.
3) Do not invent any business rule: flag what is an assumption.

What AI does not replace: definition arbitration between two departments, accountability for a published figure, context knowledge (why this strange rule exists).

17. See also#

• Fundamentals and frameworks: 01 Project Management Fundamentals · 05 Predictive Methods (waterfall, V-model, spiral, RUP, PMBOK, PRINCE2, Merise, CMMI) · 06 Agile and Lean Methods (Manifesto, Kanban, XP, Lean, Shape Up, DSDM, SAFe, DevOps…) · 08 Comparison of Project Management Methods · 09 Choosing Your Method and Hybridizing
• Delivery: 10 CI/CD: Continuous Integration and Continuous Delivery · 11 CI/CD for data: foundations, datamarts, pipelines, and reports · 13 Pitfalls and good practices in project management (cross-functional summary)
• Matrices: Maturity Matrix — Data Project Management (PD) · Maturity Matrix — BI Tool/Application Development
• Tools: VCS and SQL/BI data · Project Metrics and Reporting
• Cross-functional: Glossary · Bibliography and Resources · Cross-Functional Summary — AI Contribution

✻ Written by Claude Sonnet 5.5