clement13.net
Original photo: A prism splitting light
Description
A prism splitting light
Source
Kelvinsong · original page
Licence
CC0 Public domain
Original photo, unretouched. The background is a version of it stylised by the site.

Matrix: Abstraction layer

The layer that sits between the data and the reports: single definitions, structure, reuse, row-level rights, languages, description, change, performance.

≈ 11 min readReading
In brief
  • 8 dimensions, levels 0 to 5, 112 criteria.
  • Tool: run an assessment by choosing "Abstraction layer".
  • Levels and criteria: original synthesis, references cited at the bottom of the page.
On this page
Scope

The abstraction layer between the stored data and the reports, whatever the technology (semantic model, cube, views, metrics layer): single definition of measures, model structure, reuse, row-level security, languages and formats, description, version compatibility, performance. The underlying tables belong to the "Storage" theme, the reports to the "Reporting" theme.

8 dimensions, six levels (0 to 5) and 112 observable criteria. To place your product: open the assessment for this theme. The scoring rules are here.

Overview#

Each cell is the level's signature for the dimension.

Dimension0 · Handcrafted1 · Locally tooled2 · Shared3 · Industrialized4 · Integrated and governed5 · Autonomous
Single definition of measuresEach report calculates its indicators its own way.Indicators listed.Measures centralized.Measures managed.Measures verified.Measures steered.
Model structureOne big flat table, or as many tables as there are reports.Tables separated by subject.Facts and dimensions.Conformed model.Controlled model.Model checked automatically.
Reuse and sharingA model copied and modified for each report.Models identified.Shared model.Common model and specifics.Governed composition.Reuse steered.
Row-level and object-level securityConfidentiality ensured by separate reports for each audience.Filters in the reports.Rights in the layer.Dynamic rights.Controlled rights.Rights verified continuously.
Languages, formats and labelsTechnical labels, a single language, formats specific to each report.Readable labels.Common formats.Languages managed.Translations controlled.Translations automated.
Description and discoverabilityTechnical names, no description.Explicit names.Descriptions in place.Descriptions managed.Discovery made easy.Description maintained automatically.
Versions and compatibilityA change to the model breaks reports, discovered in use.Changes known.Changes versioned.Compatibility sought.Compatibility verified.Compatibility automated.
Layer performanceSlow response, unexplained.Slowness known.Lightened model.Systematic optimization.Performance under control.Adaptive performance.
Link with the overall model

Dimensions of the overall model that are most relevant: Semantic model and metrics, Access and entitlements, Documentation and traceability and Availability and performance.

Definitions and structure#

Single definition of measures#

Is each indicator defined once, in one place, for all reports?

LevelSignatureCriteria to meet
0 · HandcraftedEach report calculates its indicators its own way.Starting point: no criteria.
1 · Locally tooledIndicators listed.
  • The main indicators are listed with their formula.
  • Known discrepancies between reports are explained.
2 · SharedMeasures centralized.
  • Indicators are defined in the abstraction layer, not in each report.
  • Each indicator has a unique name and a description.
  • A report that needs an existing indicator reuses it without redefining it.
3 · IndustrializedMeasures managed.
  • Each indicator has a business owner who validates its definition.
  • Indicators are named according to a common rule (base, period, variant).
  • A variant of an indicator (period, scope) is derived from the base indicator, without copying the formula.
4 · Integrated and governedMeasures verified.
  • Each indicator has a set of test cases with an expected value.
  • A change of definition is announced to users before it goes live.
  • Indicators can be queried from several tools (reports, spreadsheet, queries) with the same result.
5 · AutonomousMeasures steered.
  • Definitions are kept in versioned files, applied automatically to the layer.
  • An indicator that is defined twice or contradictorily is detected automatically.
  • The usage of each indicator is measured so that those nobody uses can be retired.
Effect on the user

The same figure has the same value everywhere, and a change of definition is made in one place only.

Model structure#

Does the structure serve analysis: analysis tables, relationships, grain?

LevelSignatureCriteria to meet
0 · HandcraftedOne big flat table, or as many tables as there are reports.Starting point: no criteria.
1 · Locally tooledTables separated by subject.
  • Tables are separated by subject (sales, customers, products).
  • Each table has a unique key.
2 · SharedFacts and dimensions.
  • Tables are classified as fact tables (measures) and dimensions (filter and grouping axes).
  • Relationships are declared, and their direction is chosen (one to many).
  • The grain of each fact table is homogeneous.
3 · IndustrializedConformed model.
  • Common dimensions are shared across fact tables.
  • The roles of a single dimension (for example order date, delivery date) are modeled explicitly.
  • Inactive or bidirectional relationships are rare and justified.
4 · Integrated and governedControlled model.
  • The effect of each relationship on the figures (back-filtering, double counting) is analyzed and documented.
  • Special cases (many-to-many, dimension history) have a defined treatment.
  • Technical columns are hidden, so that only what is useful for analysis is visible.
5 · AutonomousModel checked automatically.
  • Automatic checks detect any deviation from the structure rules (grain, keys, relationships).
  • The structure is generated or verified from the metadata of the storage layer.
  • Discrepancies between the model and the underlying tables are detected before go-live.
Effect on the user

Figures can be cross-analyzed as expected, without producing a wrong result through a bad chain of joins.

Reuse and sharing#

Does a single model serve several reports, several customers, several teams?

LevelSignatureCriteria to meet
0 · HandcraftedA model copied and modified for each report.Starting point: no criteria.
1 · Locally tooledModels identified.
  • Existing models are listed with their reports.
  • Copies are spotted.
2 · SharedShared model.
  • Several reports rely on one published model.
  • Reports do not contain a copy of the model's data.
  • A change to the model is visible to all its reports without republishing.
3 · IndustrializedCommon model and specifics.
  • A common model carries the standard definitions; specifics are isolated.
  • A customer- or project-specific element is added to the common model without modifying it.
  • Specific elements are listed and linked to whoever requested them.
4 · Integrated and governedGoverned composition.
  • Models can be composed (one model reuses another) according to written rules.
  • The impact of a change in the common model on the specific models is known before deployment.
  • A specific element that is useful to several parties is considered for inclusion in the common model.
5 · AutonomousReuse steered.
  • The reuse rate of the common model is measured, and the gaps to close are tracked.
  • Specific models are rebuilt automatically with each new version of the common model.
  • Elements duplicated between models are detected automatically.
Effect on the user

Fixing or enriching something once benefits everyone, and nobody maintains their own copy.

Access and security#

Row-level and object-level security#

Does everyone see only the rows and objects they are entitled to, whichever report is used?

LevelSignatureCriteria to meet
0 · HandcraftedConfidentiality ensured by separate reports for each audience.Starting point: no criteria.
1 · Locally tooledFilters in the reports.
  • Visibility scopes are filtered in the reports.
  • The list of people authorized for each scope is maintained.
2 · SharedRights in the layer.
  • Row-level restriction is defined in the abstraction layer, not in each report.
  • Role membership is managed through groups, not person by person.
  • A person with no role sees no data.
3 · IndustrializedDynamic rights.
  • A single rule depends on the person's identity and on an assignment table (dynamic rights).
  • Rights are tested by role before go-live, including edge cases.
  • Sensitive columns and objects are protected at the layer level (object security).
4 · Integrated and governedControlled rights.
  • The effect of security on the figures (totals that differ by role) is explained to users.
  • Security works in every tool that reads the layer, including direct access.
  • Special cases (administrators, service accounts, external guests) have a documented treatment.
5 · AutonomousRights verified continuously.
  • Automatic tests check each role with every new version.
  • Discrepancies between expected rights and effective rights are detected automatically.
  • Access to data outside the usual scope is flagged.
Effect on the user

The confidentiality rule is applied in the layer, and therefore everywhere, including in a spreadsheet or a new report.

Languages, formats and labels#

Do names, formats and languages adapt to the user?

LevelSignatureCriteria to meet
0 · HandcraftedTechnical labels, a single language, formats specific to each report.Starting point: no criteria.
1 · Locally tooledReadable labels.
  • Visible labels are understandable (no technical names).
  • Units are specified on measures.
2 · SharedCommon formats.
  • Each measure has a default format (decimals, separators, currency).
  • Labels are managed in a single place in the layer.
  • Dates and periods follow a common calendar.
3 · IndustrializedLanguages managed.
  • The layer's labels exist in several languages.
  • A label missing in a language displays the fallback language, not an empty value.
  • Translations are kept in a single source, outside the reports.
4 · Integrated and governedTranslations controlled.
  • Translations go through a review process.
  • A new label without a translation is detected before go-live.
  • Each user's language and formats are applied automatically.
5 · AutonomousTranslations automated.
  • A translation is proposed automatically for any new label, then validated.
  • Terminology discrepancies between languages are detected.
  • Translations are applied to the model by a process, with no manual entry.
Effect on the user

Everyone reads the figures in their own language and format, without the model being duplicated.

Change and usage#

Description and discoverability#

Does a user find the right fields, and understand their meaning and limits?

LevelSignatureCriteria to meet
0 · HandcraftedTechnical names, no description.Starting point: no criteria.
1 · Locally tooledExplicit names.
  • Tables, columns and measures have explicit names.
  • Objects that are useless for analysis are hidden.
2 · SharedDescriptions in place.
  • Each useful table, column and measure has a description that explains its meaning and use.
  • Hierarchies (year, month, day; category, product) are declared.
  • Data categories (country, address, link) are filled in.
3 · IndustrializedDescriptions managed.
  • Descriptions are reviewed with the business owners.
  • Each field states its data source and its freshness.
  • Fields being retired are flagged as such.
4 · Integrated and governedDiscovery made easy.
  • Fields and measures are searchable from report authoring tools.
  • Models are published in a catalog with their description and their owner.
  • Usage examples (typical reports, typical questions) accompany each model.
5 · AutonomousDescription maintained automatically.
  • Descriptions are generated from metadata and proposed for validation.
  • A field with no description or no usage is detected automatically.
  • Users' questions about the meaning of a field improve the descriptions.
Effect on the user

An analysis can be built without asking the technical team what a field means.

Versions and compatibility#

Does a change to the layer break existing reports, your own and your customers'?

LevelSignatureCriteria to meet
0 · HandcraftedA change to the model breaks reports, discovered in use.Starting point: no criteria.
1 · Locally tooledChanges known.
  • Changes to the layer are announced to the teams concerned.
  • Reports that use a modified field are searched for by hand.
2 · SharedChanges versioned.
  • Each change to the layer is recorded in a dated version.
  • The names of published fields do not change without reason.
  • A rename or a removal is treated as a change that must be announced.
3 · IndustrializedCompatibility sought.
  • Fields have stable identifiers: a rename does not break the reports that use them.
  • The list of reports that consume each field is known.
  • An incompatible change is preceded by a period during which the old and the new field coexist.
4 · Integrated and governedCompatibility verified.
  • Reference reports are rechecked before each go-live of the layer.
  • Reports of customers or third-party teams are covered by the check, or their owner is notified.
  • Removing a field follows an announced deprecation schedule.
5 · AutonomousCompatibility automated.
  • An incompatible change is detected automatically by comparing two versions of the layer.
  • Affected reports are listed and fixed automatically when the change is simple.
  • The stability of the model (number of breaking changes per version) is tracked.
Effect on the user

The model can evolve without the reports already built ceasing to work.

Layer performance#

Does the layer respond quickly, even with a lot of data and many people?

LevelSignatureCriteria to meet
0 · HandcraftedSlow response, unexplained.Starting point: no criteria.
1 · Locally tooledSlowness known.
  • Slow queries are identified.
  • The size of the layer is known.
2 · SharedLightened model.
  • Columns that are useless for analysis are not loaded into the layer.
  • High-cardinality columns (identifiers, timestamps) are dealt with.
  • The storage mode (copy, direct access, mixed) is chosen for each table according to its use.
3 · IndustrializedSystematic optimization.
  • Measures are written so as to limit the computation cost (reviews, analysis tools).
  • Aggregations prepare the most requested calculations.
  • Response times are measured with row-level rights applied.
4 · Integrated and governedPerformance under control.
  • A degradation in response time is detected before a version goes live.
  • The layer is tested with the expected number of concurrent users.
  • Resources are sized according to the measured load.
5 · AutonomousAdaptive performance.
  • Aggregations are created or removed according to measured usage.
  • Capacity adjusts automatically to the load.
  • A performance threshold is part of the acceptance criteria for a new version.
Effect on the user

Figures display quickly and predictably, without report authors having to work around the model.

Origin of the criteria

The dimensions are inspired by the references cited below; the levels and criteria are an original synthesis, not a reproduction of a framework. They are offered for discussion and adaptation, not as authoritative.

Sources and verification#