IDMC for Databricks Foundation key terms

A cloud-native platform of integrated services for data integration, quality, cataloging, governance, and application integration, delivered as managed microservices and used to move and govern data across on-premises and cloud systems such as Databricks.

The IDMC service that runs batch and real-time mapping pipelines in the cloud, connecting sources to targets such as Databricks SQL warehouses or Unity Catalog tables, with pushdown optimization and elastic scaling managed by Informatica.

A lightweight runtime installed in a customer network or cloud VPC that executes jobs needing private connectivity, and which can also host pushdown-eligible processing when the target database is reachable only from that network.

The design object in IDMC that defines a data flow from sources through transformations to targets, including ports, expressions, lookups, and joins, and which is executed by a mapping task or taskflow.

A deployable, schedulable instance of a mapping with its own runtime settings, parameters, and connection overrides, allowing one design to be reused across environments and execution engines.

An execution strategy that converts mapping logic into SQL or native commands run by the target or source database, such as Databricks, reducing data movement and leveraging the lakehouse engine's distributed compute.

A named, reusable set of connectivity properties, such as host, credentials, and catalog, used by mappings and tasks to reach a source or target system; connections are typically environment-specific and reference secure parameter stores.

An orchestration object that sequences mapping tasks, command tasks, decision steps, and notifications, enabling multi-step pipelines with dependencies, error handling, and scheduling across IDMC services.

The open table format underpinning Databricks that adds a transaction log to Parquet files, providing ACID transactions, schema enforcement, time travel, and efficient upserts for reliable lakehouse tables.

A layered lakehouse design pattern with bronze (raw ingested), silver (cleaned and conformed), and gold (business-level aggregates) layers, each improving data quality and serving different consumer needs.

A compute resource optimized for SQL workloads and BI connectivity, offering serverless or classic sizing, auto-stop, and JDBC/ODBC endpoints that integration tools commonly use as a target for pushed-down transformations.

Databricks' unified governance layer providing a three-level namespace of catalog, schema, and table or view, plus centralized privileges, auditing, lineage, and external location management across workspaces.

A Unity Catalog object that pairs a cloud storage path with a storage credential, authorizing Databricks to read and write that path so managed and external tables can be governed without embedding long-lived keys.

A Delta Lake SQL and API operation that applies inserts, updates, and deletes in one atomic statement based on a match condition, the standard technique for change data capture and upsert ingestion into lakehouse tables.

A technique that captures row-level inserts, updates, and deletes from a source system's logs or timestamps and replays them into a target, often via MERGE or Databricks auto CDC, keeping replicas incrementally synchronized.

The first medallion layer holding raw ingested data with minimal transformation, preserved history, and source metadata, serving as the replayable foundation for downstream cleaning and conformance.

A stored high-water mark, such as a maximum timestamp or ID, that a pipeline uses to select only new or changed source rows on each run, enabling incremental rather than full reloads.

Configurable validation checks such as not-null, range, pattern, or referential tests applied to data, which can score records, route exceptions, and optionally block promotion to downstream layers.

A dashboard that aggregates rule pass rates and dimension scores, such as completeness, accuracy, and timeliness, over time and by dataset, giving stewards visibility into fitness for purpose.

The design of pipelines to detect rejected or failing records, capture them with error reasons and source context, and continue processing valid rows, supporting remediation and audit without full job failure.

A metadata repository that inventories datasets, schemas, owners, and classifications across systems, enabling search, discovery, and linkage of technical assets to business context and governance policies.

A record of how data moves and transforms from source to target, at column or object level, used for impact analysis, root-cause investigation, and regulatory traceability of reported figures.

A curated set of business terms with approved definitions, synonyms, and owners, linked to technical assets so that analysts and stewards share consistent meaning for metrics and fields.

A role accountable for the quality, definitions, and appropriate use of a data domain, approving glossary terms, reviewing quality scores, and resolving issues with producers and consumers.

The set of grants, such as USE CATALOG, USE SCHEMA, SELECT, MODIFY, and CREATE, that determines what a principal can do on a securable object, inherited down the catalog hierarchy.

A control that filters rows returned to a user based on a mapping or policy, implemented in Databricks through row filter functions or dynamic views so different users see different subsets of one table.

A policy that substitutes sensitive column values, such as showing only the last four digits of an identifier, based on the querying user's group membership, without changing stored data.

A non-human identity, such as an application registration in the cloud identity provider, used by automated pipelines and integration agents to authenticate to Databricks and cloud storage with scoped permissions.

A Databricks-managed or Key Vault-backed container for secrets, referenced in notebooks and jobs so credentials are never hard-coded; IDMC parameter stores play an equivalent role for integration jobs.

The practice of tracking pipeline runs through status, duration, row counts, and error logs, using alerting and dashboards so failures are detected and remediated before consumers are affected.

Defining when pipelines run, via cron-like schedules, event triggers, or dependency-based orchestration, including concurrency limits, retries, and timezone handling to keep data fresh predictably.

The use of variables and in-out parameters in mappings, tasks, and connections so the same design can target different environments, dates, or filters without editing the design itself.

A set of Databricks compute nodes configured with a runtime and libraries, either all-purpose for interactive work or job clusters created per run, sized and auto-scaled for the workload.

A Unity Catalog object wrapping a cloud identity, such as an IAM role or managed identity, that Databricks assumes to access storage, referenced by external locations and external tables.

A processing step inside a mapping, such as expression, lookup, joiner, aggregator, or router, that changes, enriches, or splits data between source and target ports.

The discipline of choosing architecture, ingestion patterns, layering, and governance controls to meet latency, volume, cost, and compliance requirements, documented before implementation begins.