IDMC for Databricks Foundation flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

Secure Agent

Lightweight runtime installed on-premises or in a customer VPC that executes Informatica jobs needing private network access, and connects outbound over HTTPS to the Informatica cloud.

Runtime environment

Configuration that binds a Secure Agent group to a specific IDMC service such as Cloud Data Integration, determining which agents can run a given task.

Pushdown optimization

Execution mode where the Secure Agent generates SQL for the target and submits it to Databricks compute, so transformation logic runs in the warehouse instead of on the agent.

Cloud Hosted Agent

Informatica-managed agent running inside the IDMC pod, used for cloud-to-cloud connectivity; it cannot reach private on-premises endpoints behind a firewall.

Connection versus mapping

A connection stores endpoint, credentials, and agent association, while a mapping stores transformation logic; mappings reference connections so logic can be redeployed across environments.

Delta Lake transaction log

JSON log in the _delta_log directory recording every commit, enabling ACID transactions, schema enforcement, time travel, and consistent reads for Delta tables.

Unity Catalog namespace

Three-level naming structure of catalog.schema.table that provides centralized governance, fine-grained privileges, and automatic lineage across Databricks workspaces.

Medallion architecture

Layered lakehouse design where Bronze holds raw ingested data, Silver holds cleansed and conformed data, and Gold holds business-level aggregates for BI and machine learning.

Auto Loader

Databricks ingestion feature using cloudFiles that incrementally detects new files through notifications or directory listing, with schema inference and evolution for streaming loads.

Job cluster

Short-lived Databricks cluster created for a single scheduled job run and terminated afterward, reducing cost compared with long-running all-purpose clusters.

Managed versus external table

Managed tables use a Databricks-controlled storage location and drop with their data; external tables reference a customer path and dropping removes only metadata.

COPY INTO

Databricks SQL command that loads files idempotently from cloud storage into a Delta table, tracking already-ingested files; batch-oriented rather than continuous.

Full pushdown requirement

Pushdown to Databricks succeeds only when source, target, and every transformation in the mapping are pushdown-capable; otherwise execution is partial.

HTTP path

Databricks connection property identifying the SQL warehouse or cluster endpoint; it must match the compute resource or the Informatica connection fails.

Mass Ingestion

IDMC service for mapping-free ingestion of streaming sources, files, and database changes, configured through wizards and run on Secure Agent or Elastic Server.

MERGE INTO

Delta SQL operation performing upserts and deletes by matching source rows to target rows, the standard pattern for applying change data capture.

Checkpoint location

Directory used by a streaming query to track processed data; each stream needs a unique checkpoint to avoid duplicate ingestion or skipped records.

Active transformation

Transformation that can change the number of rows passing through, such as Aggregator, Joiner, Filter, or Router, which constrains pushdown and execution planning.

expect_or_fail

Delta Live Tables expectation that fails the pipeline update when the data quality condition is violated, preventing bad records from reaching downstream tables.

Column mask

Unity Catalog function applied to a table column that rewrites values based on the querying user's identity or group membership, protecting sensitive fields.

Row filter

Unity Catalog function attached to a table that restricts visible rows according to current_user() or group membership, enforcing row-level security without separate views.

CDGC

Informatica Cloud Data Governance and Catalog, the service storing technical metadata, business glossary terms, data domains, policies, and lineage in a graph repository.

Column-level lineage

Mapping of how individual columns flow from source to target, derived from scanned SQL and mapping definitions, used for impact analysis and root-cause tracing.

Business glossary term

Governed business definition associated with technical assets so users can search by business meaning rather than by physical table or column names.

System tables

Databricks-managed schemas such as system.access.audit, system.billing.usage, and system.access.table_lineage that expose operational and lineage metadata as queryable tables.

Storage credential

Unity Catalog object holding a managed identity or service principal that authorizes access to cloud storage, referenced by external locations instead of user credentials.

Secret scope

Databricks container for secrets such as tokens and keys, referenced in code with dbutils.secrets.get so credentials never appear in notebooks or connection strings.

Privilege inheritance

Unity Catalog rule where a privilege granted on a catalog or schema is inherited by its child objects, so granting USE SCHEMA avoids per-table grants.

Databricks Asset Bundles

YAML-based packaging of jobs, pipelines, and notebooks that supports deploying the same project to multiple workspaces and environments through CI/CD.

Repair run

Databricks job rerun that executes only the failed or skipped tasks rather than the entire job, saving time and compute on long multi-task workflows.

OPTIMIZE and Z-ORDER

Delta maintenance commands that compact small files and colocate related data for data skipping, improving read performance on frequently filtered columns.

Governance-first design

Solution approach that registers assets in Unity Catalog, tags and masks sensitive data, and publishes glossary terms before exposing data to analytics consumers.