GCP Data Practitioner cheat sheet

BigQuery load jobs accept CSV, JSON (newline-delimited), Avro, Parquet, ORC; Avro and Parquet preserve nested and typed schemas, CSV does not.

Storage Transfer Service moves data from Cloud Storage, Amazon S3, Azure Blob, HTTP/HTTPS URLs, or on-premises sources into Cloud Storage or BigQuery on a schedule.

Datastream provides change data capture (CDC) from MySQL, PostgreSQL, SQL Server, and Oracle into BigQuery or Cloud Storage for near-real-time replication.

Pub/Sub plus Dataflow is the pattern for streaming ingestion when source systems have no managed connector; Pub/Sub retains messages up to 7 days by default (31 days maximum with message retention configured).

BigQuery external tables read data in place from Cloud Storage, BigLake, or Drive; they do not incur BigQuery storage cost but queries scan the external files each time.

BigQuery autodetect infers schema from a sample of rows; explicit schemas are required when column types are ambiguous, when loading empty files, or when you need strict typing.

The bq load command and the jobs.insert API both create load jobs; the --replace flag truncates the destination table, while --append (write disposition WRITE_APPEND) adds rows.

Data Transfer Service (BigQuery Data Transfer Service) schedules recurring loads from SaaS sources such as Google Ads, YouTube, and Amazon S3, and supports backfill of historical runs.

Partitioning by ingestion time or a DATE/TIMESTAMP column and clustering on high-cardinality filter columns reduce bytes scanned and cost in BigQuery.

BigQuery on-demand pricing bills per byte scanned (with a 1 TB/month free tier per project); flat-rate and slot-based editions bill for compute capacity instead.

A dry run (bq query --dry_run or the API dryRun flag) returns the bytes that would be processed without running the query or incurring cost.

SELECT * is billed for all columns; selecting only needed columns on a columnar table reduces bytes scanned and is the primary cost lever.

Looker Studio connects to BigQuery, Looker, Sheets, and other sources; BigQuery BI Engine provides an in-memory acceleration layer for sub-second dashboard queries.

Materialized views in BigQuery precompute and incrementally refresh results; they are automatically used by the query optimizer when a query matches the view definition.

Approximate aggregation functions (APPROX_COUNT_DISTINCT, APPROX_QUANTILES) trade exactness for speed and lower cost on large datasets.

Window functions (ROW_NUMBER, RANK, SUM OVER) compute per-partition results without collapsing rows; QUALIFY filters on window function output.

Looker (the enterprise BI product) uses LookML to define a semantic model; Looker Studio is the free self-service reporting tool and does not use LookML.

Cloud Composer is managed Apache Airflow; DAGs are written in Python, and operators/tasks define dependencies via >> or set_upstream/set_downstream.

Dataflow runs Apache Beam pipelines; it supports both batch and streaming from the same code, with autoscaling of worker VMs based on pipeline backlog.

Dataflow templates (Google-provided or custom Flex templates) let you launch pipelines without writing code; Flex templates package the pipeline as a container image.

Cloud Scheduler is a cron-based job trigger; it can invoke Cloud Functions, Pub/Sub, or HTTP endpoints but does not itself process data.

Cloud Functions are event-driven and suited to lightweight triggers (file arrival, Pub/Sub message) rather than long-running data transforms.

Dataform and dbt-style SQL transformation workflows orchestrate SQL in BigQuery; Dataform is a Google Cloud service integrated with BigQuery and Git repositories.

Airflow sensors (e.g., GCSObjectExistenceSensor) wait for conditions but consume worker slots; deferrable operators release the slot while waiting.

Dataproc runs managed Spark and Hadoop clusters; ephemeral clusters created per job and workflow templates are the cost-efficient pattern.

IAM roles are additive; the most permissive binding wins. BigQuery predefined roles include roles/bigquery.dataViewer, dataEditor, dataOwner, jobUser, and user.

A BigQuery dataset is the unit of access control; authorized views and authorized datasets grant access to derived data without granting access to underlying tables.

Column-level access control uses policy tags in Data Catalog (now Dataplex Universal Catalog); row-level security uses row access policies with filter predicates.

Cloud Storage IAM applies bucket-wide or object-level; uniform bucket-level access disables object ACLs and is recommended for consistent permission management.

VPC Service Controls create a service perimeter that restricts data exfiltration for services such as BigQuery and Cloud Storage.

Cloud KMS customer-managed encryption keys (CMEK) can encrypt BigQuery datasets and Cloud Storage buckets; BigQuery also supports customer-supplied encryption keys (CSEK) per query or load job.

BigQuery table and dataset expiration (default_table_expiration_ms) automatically deletes data after a set period; time travel retains deleted or changed data for 7 days by default.

Information Schema views (INFORMATION_SCHEMA.JOBS, TABLES, COLUMNS) expose job history, metadata, and access patterns for auditing and cost analysis.

BigQuery load jobs accept CSV, JSON (newline-delimited), Avro, Parquet, ORC; Avro and Parquet preserve nested and typed schemas, CSV does not.

Storage Transfer Service moves data from Cloud Storage, Amazon S3, Azure Blob, HTTP/HTTPS URLs, or on-premises sources into Cloud Storage or BigQuery on a schedule.

Datastream provides change data capture (CDC) from MySQL, PostgreSQL, SQL Server, and Oracle into BigQuery or Cloud Storage for near-real-time replication.

Pub/Sub plus Dataflow is the pattern for streaming ingestion when source systems have no managed connector; Pub/Sub retains messages up to 7 days by default (31 days maximum with message retention configured).

BigQuery external tables read data in place from Cloud Storage, BigLake, or Drive; they do not incur BigQuery storage cost but queries scan the external files each time.

BigQuery autodetect infers schema from a sample of rows; explicit schemas are required when column types are ambiguous, when loading empty files, or when you need strict typing.

The bq load command and the jobs.insert API both create load jobs; the --replace flag truncates the destination table, while --append (write disposition WRITE_APPEND) adds rows.

Data Transfer Service (BigQuery Data Transfer Service) schedules recurring loads from SaaS sources such as Google Ads, YouTube, and Amazon S3, and supports backfill of historical runs.

Partitioning by ingestion time or a DATE/TIMESTAMP column and clustering on high-cardinality filter columns reduce bytes scanned and cost in BigQuery.

BigQuery on-demand pricing bills per byte scanned (with a 1 TB/month free tier per project); flat-rate and slot-based editions bill for compute capacity instead.

A dry run (bq query --dry_run or the API dryRun flag) returns the bytes that would be processed without running the query or incurring cost.

SELECT * is billed for all columns; selecting only needed columns on a columnar table reduces bytes scanned and is the primary cost lever.

Looker Studio connects to BigQuery, Looker, Sheets, and other sources; BigQuery BI Engine provides an in-memory acceleration layer for sub-second dashboard queries.

Materialized views in BigQuery precompute and incrementally refresh results; they are automatically used by the query optimizer when a query matches the view definition.

Approximate aggregation functions (APPROX_COUNT_DISTINCT, APPROX_QUANTILES) trade exactness for speed and lower cost on large datasets.

Window functions (ROW_NUMBER, RANK, SUM OVER) compute per-partition results without collapsing rows; QUALIFY filters on window function output.

Looker (the enterprise BI product) uses LookML to define a semantic model; Looker Studio is the free self-service reporting tool and does not use LookML.

Cloud Composer is managed Apache Airflow; DAGs are written in Python, and operators/tasks define dependencies via >> or set_upstream/set_downstream.

Dataflow runs Apache Beam pipelines; it supports both batch and streaming from the same code, with autoscaling of worker VMs based on pipeline backlog.

Dataflow templates (Google-provided or custom Flex templates) let you launch pipelines without writing code; Flex templates package the pipeline as a container image.

Cloud Scheduler is a cron-based job trigger; it can invoke Cloud Functions, Pub/Sub, or HTTP endpoints but does not itself process data.

Cloud Functions are event-driven and suited to lightweight triggers (file arrival, Pub/Sub message) rather than long-running data transforms.

Dataform and dbt-style SQL transformation workflows orchestrate SQL in BigQuery; Dataform is a Google Cloud service integrated with BigQuery and Git repositories.

Airflow sensors (e.g., GCSObjectExistenceSensor) wait for conditions but consume worker slots; deferrable operators release the slot while waiting.

Dataproc runs managed Spark and Hadoop clusters; ephemeral clusters created per job and workflow templates are the cost-efficient pattern.

IAM roles are additive; the most permissive binding wins. BigQuery predefined roles include roles/bigquery.dataViewer, dataEditor, dataOwner, jobUser, and user.

A BigQuery dataset is the unit of access control; authorized views and authorized datasets grant access to derived data without granting access to underlying tables.

Column-level access control uses policy tags in Data Catalog (now Dataplex Universal Catalog); row-level security uses row access policies with filter predicates.

Cloud Storage IAM applies bucket-wide or object-level; uniform bucket-level access disables object ACLs and is recommended for consistent permission management.

VPC Service Controls create a service perimeter that restricts data exfiltration for services such as BigQuery and Cloud Storage.

Cloud KMS customer-managed encryption keys (CMEK) can encrypt BigQuery datasets and Cloud Storage buckets; BigQuery also supports customer-supplied encryption keys (CSEK) per query or load job.

BigQuery table and dataset expiration (default_table_expiration_ms) automatically deletes data after a set period; time travel retains deleted or changed data for 7 days by default.

Information Schema views (INFORMATION_SCHEMA.JOBS, TABLES, COLUMNS) expose job history, metadata, and access patterns for auditing and cost analysis.

BigQuery load jobs accept CSV, JSON (newline-delimited), Avro, Parquet, ORC; Avro and Parquet preserve nested and typed schemas, CSV does not.

Storage Transfer Service moves data from Cloud Storage, Amazon S3, Azure Blob, HTTP/HTTPS URLs, or on-premises sources into Cloud Storage or BigQuery on a schedule.

Datastream provides change data capture (CDC) from MySQL, PostgreSQL, SQL Server, and Oracle into BigQuery or Cloud Storage for near-real-time replication.

Pub/Sub plus Dataflow is the pattern for streaming ingestion when source systems have no managed connector; Pub/Sub retains messages up to 7 days by default (31 days maximum with message retention configured).

BigQuery external tables read data in place from Cloud Storage, BigLake, or Drive; they do not incur BigQuery storage cost but queries scan the external files each time.

BigQuery autodetect infers schema from a sample of rows; explicit schemas are required when column types are ambiguous, when loading empty files, or when you need strict typing.

The bq load command and the jobs.insert API both create load jobs; the --replace flag truncates the destination table, while --append (write disposition WRITE_APPEND) adds rows.

Data Transfer Service (BigQuery Data Transfer Service) schedules recurring loads from SaaS sources such as Google Ads, YouTube, and Amazon S3, and supports backfill of historical runs.

Partitioning by ingestion time or a DATE/TIMESTAMP column and clustering on high-cardinality filter columns reduce bytes scanned and cost in BigQuery.

BigQuery on-demand pricing bills per byte scanned (with a 1 TB/month free tier per project); flat-rate and slot-based editions bill for compute capacity instead.

A dry run (bq query --dry_run or the API dryRun flag) returns the bytes that would be processed without running the query or incurring cost.

SELECT * is billed for all columns; selecting only needed columns on a columnar table reduces bytes scanned and is the primary cost lever.

Looker Studio connects to BigQuery, Looker, Sheets, and other sources; BigQuery BI Engine provides an in-memory acceleration layer for sub-second dashboard queries.

Materialized views in BigQuery precompute and incrementally refresh results; they are automatically used by the query optimizer when a query matches the view definition.

Approximate aggregation functions (APPROX_COUNT_DISTINCT, APPROX_QUANTILES) trade exactness for speed and lower cost on large datasets.

Window functions (ROW_NUMBER, RANK, SUM OVER) compute per-partition results without collapsing rows; QUALIFY filters on window function output.

Looker (the enterprise BI product) uses LookML to define a semantic model; Looker Studio is the free self-service reporting tool and does not use LookML.

Cloud Composer is managed Apache Airflow; DAGs are written in Python, and operators/tasks define dependencies via >> or set_upstream/set_downstream.

Dataflow runs Apache Beam pipelines; it supports both batch and streaming from the same code, with autoscaling of worker VMs based on pipeline backlog.

Dataflow templates (Google-provided or custom Flex templates) let you launch pipelines without writing code; Flex templates package the pipeline as a container image.

Cloud Scheduler is a cron-based job trigger; it can invoke Cloud Functions, Pub/Sub, or HTTP endpoints but does not itself process data.

Cloud Functions are event-driven and suited to lightweight triggers (file arrival, Pub/Sub message) rather than long-running data transforms.

Dataform and dbt-style SQL transformation workflows orchestrate SQL in BigQuery; Dataform is a Google Cloud service integrated with BigQuery and Git repositories.

Airflow sensors (e.g., GCSObjectExistenceSensor) wait for conditions but consume worker slots; deferrable operators release the slot while waiting.

Dataproc runs managed Spark and Hadoop clusters; ephemeral clusters created per job and workflow templates are the cost-efficient pattern.

IAM roles are additive; the most permissive binding wins. BigQuery predefined roles include roles/bigquery.dataViewer, dataEditor, dataOwner, jobUser, and user.

A BigQuery dataset is the unit of access control; authorized views and authorized datasets grant access to derived data without granting access to underlying tables.

Column-level access control uses policy tags in Data Catalog (now Dataplex Universal Catalog); row-level security uses row access policies with filter predicates.

Cloud Storage IAM applies bucket-wide or object-level; uniform bucket-level access disables object ACLs and is recommended for consistent permission management.

VPC Service Controls create a service perimeter that restricts data exfiltration for services such as BigQuery and Cloud Storage.

Cloud KMS customer-managed encryption keys (CMEK) can encrypt BigQuery datasets and Cloud Storage buckets; BigQuery also supports customer-supplied encryption keys (CSEK) per query or load job.

BigQuery table and dataset expiration (default_table_expiration_ms) automatically deletes data after a set period; time travel retains deleted or changed data for 7 days by default.

Information Schema views (INFORMATION_SCHEMA.JOBS, TABLES, COLUMNS) expose job history, metadata, and access patterns for auditing and cost analysis.

BigQuery load jobs accept CSV, JSON (newline-delimited), Avro, Parquet, ORC; Avro and Parquet preserve nested and typed schemas, CSV does not.

Storage Transfer Service moves data from Cloud Storage, Amazon S3, Azure Blob, HTTP/HTTPS URLs, or on-premises sources into Cloud Storage or BigQuery on a schedule.

Datastream provides change data capture (CDC) from MySQL, PostgreSQL, SQL Server, and Oracle into BigQuery or Cloud Storage for near-real-time replication.

Pub/Sub plus Dataflow is the pattern for streaming ingestion when source systems have no managed connector; Pub/Sub retains messages up to 7 days by default (31 days maximum with message retention configured).

BigQuery external tables read data in place from Cloud Storage, BigLake, or Drive; they do not incur BigQuery storage cost but queries scan the external files each time.

BigQuery autodetect infers schema from a sample of rows; explicit schemas are required when column types are ambiguous, when loading empty files, or when you need strict typing.

The bq load command and the jobs.insert API both create load jobs; the --replace flag truncates the destination table, while --append (write disposition WRITE_APPEND) adds rows.

Data Transfer Service (BigQuery Data Transfer Service) schedules recurring loads from SaaS sources such as Google Ads, YouTube, and Amazon S3, and supports backfill of historical runs.

Partitioning by ingestion time or a DATE/TIMESTAMP column and clustering on high-cardinality filter columns reduce bytes scanned and cost in BigQuery.

BigQuery on-demand pricing bills per byte scanned (with a 1 TB/month free tier per project); flat-rate and slot-based editions bill for compute capacity instead.

A dry run (bq query --dry_run or the API dryRun flag) returns the bytes that would be processed without running the query or incurring cost.

SELECT * is billed for all columns; selecting only needed columns on a columnar table reduces bytes scanned and is the primary cost lever.

Looker Studio connects to BigQuery, Looker, Sheets, and other sources; BigQuery BI Engine provides an in-memory acceleration layer for sub-second dashboard queries.

Materialized views in BigQuery precompute and incrementally refresh results; they are automatically used by the query optimizer when a query matches the view definition.

Approximate aggregation functions (APPROX_COUNT_DISTINCT, APPROX_QUANTILES) trade exactness for speed and lower cost on large datasets.

Window functions (ROW_NUMBER, RANK, SUM OVER) compute per-partition results without collapsing rows; QUALIFY filters on window function output.

Looker (the enterprise BI product) uses LookML to define a semantic model; Looker Studio is the free self-service reporting tool and does not use LookML.

Cloud Composer is managed Apache Airflow; DAGs are written in Python, and operators/tasks define dependencies via >> or set_upstream/set_downstream.

Dataflow runs Apache Beam pipelines; it supports both batch and streaming from the same code, with autoscaling of worker VMs based on pipeline backlog.

Dataflow templates (Google-provided or custom Flex templates) let you launch pipelines without writing code; Flex templates package the pipeline as a container image.

Cloud Scheduler is a cron-based job trigger; it can invoke Cloud Functions, Pub/Sub, or HTTP endpoints but does not itself process data.

Cloud Functions are event-driven and suited to lightweight triggers (file arrival, Pub/Sub message) rather than long-running data transforms.

Dataform and dbt-style SQL transformation workflows orchestrate SQL in BigQuery; Dataform is a Google Cloud service integrated with BigQuery and Git repositories.

Airflow sensors (e.g., GCSObjectExistenceSensor) wait for conditions but consume worker slots; deferrable operators release the slot while waiting.

Dataproc runs managed Spark and Hadoop clusters; ephemeral clusters created per job and workflow templates are the cost-efficient pattern.

IAM roles are additive; the most permissive binding wins. BigQuery predefined roles include roles/bigquery.dataViewer, dataEditor, dataOwner, jobUser, and user.

A BigQuery dataset is the unit of access control; authorized views and authorized datasets grant access to derived data without granting access to underlying tables.

Column-level access control uses policy tags in Data Catalog (now Dataplex Universal Catalog); row-level security uses row access policies with filter predicates.

Cloud Storage IAM applies bucket-wide or object-level; uniform bucket-level access disables object ACLs and is recommended for consistent permission management.

VPC Service Controls create a service perimeter that restricts data exfiltration for services such as BigQuery and Cloud Storage.

Cloud KMS customer-managed encryption keys (CMEK) can encrypt BigQuery datasets and Cloud Storage buckets; BigQuery also supports customer-supplied encryption keys (CSEK) per query or load job.

BigQuery table and dataset expiration (default_table_expiration_ms) automatically deletes data after a set period; time travel retains deleted or changed data for 7 days by default.

Information Schema views (INFORMATION_SCHEMA.JOBS, TABLES, COLUMNS) expose job history, metadata, and access patterns for auditing and cost analysis.

BigQuery load jobs accept CSV, JSON (newline-delimited), Avro, Parquet, ORC; Avro and Parquet preserve nested and typed schemas, CSV does not.

Storage Transfer Service moves data from Cloud Storage, Amazon S3, Azure Blob, HTTP/HTTPS URLs, or on-premises sources into Cloud Storage or BigQuery on a schedule.

Datastream provides change data capture (CDC) from MySQL, PostgreSQL, SQL Server, and Oracle into BigQuery or Cloud Storage for near-real-time replication.

Pub/Sub plus Dataflow is the pattern for streaming ingestion when source systems have no managed connector; Pub/Sub retains messages up to 7 days by default (31 days maximum with message retention configured).

BigQuery external tables read data in place from Cloud Storage, BigLake, or Drive; they do not incur BigQuery storage cost but queries scan the external files each time.

BigQuery autodetect infers schema from a sample of rows; explicit schemas are required when column types are ambiguous, when loading empty files, or when you need strict typing.

The bq load command and the jobs.insert API both create load jobs; the --replace flag truncates the destination table, while --append (write disposition WRITE_APPEND) adds rows.

Data Transfer Service (BigQuery Data Transfer Service) schedules recurring loads from SaaS sources such as Google Ads, YouTube, and Amazon S3, and supports backfill of historical runs.

Partitioning by ingestion time or a DATE/TIMESTAMP column and clustering on high-cardinality filter columns reduce bytes scanned and cost in BigQuery.

BigQuery on-demand pricing bills per byte scanned (with a 1 TB/month free tier per project); flat-rate and slot-based editions bill for compute capacity instead.

A dry run (bq query --dry_run or the API dryRun flag) returns the bytes that would be processed without running the query or incurring cost.

SELECT * is billed for all columns; selecting only needed columns on a columnar table reduces bytes scanned and is the primary cost lever.

Looker Studio connects to BigQuery, Looker, Sheets, and other sources; BigQuery BI Engine provides an in-memory acceleration layer for sub-second dashboard queries.

Materialized views in BigQuery precompute and incrementally refresh results; they are automatically used by the query optimizer when a query matches the view definition.

Approximate aggregation functions (APPROX_COUNT_DISTINCT, APPROX_QUANTILES) trade exactness for speed and lower cost on large datasets.

Window functions (ROW_NUMBER, RANK, SUM OVER) compute per-partition results without collapsing rows; QUALIFY filters on window function output.

Looker (the enterprise BI product) uses LookML to define a semantic model; Looker Studio is the free self-service reporting tool and does not use LookML.

Cloud Composer is managed Apache Airflow; DAGs are written in Python, and operators/tasks define dependencies via >> or set_upstream/set_downstream.

Dataflow runs Apache Beam pipelines; it supports both batch and streaming from the same code, with autoscaling of worker VMs based on pipeline backlog.

Dataflow templates (Google-provided or custom Flex templates) let you launch pipelines without writing code; Flex templates package the pipeline as a container image.

Cloud Scheduler is a cron-based job trigger; it can invoke Cloud Functions, Pub/Sub, or HTTP endpoints but does not itself process data.

Cloud Functions are event-driven and suited to lightweight triggers (file arrival, Pub/Sub message) rather than long-running data transforms.

Dataform and dbt-style SQL transformation workflows orchestrate SQL in BigQuery; Dataform is a Google Cloud service integrated with BigQuery and Git repositories.

Airflow sensors (e.g., GCSObjectExistenceSensor) wait for conditions but consume worker slots; deferrable operators release the slot while waiting.

Dataproc runs managed Spark and Hadoop clusters; ephemeral clusters created per job and workflow templates are the cost-efficient pattern.

IAM roles are additive; the most permissive binding wins. BigQuery predefined roles include roles/bigquery.dataViewer, dataEditor, dataOwner, jobUser, and user.

A BigQuery dataset is the unit of access control; authorized views and authorized datasets grant access to derived data without granting access to underlying tables.

Column-level access control uses policy tags in Data Catalog (now Dataplex Universal Catalog); row-level security uses row access policies with filter predicates.

Cloud Storage IAM applies bucket-wide or object-level; uniform bucket-level access disables object ACLs and is recommended for consistent permission management.

VPC Service Controls create a service perimeter that restricts data exfiltration for services such as BigQuery and Cloud Storage.

Cloud KMS customer-managed encryption keys (CMEK) can encrypt BigQuery datasets and Cloud Storage buckets; BigQuery also supports customer-supplied encryption keys (CSEK) per query or load job.

BigQuery table and dataset expiration (default_table_expiration_ms) automatically deletes data after a set period; time travel retains deleted or changed data for 7 days by default.

Information Schema views (INFORMATION_SCHEMA.JOBS, TABLES, COLUMNS) expose job history, metadata, and access patterns for auditing and cost analysis.

BigQuery load jobs accept CSV, JSON (newline-delimited), Avro, Parquet, ORC; Avro and Parquet preserve nested and typed schemas, CSV does not.

Storage Transfer Service moves data from Cloud Storage, Amazon S3, Azure Blob, HTTP/HTTPS URLs, or on-premises sources into Cloud Storage or BigQuery on a schedule.

Datastream provides change data capture (CDC) from MySQL, PostgreSQL, SQL Server, and Oracle into BigQuery or Cloud Storage for near-real-time replication.

Pub/Sub plus Dataflow is the pattern for streaming ingestion when source systems have no managed connector; Pub/Sub retains messages up to 7 days by default (31 days maximum with message retention configured).

BigQuery external tables read data in place from Cloud Storage, BigLake, or Drive; they do not incur BigQuery storage cost but queries scan the external files each time.

BigQuery autodetect infers schema from a sample of rows; explicit schemas are required when column types are ambiguous, when loading empty files, or when you need strict typing.

The bq load command and the jobs.insert API both create load jobs; the --replace flag truncates the destination table, while --append (write disposition WRITE_APPEND) adds rows.

Data Transfer Service (BigQuery Data Transfer Service) schedules recurring loads from SaaS sources such as Google Ads, YouTube, and Amazon S3, and supports backfill of historical runs.

Partitioning by ingestion time or a DATE/TIMESTAMP column and clustering on high-cardinality filter columns reduce bytes scanned and cost in BigQuery.

BigQuery on-demand pricing bills per byte scanned (with a 1 TB/month free tier per project); flat-rate and slot-based editions bill for compute capacity instead.

A dry run (bq query --dry_run or the API dryRun flag) returns the bytes that would be processed without running the query or incurring cost.

SELECT * is billed for all columns; selecting only needed columns on a columnar table reduces bytes scanned and is the primary cost lever.

Looker Studio connects to BigQuery, Looker, Sheets, and other sources; BigQuery BI Engine provides an in-memory acceleration layer for sub-second dashboard queries.

Materialized views in BigQuery precompute and incrementally refresh results; they are automatically used by the query optimizer when a query matches the view definition.

Approximate aggregation functions (APPROX_COUNT_DISTINCT, APPROX_QUANTILES) trade exactness for speed and lower cost on large datasets.

Window functions (ROW_NUMBER, RANK, SUM OVER) compute per-partition results without collapsing rows; QUALIFY filters on window function output.

Looker (the enterprise BI product) uses LookML to define a semantic model; Looker Studio is the free self-service reporting tool and does not use LookML.

Cloud Composer is managed Apache Airflow; DAGs are written in Python, and operators/tasks define dependencies via >> or set_upstream/set_downstream.

Dataflow runs Apache Beam pipelines; it supports both batch and streaming from the same code, with autoscaling of worker VMs based on pipeline backlog.

Dataflow templates (Google-provided or custom Flex templates) let you launch pipelines without writing code; Flex templates package the pipeline as a container image.

Cloud Scheduler is a cron-based job trigger; it can invoke Cloud Functions, Pub/Sub, or HTTP endpoints but does not itself process data.

Cloud Functions are event-driven and suited to lightweight triggers (file arrival, Pub/Sub message) rather than long-running data transforms.

Dataform and dbt-style SQL transformation workflows orchestrate SQL in BigQuery; Dataform is a Google Cloud service integrated with BigQuery and Git repositories.

Airflow sensors (e.g., GCSObjectExistenceSensor) wait for conditions but consume worker slots; deferrable operators release the slot while waiting.

Dataproc runs managed Spark and Hadoop clusters; ephemeral clusters created per job and workflow templates are the cost-efficient pattern.

IAM roles are additive; the most permissive binding wins. BigQuery predefined roles include roles/bigquery.dataViewer, dataEditor, dataOwner, jobUser, and user.

A BigQuery dataset is the unit of access control; authorized views and authorized datasets grant access to derived data without granting access to underlying tables.

Column-level access control uses policy tags in Data Catalog (now Dataplex Universal Catalog); row-level security uses row access policies with filter predicates.

Cloud Storage IAM applies bucket-wide or object-level; uniform bucket-level access disables object ACLs and is recommended for consistent permission management.

VPC Service Controls create a service perimeter that restricts data exfiltration for services such as BigQuery and Cloud Storage.

Cloud KMS customer-managed encryption keys (CMEK) can encrypt BigQuery datasets and Cloud Storage buckets; BigQuery also supports customer-supplied encryption keys (CSEK) per query or load job.

BigQuery table and dataset expiration (default_table_expiration_ms) automatically deletes data after a set period; time travel retains deleted or changed data for 7 days by default.

Information Schema views (INFORMATION_SCHEMA.JOBS, TABLES, COLUMNS) expose job history, metadata, and access patterns for auditing and cost analysis.

BigQuery load jobs accept CSV, JSON (newline-delimited), Avro, Parquet, ORC; Avro and Parquet preserve nested and typed schemas, CSV does not.

Storage Transfer Service moves data from Cloud Storage, Amazon S3, Azure Blob, HTTP/HTTPS URLs, or on-premises sources into Cloud Storage or BigQuery on a schedule.

Datastream provides change data capture (CDC) from MySQL, PostgreSQL, SQL Server, and Oracle into BigQuery or Cloud Storage for near-real-time replication.

Pub/Sub plus Dataflow is the pattern for streaming ingestion when source systems have no managed connector; Pub/Sub retains messages up to 7 days by default (31 days maximum with message retention configured).

BigQuery external tables read data in place from Cloud Storage, BigLake, or Drive; they do not incur BigQuery storage cost but queries scan the external files each time.

BigQuery autodetect infers schema from a sample of rows; explicit schemas are required when column types are ambiguous, when loading empty files, or when you need strict typing.

The bq load command and the jobs.insert API both create load jobs; the --replace flag truncates the destination table, while --append (write disposition WRITE_APPEND) adds rows.

Data Transfer Service (BigQuery Data Transfer Service) schedules recurring loads from SaaS sources such as Google Ads, YouTube, and Amazon S3, and supports backfill of historical runs.

Partitioning by ingestion time or a DATE/TIMESTAMP column and clustering on high-cardinality filter columns reduce bytes scanned and cost in BigQuery.

BigQuery on-demand pricing bills per byte scanned (with a 1 TB/month free tier per project); flat-rate and slot-based editions bill for compute capacity instead.

A dry run (bq query --dry_run or the API dryRun flag) returns the bytes that would be processed without running the query or incurring cost.

SELECT * is billed for all columns; selecting only needed columns on a columnar table reduces bytes scanned and is the primary cost lever.

Looker Studio connects to BigQuery, Looker, Sheets, and other sources; BigQuery BI Engine provides an in-memory acceleration layer for sub-second dashboard queries.

Materialized views in BigQuery precompute and incrementally refresh results; they are automatically used by the query optimizer when a query matches the view definition.

Approximate aggregation functions (APPROX_COUNT_DISTINCT, APPROX_QUANTILES) trade exactness for speed and lower cost on large datasets.

Window functions (ROW_NUMBER, RANK, SUM OVER) compute per-partition results without collapsing rows; QUALIFY filters on window function output.

Looker (the enterprise BI product) uses LookML to define a semantic model; Looker Studio is the free self-service reporting tool and does not use LookML.

Cloud Composer is managed Apache Airflow; DAGs are written in Python, and operators/tasks define dependencies via >> or set_upstream/set_downstream.

Dataflow runs Apache Beam pipelines; it supports both batch and streaming from the same code, with autoscaling of worker VMs based on pipeline backlog.

Dataflow templates (Google-provided or custom Flex templates) let you launch pipelines without writing code; Flex templates package the pipeline as a container image.

Cloud Scheduler is a cron-based job trigger; it can invoke Cloud Functions, Pub/Sub, or HTTP endpoints but does not itself process data.

Cloud Functions are event-driven and suited to lightweight triggers (file arrival, Pub/Sub message) rather than long-running data transforms.

Dataform and dbt-style SQL transformation workflows orchestrate SQL in BigQuery; Dataform is a Google Cloud service integrated with BigQuery and Git repositories.

Airflow sensors (e.g., GCSObjectExistenceSensor) wait for conditions but consume worker slots; deferrable operators release the slot while waiting.

Dataproc runs managed Spark and Hadoop clusters; ephemeral clusters created per job and workflow templates are the cost-efficient pattern.

IAM roles are additive; the most permissive binding wins. BigQuery predefined roles include roles/bigquery.dataViewer, dataEditor, dataOwner, jobUser, and user.

A BigQuery dataset is the unit of access control; authorized views and authorized datasets grant access to derived data without granting access to underlying tables.

Column-level access control uses policy tags in Data Catalog (now Dataplex Universal Catalog); row-level security uses row access policies with filter predicates.

Cloud Storage IAM applies bucket-wide or object-level; uniform bucket-level access disables object ACLs and is recommended for consistent permission management.

VPC Service Controls create a service perimeter that restricts data exfiltration for services such as BigQuery and Cloud Storage.

Cloud KMS customer-managed encryption keys (CMEK) can encrypt BigQuery datasets and Cloud Storage buckets; BigQuery also supports customer-supplied encryption keys (CSEK) per query or load job.

BigQuery table and dataset expiration (default_table_expiration_ms) automatically deletes data after a set period; time travel retains deleted or changed data for 7 days by default.

Information Schema views (INFORMATION_SCHEMA.JOBS, TABLES, COLUMNS) expose job history, metadata, and access patterns for auditing and cost analysis.

BigQuery load jobs accept CSV, JSON (newline-delimited), Avro, Parquet, ORC; Avro and Parquet preserve nested and typed schemas, CSV does not.

Storage Transfer Service moves data from Cloud Storage, Amazon S3, Azure Blob, HTTP/HTTPS URLs, or on-premises sources into Cloud Storage or BigQuery on a schedule.

Datastream provides change data capture (CDC) from MySQL, PostgreSQL, SQL Server, and Oracle into BigQuery or Cloud Storage for near-real-time replication.

Pub/Sub plus Dataflow is the pattern for streaming ingestion when source systems have no managed connector; Pub/Sub retains messages up to 7 days by default (31 days maximum with message retention configured).

BigQuery external tables read data in place from Cloud Storage, BigLake, or Drive; they do not incur BigQuery storage cost but queries scan the external files each time.

BigQuery autodetect infers schema from a sample of rows; explicit schemas are required when column types are ambiguous, when loading empty files, or when you need strict typing.

The bq load command and the jobs.insert API both create load jobs; the --replace flag truncates the destination table, while --append (write disposition WRITE_APPEND) adds rows.

Data Transfer Service (BigQuery Data Transfer Service) schedules recurring loads from SaaS sources such as Google Ads, YouTube, and Amazon S3, and supports backfill of historical runs.

Partitioning by ingestion time or a DATE/TIMESTAMP column and clustering on high-cardinality filter columns reduce bytes scanned and cost in BigQuery.

BigQuery on-demand pricing bills per byte scanned (with a 1 TB/month free tier per project); flat-rate and slot-based editions bill for compute capacity instead.

A dry run (bq query --dry_run or the API dryRun flag) returns the bytes that would be processed without running the query or incurring cost.

SELECT * is billed for all columns; selecting only needed columns on a columnar table reduces bytes scanned and is the primary cost lever.

Looker Studio connects to BigQuery, Looker, Sheets, and other sources; BigQuery BI Engine provides an in-memory acceleration layer for sub-second dashboard queries.

Materialized views in BigQuery precompute and incrementally refresh results; they are automatically used by the query optimizer when a query matches the view definition.

Approximate aggregation functions (APPROX_COUNT_DISTINCT, APPROX_QUANTILES) trade exactness for speed and lower cost on large datasets.

Window functions (ROW_NUMBER, RANK, SUM OVER) compute per-partition results without collapsing rows; QUALIFY filters on window function output.

Looker (the enterprise BI product) uses LookML to define a semantic model; Looker Studio is the free self-service reporting tool and does not use LookML.

Cloud Composer is managed Apache Airflow; DAGs are written in Python, and operators/tasks define dependencies via >> or set_upstream/set_downstream.

Dataflow runs Apache Beam pipelines; it supports both batch and streaming from the same code, with autoscaling of worker VMs based on pipeline backlog.

Dataflow templates (Google-provided or custom Flex templates) let you launch pipelines without writing code; Flex templates package the pipeline as a container image.

Cloud Scheduler is a cron-based job trigger; it can invoke Cloud Functions, Pub/Sub, or HTTP endpoints but does not itself process data.

Cloud Functions are event-driven and suited to lightweight triggers (file arrival, Pub/Sub message) rather than long-running data transforms.

Dataform and dbt-style SQL transformation workflows orchestrate SQL in BigQuery; Dataform is a Google Cloud service integrated with BigQuery and Git repositories.

Airflow sensors (e.g., GCSObjectExistenceSensor) wait for conditions but consume worker slots; deferrable operators release the slot while waiting.

Dataproc runs managed Spark and Hadoop clusters; ephemeral clusters created per job and workflow templates are the cost-efficient pattern.

IAM roles are additive; the most permissive binding wins. BigQuery predefined roles include roles/bigquery.dataViewer, dataEditor, dataOwner, jobUser, and user.

A BigQuery dataset is the unit of access control; authorized views and authorized datasets grant access to derived data without granting access to underlying tables.

Column-level access control uses policy tags in Data Catalog (now Dataplex Universal Catalog); row-level security uses row access policies with filter predicates.

Cloud Storage IAM applies bucket-wide or object-level; uniform bucket-level access disables object ACLs and is recommended for consistent permission management.

VPC Service Controls create a service perimeter that restricts data exfiltration for services such as BigQuery and Cloud Storage.

Cloud KMS customer-managed encryption keys (CMEK) can encrypt BigQuery datasets and Cloud Storage buckets; BigQuery also supports customer-supplied encryption keys (CSEK) per query or load job.

BigQuery table and dataset expiration (default_table_expiration_ms) automatically deletes data after a set period; time travel retains deleted or changed data for 7 days by default.

Information Schema views (INFORMATION_SCHEMA.JOBS, TABLES, COLUMNS) expose job history, metadata, and access patterns for auditing and cost analysis.

BigQuery load jobs accept CSV, JSON (newline-delimited), Avro, Parquet, ORC; Avro and Parquet preserve nested and typed schemas, CSV does not.

Storage Transfer Service moves data from Cloud Storage, Amazon S3, Azure Blob, HTTP/HTTPS URLs, or on-premises sources into Cloud Storage or BigQuery on a schedule.

Datastream provides change data capture (CDC) from MySQL, PostgreSQL, SQL Server, and Oracle into BigQuery or Cloud Storage for near-real-time replication.

Pub/Sub plus Dataflow is the pattern for streaming ingestion when source systems have no managed connector; Pub/Sub retains messages up to 7 days by default (31 days maximum with message retention configured).

BigQuery external tables read data in place from Cloud Storage, BigLake, or Drive; they do not incur BigQuery storage cost but queries scan the external files each time.

BigQuery autodetect infers schema from a sample of rows; explicit schemas are required when column types are ambiguous, when loading empty files, or when you need strict typing.

The bq load command and the jobs.insert API both create load jobs; the --replace flag truncates the destination table, while --append (write disposition WRITE_APPEND) adds rows.

Data Transfer Service (BigQuery Data Transfer Service) schedules recurring loads from SaaS sources such as Google Ads, YouTube, and Amazon S3, and supports backfill of historical runs.

Partitioning by ingestion time or a DATE/TIMESTAMP column and clustering on high-cardinality filter columns reduce bytes scanned and cost in BigQuery.

BigQuery on-demand pricing bills per byte scanned (with a 1 TB/month free tier per project); flat-rate and slot-based editions bill for compute capacity instead.

A dry run (bq query --dry_run or the API dryRun flag) returns the bytes that would be processed without running the query or incurring cost.

SELECT * is billed for all columns; selecting only needed columns on a columnar table reduces bytes scanned and is the primary cost lever.

Looker Studio connects to BigQuery, Looker, Sheets, and other sources; BigQuery BI Engine provides an in-memory acceleration layer for sub-second dashboard queries.

Materialized views in BigQuery precompute and incrementally refresh results; they are automatically used by the query optimizer when a query matches the view definition.

Approximate aggregation functions (APPROX_COUNT_DISTINCT, APPROX_QUANTILES) trade exactness for speed and lower cost on large datasets.

Window functions (ROW_NUMBER, RANK, SUM OVER) compute per-partition results without collapsing rows; QUALIFY filters on window function output.

Looker (the enterprise BI product) uses LookML to define a semantic model; Looker Studio is the free self-service reporting tool and does not use LookML.

Cloud Composer is managed Apache Airflow; DAGs are written in Python, and operators/tasks define dependencies via >> or set_upstream/set_downstream.

Dataflow runs Apache Beam pipelines; it supports both batch and streaming from the same code, with autoscaling of worker VMs based on pipeline backlog.

Dataflow templates (Google-provided or custom Flex templates) let you launch pipelines without writing code; Flex templates package the pipeline as a container image.

Cloud Scheduler is a cron-based job trigger; it can invoke Cloud Functions, Pub/Sub, or HTTP endpoints but does not itself process data.

Cloud Functions are event-driven and suited to lightweight triggers (file arrival, Pub/Sub message) rather than long-running data transforms.

Dataform and dbt-style SQL transformation workflows orchestrate SQL in BigQuery; Dataform is a Google Cloud service integrated with BigQuery and Git repositories.

Airflow sensors (e.g., GCSObjectExistenceSensor) wait for conditions but consume worker slots; deferrable operators release the slot while waiting.

Dataproc runs managed Spark and Hadoop clusters; ephemeral clusters created per job and workflow templates are the cost-efficient pattern.

IAM roles are additive; the most permissive binding wins. BigQuery predefined roles include roles/bigquery.dataViewer, dataEditor, dataOwner, jobUser, and user.

A BigQuery dataset is the unit of access control; authorized views and authorized datasets grant access to derived data without granting access to underlying tables.

Column-level access control uses policy tags in Data Catalog (now Dataplex Universal Catalog); row-level security uses row access policies with filter predicates.

Cloud Storage IAM applies bucket-wide or object-level; uniform bucket-level access disables object ACLs and is recommended for consistent permission management.

VPC Service Controls create a service perimeter that restricts data exfiltration for services such as BigQuery and Cloud Storage.

Cloud KMS customer-managed encryption keys (CMEK) can encrypt BigQuery datasets and Cloud Storage buckets; BigQuery also supports customer-supplied encryption keys (CSEK) per query or load job.

BigQuery table and dataset expiration (default_table_expiration_ms) automatically deletes data after a set period; time travel retains deleted or changed data for 7 days by default.

Information Schema views (INFORMATION_SCHEMA.JOBS, TABLES, COLUMNS) expose job history, metadata, and access patterns for auditing and cost analysis.

BigQuery load jobs accept CSV, JSON (newline-delimited), Avro, Parquet, ORC; Avro and Parquet preserve nested and typed schemas, CSV does not.

Storage Transfer Service moves data from Cloud Storage, Amazon S3, Azure Blob, HTTP/HTTPS URLs, or on-premises sources into Cloud Storage or BigQuery on a schedule.

Datastream provides change data capture (CDC) from MySQL, PostgreSQL, SQL Server, and Oracle into BigQuery or Cloud Storage for near-real-time replication.

Pub/Sub plus Dataflow is the pattern for streaming ingestion when source systems have no managed connector; Pub/Sub retains messages up to 7 days by default (31 days maximum with message retention configured).

BigQuery external tables read data in place from Cloud Storage, BigLake, or Drive; they do not incur BigQuery storage cost but queries scan the external files each time.

BigQuery autodetect infers schema from a sample of rows; explicit schemas are required when column types are ambiguous, when loading empty files, or when you need strict typing.

The bq load command and the jobs.insert API both create load jobs; the --replace flag truncates the destination table, while --append (write disposition WRITE_APPEND) adds rows.

Data Transfer Service (BigQuery Data Transfer Service) schedules recurring loads from SaaS sources such as Google Ads, YouTube, and Amazon S3, and supports backfill of historical runs.

Partitioning by ingestion time or a DATE/TIMESTAMP column and clustering on high-cardinality filter columns reduce bytes scanned and cost in BigQuery.

BigQuery on-demand pricing bills per byte scanned (with a 1 TB/month free tier per project); flat-rate and slot-based editions bill for compute capacity instead.

A dry run (bq query --dry_run or the API dryRun flag) returns the bytes that would be processed without running the query or incurring cost.

SELECT * is billed for all columns; selecting only needed columns on a columnar table reduces bytes scanned and is the primary cost lever.

Looker Studio connects to BigQuery, Looker, Sheets, and other sources; BigQuery BI Engine provides an in-memory acceleration layer for sub-second dashboard queries.

Materialized views in BigQuery precompute and incrementally refresh results; they are automatically used by the query optimizer when a query matches the view definition.

Approximate aggregation functions (APPROX_COUNT_DISTINCT, APPROX_QUANTILES) trade exactness for speed and lower cost on large datasets.

Window functions (ROW_NUMBER, RANK, SUM OVER) compute per-partition results without collapsing rows; QUALIFY filters on window function output.

Looker (the enterprise BI product) uses LookML to define a semantic model; Looker Studio is the free self-service reporting tool and does not use LookML.

Cloud Composer is managed Apache Airflow; DAGs are written in Python, and operators/tasks define dependencies via >> or set_upstream/set_downstream.

Dataflow runs Apache Beam pipelines; it supports both batch and streaming from the same code, with autoscaling of worker VMs based on pipeline backlog.

Dataflow templates (Google-provided or custom Flex templates) let you launch pipelines without writing code; Flex templates package the pipeline as a container image.

Cloud Scheduler is a cron-based job trigger; it can invoke Cloud Functions, Pub/Sub, or HTTP endpoints but does not itself process data.

Cloud Functions are event-driven and suited to lightweight triggers (file arrival, Pub/Sub message) rather than long-running data transforms.

Dataform and dbt-style SQL transformation workflows orchestrate SQL in BigQuery; Dataform is a Google Cloud service integrated with BigQuery and Git repositories.

Airflow sensors (e.g., GCSObjectExistenceSensor) wait for conditions but consume worker slots; deferrable operators release the slot while waiting.

Dataproc runs managed Spark and Hadoop clusters; ephemeral clusters created per job and workflow templates are the cost-efficient pattern.

IAM roles are additive; the most permissive binding wins. BigQuery predefined roles include roles/bigquery.dataViewer, dataEditor, dataOwner, jobUser, and user.

A BigQuery dataset is the unit of access control; authorized views and authorized datasets grant access to derived data without granting access to underlying tables.

Column-level access control uses policy tags in Data Catalog (now Dataplex Universal Catalog); row-level security uses row access policies with filter predicates.

Cloud Storage IAM applies bucket-wide or object-level; uniform bucket-level access disables object ACLs and is recommended for consistent permission management.

VPC Service Controls create a service perimeter that restricts data exfiltration for services such as BigQuery and Cloud Storage.

Cloud KMS customer-managed encryption keys (CMEK) can encrypt BigQuery datasets and Cloud Storage buckets; BigQuery also supports customer-supplied encryption keys (CSEK) per query or load job.

BigQuery table and dataset expiration (default_table_expiration_ms) automatically deletes data after a set period; time travel retains deleted or changed data for 7 days by default.

Information Schema views (INFORMATION_SCHEMA.JOBS, TABLES, COLUMNS) expose job history, metadata, and access patterns for auditing and cost analysis.

BigQuery load jobs accept CSV, JSON (newline-delimited), Avro, Parquet, ORC; Avro and Parquet preserve nested and typed schemas, CSV does not.

Storage Transfer Service moves data from Cloud Storage, Amazon S3, Azure Blob, HTTP/HTTPS URLs, or on-premises sources into Cloud Storage or BigQuery on a schedule.

Datastream provides change data capture (CDC) from MySQL, PostgreSQL, SQL Server, and Oracle into BigQuery or Cloud Storage for near-real-time replication.

Pub/Sub plus Dataflow is the pattern for streaming ingestion when source systems have no managed connector; Pub/Sub retains messages up to 7 days by default (31 days maximum with message retention configured).

BigQuery external tables read data in place from Cloud Storage, BigLake, or Drive; they do not incur BigQuery storage cost but queries scan the external files each time.

BigQuery autodetect infers schema from a sample of rows; explicit schemas are required when column types are ambiguous, when loading empty files, or when you need strict typing.

The bq load command and the jobs.insert API both create load jobs; the --replace flag truncates the destination table, while --append (write disposition WRITE_APPEND) adds rows.

Data Transfer Service (BigQuery Data Transfer Service) schedules recurring loads from SaaS sources such as Google Ads, YouTube, and Amazon S3, and supports backfill of historical runs.

Partitioning by ingestion time or a DATE/TIMESTAMP column and clustering on high-cardinality filter columns reduce bytes scanned and cost in BigQuery.

BigQuery on-demand pricing bills per byte scanned (with a 1 TB/month free tier per project); flat-rate and slot-based editions bill for compute capacity instead.

A dry run (bq query --dry_run or the API dryRun flag) returns the bytes that would be processed without running the query or incurring cost.

SELECT * is billed for all columns; selecting only needed columns on a columnar table reduces bytes scanned and is the primary cost lever.

Looker Studio connects to BigQuery, Looker, Sheets, and other sources; BigQuery BI Engine provides an in-memory acceleration layer for sub-second dashboard queries.

Materialized views in BigQuery precompute and incrementally refresh results; they are automatically used by the query optimizer when a query matches the view definition.

Approximate aggregation functions (APPROX_COUNT_DISTINCT, APPROX_QUANTILES) trade exactness for speed and lower cost on large datasets.

Window functions (ROW_NUMBER, RANK, SUM OVER) compute per-partition results without collapsing rows; QUALIFY filters on window function output.

Looker (the enterprise BI product) uses LookML to define a semantic model; Looker Studio is the free self-service reporting tool and does not use LookML.

Cloud Composer is managed Apache Airflow; DAGs are written in Python, and operators/tasks define dependencies via >> or set_upstream/set_downstream.

Dataflow runs Apache Beam pipelines; it supports both batch and streaming from the same code, with autoscaling of worker VMs based on pipeline backlog.

Dataflow templates (Google-provided or custom Flex templates) let you launch pipelines without writing code; Flex templates package the pipeline as a container image.

Cloud Scheduler is a cron-based job trigger; it can invoke Cloud Functions, Pub/Sub, or HTTP endpoints but does not itself process data.

Cloud Functions are event-driven and suited to lightweight triggers (file arrival, Pub/Sub message) rather than long-running data transforms.

Dataform and dbt-style SQL transformation workflows orchestrate SQL in BigQuery; Dataform is a Google Cloud service integrated with BigQuery and Git repositories.

Airflow sensors (e.g., GCSObjectExistenceSensor) wait for conditions but consume worker slots; deferrable operators release the slot while waiting.

Dataproc runs managed Spark and Hadoop clusters; ephemeral clusters created per job and workflow templates are the cost-efficient pattern.

IAM roles are additive; the most permissive binding wins. BigQuery predefined roles include roles/bigquery.dataViewer, dataEditor, dataOwner, jobUser, and user.

A BigQuery dataset is the unit of access control; authorized views and authorized datasets grant access to derived data without granting access to underlying tables.

Column-level access control uses policy tags in Data Catalog (now Dataplex Universal Catalog); row-level security uses row access policies with filter predicates.

Cloud Storage IAM applies bucket-wide or object-level; uniform bucket-level access disables object ACLs and is recommended for consistent permission management.

VPC Service Controls create a service perimeter that restricts data exfiltration for services such as BigQuery and Cloud Storage.

Cloud KMS customer-managed encryption keys (CMEK) can encrypt BigQuery datasets and Cloud Storage buckets; BigQuery also supports customer-supplied encryption keys (CSEK) per query or load job.

BigQuery table and dataset expiration (default_table_expiration_ms) automatically deletes data after a set period; time travel retains deleted or changed data for 7 days by default.

Information Schema views (INFORMATION_SCHEMA.JOBS, TABLES, COLUMNS) expose job history, metadata, and access patterns for auditing and cost analysis.