Data Ingestion and Modeling
Ingestion paths
Data streams are the entry point. Connectors cover Salesforce CRM and other Salesforce clouds, cloud storage (S3, Azure Blob, GCS), warehouses (Snowflake, BigQuery, Redshift), SFTP and file uploads, web and mobile SDKs, MuleSoft, Platform Events, and the Ingestion API. Choose the ingestion type deliberately: batch for periodic bulk loads, streaming for continuous events, and the Ingestion API in either bulk (file-based) or streaming (record-by-record) mode.
From raw to harmonized
Each stream lands data in a DLO. Semi-structured payloads such as JSON get schema detection and flattening, and transforms (batch or streaming) let you reshape data with SQL-style logic. Field mapping then binds DLO fields to DMO fields, requiring a primary key, correct data types, and relationship mappings for lookups. Formula fields derive values on a DMO without re-ingesting. Relationships such as Contact Point Email to Individual are what allow segmentation on related attributes.
Traps
- Mixing up a stream refresh (re-pulling source data) with a mapping edit.
- Assuming you can segment directly on a DLO field or on an unmapped field.
- Treating the Ingestion API as a single mode when it has bulk and streaming behaviors.
- Mapping a source identifier to the wrong DMO field or type, or omitting the primary key.
- Believing transforms replace mapping - mapping is still required to reach the DMO.
Rule of thumb
Ingest raw, transform for shape, map for meaning. If a source field's meaning needs to be derived or cleaned, that belongs in a transform; if it identifies or describes a known entity, it belongs in a mapping to a standard DMO.