ACA Big Data cheat sheet

The five V's: Volume, Velocity, Variety, Veracity, Value; Value is the only V describing business worth extracted, not a data property.

Batch processing operates on bounded, at-rest datasets with high latency; stream processing operates on unbounded in-flight data with low latency.

Lambda architecture = batch layer + speed layer + serving layer; Kappa architecture drops the batch layer and reprocesses streams only.

Alibaba Cloud big data product families: MaxCompute (warehouse compute), DataWorks (integration/scheduling/governance), Hologres (HSAP), Flink (streaming), E-MapReduce (Hadoop/Spark), DataHub (stream ingestion).

Hologres is a hybrid serving and analytical processing (HSAP) engine supporting high-concurrency point queries and OLAP in one system.

Data Lake Analytics (DLA) and Object Storage Service (OSS) support schema-on-read lake patterns; MaxCompute uses schema-on-write tables.

CAP theorem: under a network partition a distributed store chooses consistency or availability, never both.

Sharding/partitioning scales horizontally by key range or hash; replication scales reads and improves fault tolerance at the cost of consistency work.

DataWorks core modules: Data Integration (sync), DataStudio (develop/schedule), Operation Center (monitor/rerun), Data Quality, Data Map (lineage), Security Guard.

Data Integration offline sync modes: full (whole table) and incremental (WHERE-based watermark, e.g. gmt_modified > ${bizdate}); real-time sync uses a streaming reader.

Scheduling parameters: ${bizdate} is the prior day in yyyymmdd by default; ${bdp.system.cyctime} is the instance scheduled run timestamp.

Node dependency types: cross-node (same project), cross-project (projectname.nodename), and workspace root; a node runs only after all upstream instances succeed.

Instance statuses progress WAIT -> RUNNING -> SUCCESS/FAILURE; rerun regenerates downstream instances only when dependencies are re-triggered.

Data Quality rules: table-level (row count, size change) and column-level (null count, distinct count, custom SQL); blocking rules stop downstream scheduling.

DataStudio commit publishes to the scheduling system; changes are not scheduled until committed and (in multi-environment) deployed to production.

Data Map provides end-to-end lineage from source table to target table via node dependencies and SQL parsing.

MaxCompute objects: Project (isolation and billing unit), Table, Partition, View, Resource, Function, and Tunnel for data transfer.

Partitioned tables require a partition spec on write (e.g. ds='20240101'); full-table scans without partition pruning raise cost and fail on limits.

SQL execution order: FROM -> WHERE -> GROUP BY -> HAVING -> SELECT -> DISTINCT -> ORDER BY -> LIMIT; aliases from SELECT are usable in ORDER BY but not WHERE.

Join types: INNER, LEFT/RIGHT OUTER, FULL OUTER, LEFT SEMI (existence, no right columns), ANTI JOIN (non-existence); map-join hint /+ mapjoin(t)/ avoids shuffling a small table.

GROUP BY aggregation with DISTINCT inside one aggregate is restricted; use nested subqueries or ROW_NUMBER() for deduplication.

Window functions: ROW_NUMBER (unique ranking), RANK (gaps on ties), DENSE_RANK (no gaps), plus LAG/LEAD for adjacent-row access.

Built-in functions: GETDATE(), TO_CHAR(date,'yyyymmdd'), DATEADD, CONCAT, COALESCE, NVL, CASE WHEN; STRING type is variable-length UTF-8.

Lifecycle (LIFECYCLE) on tables auto-drops partitions older than N days; Tunnel uploads are billed by data volume, SQL by scanned bytes.

Flink core abstractions: DataStream API, Source -> Transformation -> Sink; the job graph is built from operators and executed by TaskManagers under a JobManager.

Event time vs processing time: event time uses timestamps embedded in records and requires watermarks to bound out-of-order data.

Watermark = a monotonically advancing assertion that no event earlier than time T will arrive; late events after the watermark are dropped or side-output.

Windows: tumbling (fixed, non-overlapping), sliding (fixed size, sliding step), session (gap-based); window fires when watermark passes window end.

State backends: HashMapStateBackend (heap) and EmbeddedRocksDBStateBackend (disk, large state); checkpoints give exactly-once via barriers.

Exactly-once needs checkpointing plus a transactional/idempotent sink; at-least-once allows duplicates but never loss.

Kafka partitions bound parallelism of a source; each partition is consumed by at most one subtask of a consumer group member.

DataHub and Log Service (SLS) are common Alibaba Cloud streaming sources; Realtime Compute for Apache Flink is the managed runtime.

Quick BI asset hierarchy: data source -> dataset -> dashboard; a dataset defines fields, calculated fields, and row-level permissions.

Chart selection rule: comparison across categories = bar; trend over time = line; part-to-whole = stacked/pie; distribution = histogram; correlation = scatter.

Row-level permission in Quick BI filters rows by a user-attribute mapping (e.g. region = user's region) evaluated at query time.

DataWorks Security Guard classification levels (e.g. L1-L4) drive access approval; higher sensitivity requires explicit project/table authorization.

MaxCompute ACL: project-level roles (Admin, Developer, Guest) plus table/column grants; package mechanism shares resources across projects.

RAM (Resource Access Management) governs identity and API permissions; STS issues temporary credentials for short-lived access.

Encryption: server-side at rest (OSS SSE, MaxCompute transparent encryption) and in transit via TLS; masking functions hide PII in query output.

Audit and monitoring use ActionTrail (API audit logs) and DataWorks Operation Center logs; anomalies are detected via Data Quality and baseline alerts.