ACA Big Data flashcards
ACA Big Data · Build recall with a focused review session.
34 cards. Read the term and the definition here. The deck at the top of this page is optional.
- Three Vs of big data
- Volume (data scale), Velocity (speed of arrival and processing), and Variety (structured, semi-structured, unstructured formats). Later authors add Veracity and Value.
- Batch versus stream processing
- Batch processes bounded data at rest with high latency and full reprocessing; stream processes unbounded data in flight with low latency and continuous state.
- Lambda architecture
- A design combining a batch layer for accurate historical results, a speed layer for low-latency recent results, and a serving layer that merges both for queries.
- Kappa architecture
- An architecture that removes the batch layer and treats all data as a stream, reprocessing history by replaying the log through the same streaming job.
- Hologres HSAP engine
- Alibaba Cloud's hybrid serving and analytical processing database that answers high-concurrency point queries and large OLAP scans from one columnar store.
- DataHub as ingestion bus
- Alibaba Cloud streaming data bus that collects logs, metrics, and events and delivers them to MaxCompute, Flink, or OSS for downstream processing.
- CAP theorem
- A distributed data store under a network partition can guarantee only consistency or availability; partition tolerance is assumed mandatory in practice.
- Schema-on-read versus write
- Schema-on-write validates structure at load time (warehouse tables); schema-on-read applies structure at query time (files in a data lake).
- DataWorks Data Integration
- The synchronization module that moves data between sources and MaxCompute using offline full/incremental sync or real-time streaming sync tasks.
- bizdate scheduling parameter
- Built-in DataWorks variable resolving to the prior day in yyyymmdd format by default, used for partition filters in incremental sync and SQL nodes.
- Cross-project dependency
- A DataWorks node dependency expressed as projectname.nodename, letting a node in one workspace wait on a node in another before running.
- DataWorks Operation Center
- The monitoring console showing scheduled instance status, DAG dependencies, and logs, and providing rerun, pause, and fault-recovery actions.
- Data Quality blocking rule
- A validation rule whose failure marks the instance failed and stops downstream nodes from running, unlike a warning rule that only alerts.
- DataStudio commit versus deploy
- Saving edits a draft; committing publishes the node to scheduling; in multi-environment workspaces a deploy step then pushes it to production.
- Data Map lineage
- A DataWorks feature that traces table and field lineage end-to-end by parsing SQL and node dependencies, supporting impact analysis.
- MaxCompute project
- The top-level MaxCompute unit providing namespace isolation, quota and billing boundaries, and containing tables, views, functions, and resources.
- Partition pruning
- Restricting a query with a partition filter so the engine reads only matching partitions, cutting scanned bytes and cost.
- LEFT SEMI join
- A join returning left-table rows that have at least one match on the right, without adding right-table columns; used for existence filtering.
- Map-side join hint
- The /*+ mapjoin(small_table) */ hint that broadcasts a small table into memory to avoid a shuffle during a join.
- ROW_NUMBER window function
- Assigns a unique sequential integer per row within a partition ordered by a key, commonly used to keep one latest record per group.
- RANK versus DENSE_RANK
- RANK leaves gaps after ties (1,1,3); DENSE_RANK assigns consecutive values after ties (1,1,2). Both differ from ROW_NUMBER's uniqueness.
- Table LIFECYCLE setting
- A MaxCompute table property that automatically deletes partitions older than the specified number of days, controlling storage retention.
- MaxCompute Tunnel
- The data transfer service used to upload and download bulk data to MaxCompute tables, billed by volume moved rather than bytes scanned.
- Event time versus processing time
- Event time is when a record was generated, embedded in the data; processing time is when the engine handles it. Windows on event time need watermarks.
- Watermark in Flink
- A monotonically increasing marker asserting no events earlier than time T will follow, letting event-time windows close and fire.
- Tumbling versus sliding window
- Tumbling windows are fixed-size and non-overlapping; sliding windows have a fixed size but advance by a step, so records can belong to several windows.
- Session window
- A dynamic window that groups events separated by less than a configured inactivity gap and closes after the gap elapses.
- Checkpoint versus savepoint
- Checkpoints are periodic automatic state snapshots for failure recovery; savepoints are manually triggered snapshots used for upgrades and job migration.
- Exactly-once semantics
- A processing guarantee where each record affects the result once even after failures, requiring checkpointed state plus transactional or idempotent sinks.
- Kafka partition parallelism
- Each Kafka partition is consumed by at most one consumer subtask within a group, so partition count caps achievable source parallelism.
- Quick BI dataset layer
- The semantic layer between data source and dashboard that defines fields, calculated measures, aggregation, and row-level permission rules.
- Quick BI row-level permission
- A dataset rule that filters visible rows at query time based on the signed-in user's attributes, such as matching region to the user's region.
- RAM and STS
- Resource Access Management controls identities and API permissions; Security Token Service issues temporary credentials for short-lived, scoped access.
- ActionTrail audit
- An Alibaba Cloud service recording API calls and console actions across accounts, used for security investigation and compliance evidence.