IBM Data Science Professional Certificate cheat sheet

CRISP-DM phases in order: business understanding, data understanding, data preparation, modeling, evaluation, deployment; iteration back to earlier phases is normal.

OSEMN pipeline: Obtain, Scrub, Explore, Model, iNterpret - a lightweight alternative framing to CRISP-DM.

Data types: nominal (unordered categories), ordinal (ordered categories), interval (equal spacing, no true zero), ratio (equal spacing plus true zero).

Population vs sample: population parameters are usually unknown; sample statistics (mean, proportion) estimate them, with sampling error shrinking roughly as 1/sqrt(n).

Central tendency: mean is sensitive to outliers; median is robust; mode suits nominal data. Spread: range, IQR (Q3-Q1), variance, standard deviation.

Correlation coefficient r ranges -1 to 1 and measures linear association only; correlation does not imply causation, and r is not the slope of a regression line.

Hypothesis testing logic: assume H0, compute a test statistic, derive a p-value; reject H0 when p < alpha (commonly 0.05). p is not the probability that H0 is true.

Type I error = rejecting a true H0 (probability alpha); Type II error = failing to reject a false H0 (probability beta); power = 1 - beta.

Bias-variance tradeoff: high bias underfits (too simple), high variance overfits (too sensitive to training noise); total error decomposes into bias^2 + variance + irreducible error.

Core containers: list (ordered, mutable, allows duplicates), tuple (ordered, immutable), set (unordered, unique elements), dict (key-value mapping, keys hashable).

Slicing rule seq[start:stop:step] includes start, excludes stop; negative indices count from the end; seq[::-1] reverses.

Mutability trap: assigning b = a for a list aliases the object; use a.copy() or a[:] for a shallow copy, and copy.deepcopy for nested structures.

Comprehensions: [f(x) for x in xs if cond] builds a list; {k: v for ...} builds a dict; generator expressions with parentheses are lazy and memory-efficient.

Functions: default arguments are evaluated once at definition time, so never use a mutable default (def f(x, acc=[])). *args collects positional, **kwargs keyword arguments.

Exception handling: try/except/else/finally; catch specific exception types; raise re-raises or raises new errors; with-statements use context managers to guarantee cleanup.

PEP 8 essentials: 4-space indentation, snake_case for functions and variables, CapWords for classes, UPPER_CASE for constants, two blank lines between top-level definitions.

Virtual environments isolate dependencies: python -m venv env, then activate, then pip install; requirements.txt pins reproducible package versions.

Logical execution order: FROM/JOIN, WHERE, GROUP BY, HAVING, SELECT, DISTINCT, ORDER BY, LIMIT - so WHERE cannot filter aggregates and column aliases are unavailable in WHERE.

WHERE filters rows before grouping; HAVING filters groups after aggregation. COUNT(*) counts rows; COUNT(col) skips NULLs.

JOIN types: INNER returns matches only; LEFT/RIGHT keep all rows of one side with NULLs for misses; FULL OUTER keeps both; CROSS JOIN is the Cartesian product.

NULL semantics: NULL is unknown, so col = NULL is never true; use IS NULL / IS NOT NULL. Aggregates other than COUNT(*) ignore NULLs; COALESCE substitutes defaults.

Aggregate functions: COUNT, SUM, AVG, MIN, MAX; AVG ignores NULLs, so AVG(COALESCE(col,0)) changes the denominator and the result.

Subqueries: scalar subqueries return one value; IN/EXISTS test membership; correlated subqueries reference the outer query and run per outer row.

Window functions: ROW_NUMBER, RANK, DENSE_RANK assign orderings; SUM(x) OVER (PARTITION BY g ORDER BY d) computes running totals without collapsing rows.

Set operations UNION (deduplicates) vs UNION ALL (keeps duplicates) require matching column counts and compatible types; GROUP BY groups NULLs together.

NumPy arrays are homogeneous and vectorized; broadcasting stretches compatible shapes (e.g., (3,1) with (1,4) yields (3,4)); axis=0 aggregates down rows, axis=1 across columns.

Boolean indexing: arr[arr > 0] filters; np.where(cond, a, b) selects elementwise; fancy indexing with an integer array returns a copy, basic slicing returns a view.

Pandas Series is 1-D labeled; DataFrame is 2-D with row index and column labels. .loc selects by label (inclusive of the stop label), .iloc by integer position (exclusive).

Missing data: df.isna().sum() counts; df.dropna() removes; df.fillna(value) imputes; df.replace() maps values. NaN propagates through arithmetic unless skipna applies.

Reshaping: df.pivot_table(values, index, columns, aggfunc) aggregates; df.melt turns wide to long; pd.concat stacks along an axis; df.merge joins on keys like SQL.

Grouping: df.groupby(key)[col].agg(['mean','count']) returns a MultiIndex result; .reset_index() flattens it; .transform() broadcasts group results back to row shape.

Data quality checks: dtype inspection with df.info(), duplicate removal with df.drop_duplicates(), range checks, and outlier flags via IQR (below Q1-1.5IQR or above Q3+1.5IQR).

String and date handling: .str accessor for text cleaning, pd.to_datetime for parsing, .dt accessor for year/month extraction, and timezone-aware timestamps for cross-region joins.

Encoding hierarchy for accuracy: position on a common scale > length > angle/slope > area > color hue; pie charts encode by angle and area and are weak for fine comparisons.

Chart choice by question: distribution (histogram, box plot, violin), relationship (scatter, line), comparison across categories (bar), composition (stacked bar, treemap).

Histograms depend on bin width; always state binning. Box plots show median, quartiles, and IQR-based whiskers, hiding multimodality that a violin or histogram reveals.

Axis rules: bar charts must start at zero because length encodes value; line charts may use a non-zero baseline when the trend matters, but the truncation must be labeled.

Color: sequential palettes for ordered magnitude, diverging for deviations around a midpoint, qualitative for unordered categories; avoid red-green pairs for accessibility.

Matplotlib object hierarchy: Figure contains Axes; plt.subplots(nrows, ncols) creates both; ax.set_xlabel, ax.set_title, ax.legend control annotation.

Seaborn builds on Matplotlib: sns.histplot, sns.boxplot, sns.scatterplot, sns.heatmap for correlation matrices; the figure-level functions (relplot, catplot) return FacetGrid.

Dashboard design: one key metric per view, consistent scales across compared panels, filters that update all linked charts, and a stated data refresh timestamp.

Supervised vs unsupervised: labeled targets (regression, classification) vs unlabeled structure discovery (clustering, dimensionality reduction).

Train/test split (commonly 70/30 or 80/20) estimates generalization; k-fold cross-validation (k=5 or 10) reduces variance of the estimate; the test set is touched once.

Classification metrics from the confusion matrix: accuracy, precision = TP/(TP+FP), recall = TP/(TP+FN), F1 = harmonic mean of precision and recall; ROC-AUC measures ranking across thresholds.

Regression metrics: MAE, MSE, RMSE (same units as target), R^2 = 1 - SS_res/SS_tot; R^2 can be negative on unseen data for a badly fit model.

Regularization: L1 (Lasso) drives coefficients to zero and selects features; L2 (Ridge) shrinks coefficients; the strength is tuned by cross-validated grid search.

k-NN requires feature scaling because distance dominates; k too small overfits, k too large oversmooths. Decision trees split on impurity reduction (Gini or entropy).

k-means minimizes within-cluster sum of squares, requires k in advance, and assumes roughly spherical clusters; the elbow method and silhouette score guide k.

Leakage rule: fit scalers, imputers, and encoders inside a Pipeline on training folds only; fitting before splitting inflates reported performance.

Standard capstone arc: problem statement, data acquisition (API or web scraping), wrangling, exploratory analysis, modeling or mapping, then a stakeholder-facing report.

REST API calls use requests.get(url, params=...); check response.status_code == 200, then response.json() parses the payload; pagination and rate limits must be handled explicitly.

Web scraping with BeautifulSoup: soup.find / find_all target tags and attributes; scraping is brittle to layout changes and must respect site terms and robots rules.

Geospatial stack: geopy Nominatim geocodes addresses to latitude/longitude; Folium builds interactive Leaflet maps with markers, choropleths, and CircleMarker overlays.

Foursquare-style venue queries return categories and coordinates; joining venue counts to neighborhood polygons supports cluster comparison across districts.

k-means on coordinates groups venues into districts; results depend on feature scaling, random seed, and k, so report the silhouette score alongside the clusters.

Reproducibility in the capstone: pin library versions, seed random generators, cache raw API responses, and separate notebooks for acquisition, analysis, and reporting.

Deliverable structure: executive summary with the business question and finding, methods, limitations, and one recommended action; every chart must answer a stated question.

Jupyter notebook cell types: code cells execute and persist state in the kernel; markdown cells render documentation; execution order matters because state is cumulative, not top-to-bottom guaranteed.

Magic commands: %matplotlib inline renders plots in the notebook; %timeit benchmarks a statement; !shell runs shell commands; %run executes an external script into the kernel.

Kernel management: restart-and-run-all produces a clean reproducible state; a stale kernel explains results that differ from a fresh run.

Git core workflow: git status, git add, git commit -m, git push; branches isolate work and pull requests review merges into the main branch.

GitHub artifacts: README documents the project, .gitignore excludes data and credentials, issues track tasks, and releases tag stable versions.

Never commit secrets or large raw data; use environment variables or platform credentials, and keep data in object storage referenced by path.

IBM Watson Studio concepts: project holds assets, notebooks run against an environment/runtime, and deployment exposes a model as an online endpoint for scoring.

Watson Machine Learning flow: train in a notebook, store the model in the repository, create a deployment, then score via the endpoint API with a token.

Audience-first framing: lead with the decision or recommendation, then the evidence; executives need the finding and its impact, analysts need method and caveats.

Narrative structure: situation, complication, question, answer (SCQA) or context-problem-solution; each slide carries one message in its title.

Chart annotation rules: label axes and units, state the time window and sample size, cite the data source, and remove non-data ink (gridlines, 3-D effects).

Uncertainty reporting: give confidence intervals or ranges rather than single point estimates, and state the assumptions behind any forecast.

Data ethics: obtain informed consent, minimize personally identifiable information, anonymize before sharing, and document known limitations and disparate impact.

Stakeholder pushback: separate the claim from the evidence, quantify the sensitivity of the conclusion to key assumptions, and propose a validation step.

Documentation standard: a README with purpose, data dictionary, setup steps, and results lets another analyst reproduce the analysis without asking questions.

Version control as communication: commit messages state what changed and why; a clean history and reviewed pull requests make analytical work auditable.