OpenEDG PCAD cheat sheet

requests.get(url, params=..., timeout=10) returns a Response; call .raise_for_status() to turn 4xx/5xx into HTTPError, and .json() to decode JSON bodies.

Pagination patterns: offset/limit (REST), cursor tokens, and Link header rel="next"; loop until the next token is empty, never assume a fixed page count.

Rate limiting: HTTP 429 with Retry-After header; exponential backoff sleep = base * 2**attempt with jitter avoids synchronized retries.

Web scraping: BeautifulSoup(html, 'html.parser') then .select() for CSS selectors or .find_all(); robots.txt and site Terms of Service govern permission before scraping.

pd.read_sql(query, con=engine) returns a DataFrame; SQLAlchemy create_engine('sqlite:///file.db') or 'postgresql+psycopg2://user:pw@host/db' provides the connection.

pd.read_csv(path, sep=',', dtype={...}, na_values=[...], parse_dates=[...]) controls type inference; chunking via chunksize returns an iterator of DataFrames for large files.

API auth: keys in headers (Authorization: Bearer token) rather than query strings, because URLs are logged; secrets belong in environment variables, not source code.

Data integration joins: pd.merge(left, right, how='inner'|'left'|'outer', on='key') mirrors SQL JOIN; validate row counts before and after to detect key duplication (many-to-many blowup).

Missing data: df.isna().sum() counts nulls; df.dropna(subset=[...]) removes rows; df.fillna(value) or .fillna(df['col'].median()) imputes; distinguish MCAR, MAR, MNAR before choosing.

Duplicates: df.duplicated(subset=[...]).sum() counts, df.drop_duplicates(subset=[...], keep='first') removes; keep='last' or keep=False change which rows survive.

Type coercion: pd.to_numeric(s, errors='coerce') turns unparseable strings into NaN; pd.to_datetime(s, errors='coerce', format='%Y-%m-%d') is faster and stricter than inference.

Outliers: IQR rule flags values below Q1 - 1.5IQR or above Q3 + 1.5IQR; z-score rule flags |x - mean| / std > 3, which assumes approximate normality.

String cleaning: s.str.strip(), .str.lower(), .str.replace(r'\s+', ' ', regex=True); normalize categories before grouping to avoid 'NY' vs 'ny' splits.

Validation rules: assert df['age'].between(0, 120).all(); check referential integrity (foreign keys present in parent table), uniqueness of primary keys, and allowed-value sets.

Schema checks: df.dtypes, df.shape, df.columns and df.describe() form a baseline profile; compare against expected schema to catch upstream changes early.

Selection: df.loc[row_label, col_label] is label-based and inclusive; df.iloc[row_pos, col_pos] is positional and exclusive of the end; df.query('col > 5') filters by expression.

Grouping: df.groupby('key', as_index=False)['val'].agg(['mean','sum','count']) or .agg(total=('val','sum')); named aggregation avoids MultiIndex columns.

Broadcasting: NumPy aligns shapes from the right; a (m, n) array with a (n,) vector works, (m,) with (n,) fails unless one is 1 or reshaped to (m, 1).

Vectorization beats loops: np.where(cond, a, b) and boolean masks avoid Python-level iteration; df.apply is a fallback when no vectorized operation exists.

Pivot and reshape: df.pivot_table(index=..., columns=..., values=..., aggfunc='mean') aggregates duplicates; df.melt() converts wide to long; df.pivot() requires unique index/column pairs.

Combining: pd.concat([...], axis=0) stacks rows aligning columns by name; axis=1 aligns on index, so mismatched indexes introduce NaN.

NumPy reductions: axis=0 reduces rows (per-column result), axis=1 reduces columns (per-row result); np.nanmean and np.nansum ignore NaN while np.mean propagates it.

Sorting and ranking: df.sort_values(['a','b'], ascending=[True, False]); df.nlargest(5, 'score') is faster than a full sort for top-k.

Logical execution order: FROM/JOIN -> WHERE -> GROUP BY -> HAVING -> SELECT -> DISTINCT -> ORDER BY -> LIMIT; WHERE filters rows before aggregation, HAVING filters groups after.

Joins: INNER returns matches only; LEFT keeps all left rows with NULLs for misses; FULL OUTER keeps both sides; a join on a non-unique key multiplies rows.

Aggregates: COUNT(*) counts rows, COUNT(col) skips NULLs; SUM/AVG/MIN/MAX ignore NULLs; GROUP BY requires every non-aggregated selected column to be listed.

NULL semantics: comparisons with NULL yield UNKNOWN, so use IS NULL / IS NOT NULL; COALESCE(col, default) supplies fallbacks.

Window functions: ROW_NUMBER() OVER (PARTITION BY g ORDER BY t DESC) numbers rows per group; RANK leaves gaps after ties, DENSE_RANK does not.

Parameterized queries: cur.execute('SELECT * FROM t WHERE id = ?', (id,)) prevents SQL injection; never build SQL with f-strings from user input.

Transactions: connection.commit() persists, .rollback() undoes; context managers (with engine.connect() as conn:) close resources reliably.

DB-API flow: connect() -> cursor() -> execute() -> fetchone()/fetchall() -> commit() -> close(); pandas read_sql wraps this and returns labeled DataFrames.

Central tendency and spread: mean is sensitive to outliers, median is robust; variance uses n-1 for samples (ddof=1 in NumPy/Pandas), standard deviation is its square root.

Correlation: Pearson r measures linear association (-1 to 1); Spearman uses ranks and captures monotonic relations; correlation does not imply causation.

Normal rule of thumb: about 68% of values within 1 SD, 95% within 2 SD, 99.7% within 3 SD of the mean.

Central limit theorem: sampling distribution of the mean approaches normal with standard error SE = s / sqrt(n) as n grows, enabling t and z inference.

Confidence interval for a mean: x_bar +/- t_(alpha/2, df=n-1) * s / sqrt(n); a 95% CI does not mean 95% probability the parameter lies inside this specific interval.

Hypothesis testing: p-value is the probability of data at least as extreme given H0 true; reject H0 when p < alpha; Type I error = false positive (alpha), Type II = false negative (beta).

Test selection: two-group means -> t-test (Welch when variances differ); paired data -> paired t-test; categorical association -> chi-square; three-plus group means -> ANOVA.

Linear regression model: y = b0 + b1x1 + ... + bnxn + e; coefficients minimize the sum of squared residuals (ordinary least squares).

Train/test split: train_test_split(X, y, test_size=0.2, random_state=42); fit preprocessing on training data only, then transform test data to prevent leakage.

Metrics: MAE = mean(|y - y_hat|); MSE = mean((y - y_hat)^2); RMSE = sqrt(MSE) in target units; R^2 = 1 - SS_res/SS_tot is the share of variance explained.

Cross-validation: KFold or cross_val_score(model, X, y, cv=5) gives a more stable estimate than a single split; use TimeSeriesSplit for temporal data.

Overfitting: training error much lower than validation error signals high variance; remedies are more data, regularization (Ridge L2, Lasso L1), or simpler models.

Assumptions of OLS: linearity, independent errors, homoscedasticity, normally distributed residuals, and low multicollinearity (check VIF).

Classification metrics: precision = TP/(TP+FP), recall = TP/(TP+FN), F1 is their harmonic mean; accuracy misleads on imbalanced classes.

Chart choice: line for trends over time, bar for comparing categories, histogram for one distribution, scatter for two-variable relationship, box plot for spread and outliers by group.

Matplotlib object API: fig, ax = plt.subplots(); ax.plot(x, y); ax.set_xlabel/set_ylabel/set_title; fig.savefig('out.png', dpi=300, bbox_inches='tight').

Seaborn figure-level vs axes-level: sns.relplot/sns.catplot return FacetGrid; sns.scatterplot/sns.boxplot accept ax= for composition in subplots.

sns.heatmap(df.corr(), annot=True, cmap='vlag', vmin=-1, vmax=1) needs a fixed symmetric scale for diverging correlation colors to be comparable.

Perceptual rules: bar length encodes magnitude accurately; avoid truncated y-axes on bars; pie charts fail beyond about 3 slices; colorblind-safe palettes ('viridis') are preferred.

Labeling: every axis needs units, titles state the finding not the variable name, and annotation of key points beats decorative elements.

Class design: init initializes instance state, self binds attributes, class attributes are shared across instances, and @property exposes computed values with validation.

Dunder methods: repr gives an unambiguous developer string, str a readable one, eq defines equality; defining eq without hash makes instances unhashable.

Inheritance: subclass extends a base class and calls super().init(...) to reuse initialization; prefer composition when the relationship is 'has-a'.

Exceptions: try/except/else/finally; catch the narrowest type (except ValueError) first, use raise ... from err to chain causes, and never use a bare except.

Custom exceptions subclass Exception and encode domain errors (e.g., ValidationError); EAFP ('easier to ask forgiveness') suits Python, with context managers for cleanup.

Ethics: collect only the minimum personal data needed (data minimization), anonymize or pseudonymize identifiers, and obtain informed consent with a stated purpose.

Governance: GDPR grants access, rectification, erasure, and portability rights; PII such as names and emails must not appear in logs, notebooks, or version control.

Bias and leakage: models trained on historically biased data reproduce it; report limitations and never use protected attributes as proxies without justification.