DSMA Admission: Programming and Data Manipulation

This domain tests whether you can transform data reproducibly and detect the silent bugs that leave row counts or dtypes wrong without raising an error.

Indexing and mutation

A pandas DataFrame has labelled columns and an index. .loc selects by label and is inclusive of the endpoint; .iloc selects by position and is exclusive. Chained assignment such as df[df.a > 0]['b'] = 1 may write to a temporary copy; use df.loc[df.a > 0, 'b'] = 1 instead. Beware inplace=True, which returns None and silently reassigns a variable to None.

Reshaping and joining

• merge takes how=inner/left/right/outer and validate='one_to_one'/'one_to_many'/'many_to_one', which raises when key duplication breaks your assumption.
• pivot_table goes long to wide; melt goes wide to long. groupby().agg() replaces manual loops.
• apply runs Python per row or element; vectorised operations and built-in string or datetime methods are far faster on large frames.

Types and reproducibility

An integer column turns float the moment a NaN enters; use nullable Int64 or an explicit fill. Parse dates with an explicit format and utc=True to avoid timezone-naive subtraction. Pin dependency versions, set random seeds, wrap logic in functions with asserts, and keep notebooks as thin layers over tested modules.

Worked example

Orders has 1,000 rows, customers 100 unique rows. orders.merge(customers, on='customer_id', how='left', validate='many_to_one') must return 1,000 rows. If duplicate customer ids exist, the same merge without validate returns 1,100 and revenue is double counted. Check len(after) == len(before) and customers.customer_id.is_unique before trusting any downstream total.

Traps

Ignoring SettingWithCopyWarning; mutating a DataFrame while iterating over it; comparing floats for equality; and trusting a join without checking row count and key uniqueness.