MemoryError on a 2.1-Million-Row Sensor CSV
You inherit a script that reads a 2.1-million-row, 14-column CSV of sensor readings (about 1.8 GB) on an 8 GB laptop. It opens the file with open(), uses csv.reader, appends every row to a list of dicts, then makes four separate passes over the list to compute min/max, group counts, and a daily average. It also calls datetime.strptime(row['date'], '%Y-%m-%d') without checking blanks; the file has 0.2% blank dates. The script crashes after about 6 minutes with MemoryError. When you remove the date parsing, it finishes, but results are off by one day for some rows because some dates are in DD/MM/YYYY. The stakeholder needs daily averages by sensor tomorrow. There is no database available, and you must use only the Python standard library plus NumPy if needed.
1. Diagnose the likely causes of the MemoryError and the date errors, including why removing date parsing changed the runtime but not the correctness.
Read the worked answer
The MemoryError is likely caused by materializing 2.1 million rows as a list of Python dicts. Each dict has substantial per-object overhead, so 14 columns of strings can consume several gigabytes before the four passes even begin. The four passes add time but not the primary memory spike. The date errors have two causes: 0.2% blank dates raise ValueError in datetime.strptime, and some dates are DD/MM/YYYY while the format string expects YYYY-MM-DD. Removing date parsing avoids the exception and speeds the run, but it leaves dates as unvalidated strings; rows like 03/04/2025 are then misinterpreted or grouped incorrectly, producing off-by-one-day daily averages. The fix must address both memory and parsing, not just remove the failing line.
2. Describe a refactor using csv, datetime, collections, and functions that fixes memory and date issues. You may sketch code.
Read the worked answer
Stream the file once with csv.DictReader and aggregate into defaultdict accumulators instead of storing rows. Write a parse_date function that tries YYYY-MM-DD first, then DD/MM/YYYY, returns None for blanks, and counts failures. In the loop, update per-sensor count, sum, min, and max, and per-sensor-day sum and count. Sketch: from collections import defaultdict; from datetime import datetime; import csv then def parse_date(s): ... and a single for row in csv.DictReader(f): d = parse_date(row['date']); if d is None: missing += 1; continue; key=(row['sensor_id'], d.date()); daily_sum[key]+=float(row['value']); daily_count[key]+=1. This keeps memory proportional to the number of sensors and days, not rows. For daily averages, divide sum by count at the end. Use math.isfinite for values and log rejected rows.
3. What validation and communication steps would you take before sending the daily averages?
Read the worked answer
Validate row accounting: total CSV rows equals accepted rows plus missing-date rows plus invalid-value rows, with counts reported. Check parse failures by format and by sensor; if one sensor has many DD/MM/YYYY dates, call it out. Recompute one sensor-day by hand or with a small independent filter and compare. Check for duplicate sensor-date rows and decide whether to average duplicates or sum them, and state the rule. Before sending, report the 0.2% blank-date rate, the date formats encountered, and whether any sensor-day has a suspiciously low count. In the message to the stakeholder, state that daily averages exclude blank-date rows and describe the parsing rule; offer a sensitivity check if blanks are concentrated in one sensor. This makes the remaining uncertainty visible rather than hiding it in the final table.