The Automated Grading Tool
A coding bootcamp deploys an automated grader that runs student-submitted code in containers and scores it against hidden tests. Submissions run as root inside the container, which has network access and mounts a shared volume for reading test files. The grader returns a pass/fail plus the first failing test's input and expected output. A student reports that a submission 'accidentally' read the expected outputs from the shared volume and hard-coded them, passing all tests. Another student's submission ran for 20 minutes and consumed 8 GB of memory before being killed manually. The bootcamp wants to keep the tool fast and cheap, and asks whether to keep showing the first failing test's details.
1. Identify the security and integrity risks and propose concrete controls.
Read the worked answer
Risks:
- Test-data exposure: mounting expected outputs on a volume the submission can read lets students hard-code answers. The grader's integrity is broken -- a pass no longer means the code works.
- Privilege: running as root inside the container increases the impact of any escape or misconfiguration. Root can often modify mounted files or reach host resources if isolation is weak.
- Network access: submissions can exfiltrate test data, call external services, or attack other systems. It is unnecessary for grading.
- Resource exhaustion: no enforced CPU/memory/time limits let one submission starve others or the host. The 20-minute, 8 GB run shows limits are missing or not enforced.
- Information leakage via feedback: returning the first failing test's input and expected output hands students the hidden tests one at a time.
Controls:
- Run as a non-privileged user with a read-only root filesystem; drop capabilities.
- Remove network access from the grading container.
- Do not mount expected outputs where the submission can read them. Run the submission, capture its output, and compare outside the sandbox -- the test harness, not the student code, should hold the answers.
- Enforce hard limits: wall-clock timeout (e.g. 10 seconds), CPU quota, memory cap, process count, and output size. Kill and mark as failed on breach.
- Isolate each run (fresh container, no shared writable volume) and log resource use.
- Use a separate, minimal image per language and pin versions.
These are mostly configuration changes, so cost stays low.
2. Should the grader keep showing the first failing test's input and expected output? Argue both sides and recommend a policy.
Read the worked answer
Case for showing details: fast feedback helps students learn. Seeing the exact input and expected output lets them debug without guessing, reduces frustration, and lowers support load. For a bootcamp, learning speed matters and the tests are not high-stakes credentials in the way an exam is.
Case against: it leaks the hidden tests. A student can submit repeatedly, collect failing cases one by one, and hard-code or overfit to them -- exactly what happened with the shared-volume exploit. It also makes the score less meaningful as evidence of skill, and repeated submissions can be used to reconstruct the full test set.
Recommendation: tiered feedback.
- Show the failing test's input and expected output only for a limited number of attempts per problem (e.g. the first 2), then switch to a generic message ('a test failed; check edge cases').
- Or show a category, not the exact case: 'failed on empty input' or 'failed on a large input' -- enough to guide debugging without revealing the case.
- Keep full details visible to instructors.
- Rate-limit submissions and log them so reconstruction attempts are detectable.
- Rotate or hold back a portion of tests that are never shown, so hard-coding to visible cases still fails.
This preserves the learning benefit while protecting integrity. The shared-volume exploit must be fixed regardless; feedback policy is a second layer, not a substitute for sandboxing.