Code generator at TCL2 and a cosmetic traceability claim
You are the test manager at a Tier-1 supplier for a lane-keep assist ECU rated ASIL B. The customer requires Automotive SPICE® Capability Level 2 at the upcoming assessment in six weeks. The current metrics are: statement coverage 82 % across all software units, no MC/DC measurement, requirements-to-test traceability at 100 % but 14 safety requirements have no verification criteria defined, and the code generator used to produce 40 % of the base software is a custom Python script with no qualification record. The safety plan states tool confidence level 2 for that generator. Two integration anomalies in the last month were traced to generated code that no unit test ever executed. Management proposes to raise statement coverage to 90 % and to close the assessment as compliant. Your assessment budget allows one focused activity before the audit.
1. Management wants to spend the remaining budget on lifting statement coverage from 82 % to 90 % and to declare Capability Level 2. Which single activity would you fund instead, and how do you justify it with the evidence in the setup?
Read the worked answer
Rank the three gaps by how much evidence each one invalidates.
- Statement coverage 82 %: incomplete, but for ASIL B the structural criterion at unit level is statement/branch coverage, and a few extra percent on already-executed code buys almost nothing.
- 14 safety requirements without verification criteria: cheap to fix but fatal at assessment, because traceability without verification criteria is not consistency between the requirement and test work products.
- TCL2 custom generator, no qualification, two escapes in generated code: this is the dominant risk. It is evidence-based, not theoretical: the anomalies prove the gap produces real defects, and it contaminates the test evidence for 40 % of the base software.
So fund a generator-focused activity: determine tool impact and tool error detection, determine TCL, and derive the qualification or verification measures for the constructs actually used. Fund it because it is the only gap that invalidates other evidence and already has escaped defects behind it. Close the 14 verification criteria in the same week as a low-cost mandatory item; do not trade it away.
2. The customer assessor asks for evidence that code produced by the custom generator is adequately verified. Name three concrete artefacts and state what each one proves.
Read the worked answer
- Tool qualification record: tool impact and tool error detection determination, resulting TCL, and the qualification method chosen (validation suite covering the language constructs and generator options actually used, or an increased-confidence-from-use argument with field data). This proves the tool's failure modes are known and controlled, not that the output is correct.
- Back-to-back comparison of generated code against a reference: the same model or specification executed against the generated code and a reference implementation, with the test cases traced to the model elements. This proves the translation is correct for the constructs exercised, which is the part the qualification record cannot prove.
- Structural coverage report scoped to generated units, plus the generator configuration under version control: proves the delivered generated code was actually executed and that the options in force match the qualified configuration. Add a list of known tool anomalies with impact analysis; assessors look for that list because an empty one usually means nobody looked.
3. Two anomalies reached integration from generated code that no unit test executed. What does that reveal about the existing test strategy, and what do you change for the next release?
Read the worked answer
The coverage figure was measured over hand-written units only, or generated units were excluded from the measurement without a recorded rationale. Either way the reported 82 % describes a subset of the product while management reads it as the whole. That is a measurement-definition defect, not a testing defect.
Changes for the next release:
- Include generated units in the coverage measurement and report hand-written and generated coverage separately.
- Add a per-unit coverage gate in the build that fails on regression, rather than a project-level average that hides new uncovered units.
- Baseline the generator options together with the generated files, so a configuration change is visible as a change.
- Add a review of generated code for ASIL B units (checklist or tool-supported), since structural coverage cannot show that unexecuted defensive code is correct.
- Trace generator configuration to generated file to test, so the next anomaly can be localised in minutes.