A Defensible Model Quality Score Before a Design Review
Orion Autonomy has a 12,000-element SysML model for an autonomous tractor. Before a design review, the chief engineer asks for a 'model quality score'. Known facts: 38% of blocks have no «Rationale»; 210 ports are unconnected; 74 «satisfy» links point to requirements that no longer exist in the RM tool; 90% of activity diagrams have no «Allocate» to a block; naming convention compliance is 62%; the model has 1,400 «Comment» elements but no «View» or «Viewpoint». Two engineers have 5 days. A tool vendor offers a 'health score' that counts diagrams and elements. The chief engineer wants a single number. The review gate requires a quality assessment with evidence, not just a dashboard. The model is a delivered contract item, so defects have cost. One engineer is new to the project.
1. Critique the vendor's element-count health score. Define a quality framework that two engineers can apply in five days and that the chief engineer can defend.
Read the worked answer
The vendor's score counts diagrams and elements, which measures size and activity, not quality. It can be increased by adding comments or empty diagrams, and it says nothing about whether a «satisfy» link is true. A defensible framework uses a small set of quality attributes: traceability, consistency, completeness, understandability, and conformance to modeling standards. Each attribute gets one or two measurable indicators drawn from the known facts: dangling satisfy links (74), unconnected ports (210), naming compliance (62%), rationale coverage (38% missing), and view/viewpoint coverage (zero). Weight traceability and consistency highest because false traceability is an audit and safety risk. Publish the formula and show the raw counts, not just the composite. With five days: two days to implement queries and run them, one day to manually sample 20 elements for semantic correctness, one day to triage, and one day to write the report. The single number is acceptable only as an index with drill-down and a stated confidence based on the sample.
2. Prioritize the findings for the design review. Which findings are defects, which are debt, and what should be fixed before the gate?
Read the worked answer
The 74 dangling «satisfy» links are defects: they assert traceability to requirements that no longer exist. Quarantine or remove them before the review, and list the affected blocks as having unverified requirements. The 210 unconnected ports are ambiguous: some may be intentionally unconnected, others incomplete. Triage them by sampling and by checking against interfaces; do not report them all as defects without evidence. Missing «Allocate» on 90% of activity diagrams is a methodology debt and a gap in the model's ability to answer allocation questions, but it is not a false statement. Missing rationale and 62% naming compliance are understandability debt; they slow review but do not invalidate traceability. In five days, fix the dangling links, triage the ports, and produce evidence for the review. Do not attempt to write rationale for 38% of blocks or rename the whole model. State explicitly what was not assessed, so the chief engineer does not read the score as complete coverage.
3. How would you keep the score from being gamed and make it repeatable next quarter?
Read the worked answer
Make the metrics reproducible by storing the queries in the model or in the CM tool, versioning them with the model baseline, and running them against a fixed baseline. Require evidence for each indicator, such as the list of dangling link IDs, not just a percentage. Use a random sample for semantic checks and record the sample seed and size. Prevent gaming by forbidding teams from editing the query without review, by including negative indicators that cannot be improved by adding elements, and by comparing trends across baselines rather than rewarding a single absolute number. Calibrate the thresholds against a reference model or a previously audited baseline. Separate measured indicators from judged indicators, and have an independent reviewer sign off on the sample. Next quarter, rerun the same query set on the next baseline and report the delta; if the query or tool version changes, publish a mapping so the trend remains interpretable. The score should be a decision aid, not a target.