Cache Coherence Bug in a Dual-Core Firmware Update
A dual-core ARM Cortex-A53 SoC runs Linux on core 0 and a bare-metal real-time task on core 1. The two cores share a 512 KB SRAM region used as a message ring buffer. Core 1 writes a 64-byte status block and then sets a flag word to 1. Core 0 polls the flag and, when it sees 1, reads the status block. Intermittently, core 0 reads the flag as 1 but sees stale status data. The compiler is GCC with -O2. The flag and the status block are declared as separate volatile variables. The SRAM is mapped as cacheable write-back on core 1 and non-cacheable on core 0. A logic-analyser trace shows core 1 writing the status block and the flag in the expected order on the bus, but core 0's reads sometimes return old data. The system must guarantee the flag is never observed before the data is visible.
1. Diagnose the failure and explain why volatile is not sufficient.
Read the worked answer
The failure is a memory-ordering and cache-coherence problem, not a compiler problem in the narrow sense. Core 1's SRAM is cacheable write-back, so the status block writes may sit in core 1's L1/L2 cache and not reach the shared SRAM when the flag is written. The flag write may be visible (if the flag line happened to be evicted or is in a different state) before the data writes are flushed. Core 0 sees flag = 1 but reads stale data from SRAM. volatile tells the compiler not to optimise away the accesses, but it does not order memory operations on weakly ordered architectures and does not flush caches. ARM's weak memory model allows the flag store to be observed before the data stores. The correct fix is a release barrier (DMB ISH or DSB followed by cache clean) on core 1 before setting the flag, and an acquire barrier (DMB ISH) on core 0 after reading the flag. In addition, core 1 must clean the data cache lines for the status block before the flag store, and core 0 must invalidate its cache lines for the status block before reading. The cleanest solution is to map the shared region as non-cacheable or device memory on both cores, at the cost of performance.
2. The team proposes making the flag volatile and adding a delay loop of 10 microseconds on core 0 after seeing the flag. Evaluate this.
Read the worked answer
A delay loop only reduces the probability of the race; it does not eliminate it. The stale data could persist indefinitely if core 1's cache line is never evicted, so a 10 microsecond delay may work in the lab and fail in the field. Timing-based fixes are not acceptable for a correctness guarantee. The delay also wastes real-time budget and is not portable. The correct approach is a proper release/acquire protocol with cache maintenance and barriers. If the region is made non-cacheable, no delay is needed at all. The team should also consider using hardware mailboxes or a peripheral with coherent access, or using the SoC's hardware semaphores. Verification should include a stress test with cache-thrashing on core 1 to force evictions, run for many hours, and a formal or model-checking argument that the protocol is correct.
3. Describe how you would verify the fix and what residual risks remain.
Read the worked answer
Verification: (1) unit-level stress test where core 1 writes a known pattern, cleans caches, sets the flag, and core 0 reads and checks; run millions of iterations with varying cache pressure (e.g., core 1 also touches a large array to force evictions). (2) Use the logic analyser to confirm the cache clean operations occur before the flag store. (3) Run a memory-model litmus test (store-buffering and message-passing patterns) to confirm the barriers are effective. (4) Test at temperature extremes and with different cache configurations. Residual risks: the compiler may reorder non-volatile accesses around the barriers if the barrier is not a compiler barrier too (use asm volatile with memory clobber or C11 atomics). The cache maintenance operations must cover the correct line size and alignment. If the flag and data share a cache line, false sharing can cause performance problems and, worse, the clean of one may not cover the other. Document the protocol and require any change to the shared region to be reviewed against it.