Simulation Output Analysis and a Paired Design Comparison
A simulation study evaluates a proposed automated storage and retrieval system. One analyst runs a single 300-day replication and reports mean daily throughput 38.9 pallets, compared with a target of 40. A second analyst runs 10 independent 30-day replications, discarding the first 20 days of each, and obtains a mean of 42.3 pallets with sample standard deviation 5.1. The first analyst claims the system fails because 38.9<40; the second claims it passes because 42.3>40. The steering committee also wants to compare two design variants. The second analyst used common random numbers and paired differences of 1.8 pallets/day with sample standard deviation 2.4 across the same 10 replications.
1. Construct a 95% confidence interval for mean daily throughput from the 10 replications and test whether the system meets the 40-pallet target. What should the committee conclude?
Read the worked answer
With n=10, df=9, the 95% t critical value is 2.262. Standard error = 5.1/√10 = 1.613.
95% CI = 42.3 ± 2.262×1.613 = 42.3 ± 3.65 = [38.65, 45.95] pallets/day.
For a one-sided test of H0: μ≤40 versus H1: μ>40, t = (42.3−40)/1.613 = 1.43. The one-sided 5% critical value is 1.833, so we fail to reject H0. The data are consistent with a true mean at or below 40. The CI includes 40 and extends well below it.
The committee should not conclude that the system 'passes.' The point estimate is above 40, but the uncertainty is too large. The single 300-day run of 38.9 is also not a valid basis for a precise conclusion: it is one observation with correlated daily output, no replication-based standard error, and possible initialization bias. Neither analyst has demonstrated compliance. More replications or a longer run with proper output analysis are needed.
2. How many replications would be needed to estimate mean throughput within a half-width of 1.5 pallets at 95% confidence, using the pilot standard deviation? Discuss the iterative nature of the calculation.
Read the worked answer
Use the half-width formula h = t_(n−1,0.975) × s/√n. With pilot s=5.1 and an initial t≈2.262:
n ≈ (t s / h)² = (2.262×5.1/1.5)² = (7.69)² ≈ 59.1.
So about 60 replications are needed. This is an approximation because t depends on the final n. After running more replications, recompute s and t and check the achieved half-width; if it is still too wide, increase n. Using the normal approximation z=1.96 gives about 44, but that is optimistic for a modest sample and ignores the t correction. Also, the pilot s=5.1 may not be reliable from only 10 replications; a conservative planning value or sequential procedure is preferable. Report the final CI, not just the target half-width. If the system is near the 40-pallet target, this sample size is necessary to distinguish compliance from failure.
3. Critique the use of a single 300-day run and of common random numbers for comparing the two designs. What should be reported?
Read the worked answer
A single long run produces correlated observations, not 300 independent samples. It gives one replicate, so there is no replication-based estimate of variability. Initialization bias may contaminate the early output, and a 300-day run may still be too short if the system has long cycles. Discarding the first 20 days of each 30-day replication is also risky: only 10 days remain, and the warm-up period may not be long enough. Better practice is to use a pilot run to choose a warm-up period, use replication-deletion or batch means, and report a confidence interval from multiple independent replications.
Common random numbers can reduce variance when comparing designs, but only if the same random-number streams are synchronized across designs, the same warm-up and run length are used, and the designs are compared run by run. The paired difference is 1.8 with s_d=2.4, n=10, so the 95% paired CI is:
1.8 ± 2.262×2.4/√10 = 1.8 ± 1.72 = [0.08, 3.52].
This suggests design B is better on average, but the lower limit is close to zero. Report the paired CI, the number of replications, the warm-up method, and whether CRN synchronization was verified. If CRN was not properly synchronized, the paired interval is invalid and an unpaired, more conservative analysis is needed.