Poison Messages and a Serialized Singleton in a Payments Back Office

Why it loops. The MDB's onMessage runs in a container-managed JTA transaction (default REQUIRED) that enlists the JMS session. Throwing a RuntimeException is a system exception: the container marks the transaction for rollback, the JMS message is not acknowledged, and the provider makes it available again. Redelivery is immediate, so the same poison message returns, fails, and rolls back - forever, unless a redelivery limit and a dead-letter queue are configured. The 60,000-message backlog is mostly the same handful of messages cycling.

The classification error. A bank business rejection is a normal outcome, not an infrastructure failure. Wrapping it in RuntimeException erases that distinction and hands control to the retry machinery.

Design.

1. Do not throw for business rejections. Catch BankRejectionException, write the ledger row in state REJECTED, and send a message to PAYMENTS.REVIEW. Return normally from onMessage; the transaction commits and the message is acknowledged exactly once.
2. If you must throw, annotate the exception @ApplicationException(rollback = false) so it does not roll back - but returning normally is clearer, and the acknowledgement semantics are easier to reason about.
3. Configure the queue with a redelivery limit (for example 3) and a dead-letter queue, so genuinely transient failures (bank timeout) do not loop forever either. Log JMSXDeliveryCount.
4. Make the consumer idempotent, because JMS is at-least-once: store the message's JMSMessageID (or a business key such as settlement id) in a dedupe table with a unique constraint inside the same transaction as the ledger writes. A duplicate delivery then fails the insert and is discarded, instead of double-posting.

Verification. Replay a rejected message ten times and assert one review record and zero duplicate ledger rows.

The problem. SettlementService.post() is REQUIRED, so it joins the MDB's transaction. The ledger INSERT/UPDATE takes row locks, then the code blocks two seconds on the SOAP call before commit. With 60,000 messages and a modest MDB pool, locks are held for seconds, other transactions queue, and the connection pool is drained - the database becomes the bottleneck even though the slow part is the network.

Option A - call the bank first, then transact. Validate, call the bank outside any transaction, then run post() in a new transaction. Locks are held for milliseconds. Trade-off: if the process crashes between the bank call and the commit, the bank has acted and the ledger has not - you need a reconciliation path keyed by an idempotency reference sent to the bank.

Option B - two-phase within the application. First transaction writes the ledger row as PENDING and commits. Then call the bank (no transaction open). Then a second transaction, marked REQUIRES_NEW or started after the call, flips the row to SETTLED or REJECTED. This is an application-level saga; crashes leave PENDING rows that a sweeper resolves by querying the bank. This is the usual production answer because it is observable and recoverable.

Option C - NOT_SUPPORTED on a separate bean. A @Stateless bean with @TransactionAttribute(NOT_SUPPORTED) can wrap the remote call so no transaction is suspended or held. Simpler than B but loses the atomic boundary; crash recovery still relies on reconciliation.

What not to do. Do not use REQUIRES_NEW for the whole post() and then do the remote call inside it - that just creates a second transaction holding locks for two seconds. Also do not increase the MDB pool or the datasource size to compensate; that increases lock contention rather than reducing it. Note that the JMS session is in the same transaction: if the design in Option B commits an intermediate transaction, the message acknowledgement still happens only when onMessage returns, so replay is still possible and idempotency remains mandatory.

Cause: the default lock is exclusive. A @Singleton session bean uses container-managed concurrency by default, and the default lock type at class level is @Lock(LockType.WRITE). A WRITE lock is exclusive: the container serializes all business method calls, including the read-only getRate(). With 500 concurrent callers each taking a few milliseconds, requests queue behind one another and the observed time climbs to seconds - the bean has become a single-threaded bottleneck, not because the lookup is slow, but because it waits for the lock.

Fix.

@Singleton
@Lock(LockType.READ)          // class default: shared
public class PricingCache {
    private final Map<String, BigDecimal> rates = buildRates();

    public BigDecimal getRate(String code) { ... }   // READ lock, concurrent

    @Lock(LockType.WRITE)
    public void reload() { ... }                     // exclusive, rare
}

Optionally add @AccessTimeout(value = 500, unit = TimeUnit.MILLISECONDS) so a caller waiting on the exclusive reload fails fast instead of blocking the request thread.

Memory-safety argument. A READ lock permits concurrent execution, so every thread touches the map simultaneously. That is safe only if the map is never mutated after publication. Publish it through a final field assigned exactly once (in the field initializer or @PostConstruct, which the container calls before any business method), which gives the JVM's final-field semantics and makes the contents visible to all threads. A plain HashMap that is only read is fine under those conditions; if reloads mutate it, swap the reference (volatile Map replaced atomically) or use a ConcurrentHashMap, and do the swap under the WRITE lock.

Also check. @Startup does not change locking; @Asynchronous would return Future and is not appropriate for a cache read. Measure after the change with the same 500-caller test: the target is latency roughly flat in the number of callers.