Markov Chains and Markov Jump Processes
This domain covers state-based models of movement between conditions - healthy, sick, dead; or claim-free, one claim, two claims - in both discrete and continuous time.
Discrete-time Markov chains
- The Markov property: the future depends on the past only through the present state.
- The transition matrix P has non-negative entries and rows summing to 1. The n-step transition matrix is P^n, which is the Chapman-Kolmogorov relation written in matrix form.
- Classification: communicating classes, irreducibility, recurrence versus transience, and periodicity.
- A stationary distribution satisfies pi = pi P with entries summing to 1. For a finite, irreducible, aperiodic chain it exists, is unique, and is the limiting distribution.
- Expected hitting times solve m_i = 1 + sum over j of p_ij m_j.
Markov jump processes
- The generator Q has off-diagonal entries q_ij, the transition intensities, and diagonal entries q_ii = minus the sum of the other entries, so each row sums to zero.
- Time spent in state i is exponential with rate -q_ii; the probability that the next state is j is q_ij / (-q_ii), the jump chain.
- Kolmogorov forward equations: P'(t) = P(t)Q with P(0) = I. They are solved by matrix eigenvalues or numerically. If intensities depend on age or duration the process is time-inhomogeneous and needs piecewise or numerical methods.
Worked example
A three-state model has intensities Healthy to Sick 0.1, Healthy to Dead 0.005, Sick to Healthy 0.3 and Sick to Dead 0.02. From Healthy the total rate out is 0.105, so the expected holding time is 1/0.105 = 9.52 years, and the probability the first jump is to Sick is 0.1/0.105 = 0.952. From Sick the total rate out is 0.32, giving a mean holding time of 3.125 years.
Traps that cost marks
- Confusing transition probabilities with intensities; they have different units and only coincide for very short intervals.
- Forgetting that generator rows sum to zero.
- Using P^n for a continuous-time process, or assuming stationarity where intensities vary with age.