Markov chain

A Markov chain is specified by a number of states $N$ and a transition probability matrix $P \in M_{N \times N}(\mathbb{R})$. Intuitively think of this as a state machine which at state $i$ has probability $p_{i,j}$ of transitioning to state $j$.