A Mathematical Introduction to Markov Chains

by Martin Day

From the Preface:

I have taught various undergraduate courses on Markov chains and stochastic processes at Virginia Tech over the years.  In every instance I used a different published text and was always disappointed in their emphases and coverage.  So I began to write short sections of notes to bring out the ideas and organize the material the way I thought appropriate.  After my retirement in 2016 I decided to put those notes together, fill in some of the gaps, and try to turn them into a coherent treatment.  This document is the current status of that effort. 

So far I have organized the material in an order that makes logical sense to me and developed proofs based on that organization.   Here are some of the ideas that have guided my selection and organization of the material.

But now some candid acknowledgment of what this document is not.  I hesitate to call it a "book" for several reasons.  There are additional topics which ought to be included ... and some chapters which need to be expanded and supplied with more problems.  Many ideas could use better introduction. The use of Matlab is rather thin in the latter chapters.  The difficulty level is uneven and probably strays beyond what most undergraduates can handle in places.  Surely there are inconsistencies in my notation.  There are also certain to be many typos, misspellings, even mathematically incorrect statements (though I hope not many) which further editing would improve.   Whether I will eventually improve all these things only time will tell.  I hope that what I have written out so far at least provides an organization and development of the material that others may find useful in some way, even if it is not quite suitable for a course text (yet).

Prerequisites, in addition to the standard freshman-sophomore calculus and differential equations courses, would be a real analysis or advanced calculus course which covers the connections between continuity and sequential convergence, the properties of infinite series and power series, a linear algebra course which includes the study of diagonalization of matricies and eigenvalues, and (for Chapter 11) a course on differential equations which includes the matrix exponential.  Although not strictly necessary it would also be helpful if the student has encountered some basic ideas about random variables previously.  An appendix provides a brief synopsis of some supporting  mathematical topics that may have escaped students' backgrounds...

You are welcome to download the document and associated Matlab files from the following links.  I just ask that you do not redistribute or post it yourself, but refer anyone else who might want to use it to this page.

Download A Mathematical Introduction to Markov Chains: click here for the most recent version (dated on the title page). 

Download Matlab m-filesclick here for the directory and then download the specific ones you want.