MS&E 221

Ramesh Johari

Problem Set 2
Due: February 4, 2015, 5:00 PM, in the basement of Huang Eng. Ctr.

Reading. Read Sections 4.1-4.4, 4.5.1, 4.7 in Ross.

Problem 1. (Rewards) Suppose X0 , X1 , . . . is a Markov chain on the state space {1, 2, . . . , N }.
Suppose also that each time state i is visited, a reward r(i) is earned.
Let Rk denote the total reward earned up to time k, i.e., Rk = r(X0 ) + r(X1 ) + + r(Xk ).
Let A be a subset of the states, and let TA be the hitting time of A (as defined in the lecture
notes); this is the first time the chain enters the set A.
Note that RTA is the total reward earned up to and including time TA . If TA = , we define
RTA = limk Rk .
(a) Explain how you would calculate E[RTA |X0 = i].
(b) Suppose r(j) = 1 if j A, and r(j) = 0 if j 6 A. In words, what is E[RTA |X0 = i]?
(c) Suppose r(j) = 1 for all j. In words, what is E[RTA |X0 = i]?
Problem 2. (Adapted from Ross.) Let P and Q denote transition probability matrices on a
finite state space X = {1, . . . , N }, for Markov chains that are both aperiodic and irreducible. Let
and denote the unique invariant distributions corresponding to P and Q, respectively.
(a) Define a process as follows. X0 = 1. A coin is then flipped, and if it is heads, the remaining
states X1 , X2 , . . . are obtained via the matrix P ; and if tails, they are obtained from Q. Is {Xn , n
0} a Markov chain? If so, determine its transition probabilities. If p = P(coin comes up heads),
what is limn P(Xn = i)?
(b) Define a process as follows. X0 = 1. At each subsequent time step, a coin is flipped; if it is
heads the next state is obtained via the matrix P , and if it is tails the next state is obtained via the
matrix Q. In this case is {Xn , n 0} a Markov chain? If so, determine its transition probabilities.
Show by an example that limn P(Xn = i) need not be the same as the values you obtained in
part (a).
Problem 3. (A queueing model) Consider a queue (or a waiting room) that can hold at most N
customers at a time, including the customer currently being served. At each time step, one of three
things happens:
A new customer arrives with probability p. If the queue is already full, then this new customer is blocked, i.e., turned away.
Whenever the queue is nonempty, with probability q a customer completes service and leaves
the queue.
With the remaining probability (1 p if no one is present in the queue, and 1 p q if the
queue is nonempty), nothing happens to the state of the queue.
1

(a). Draw the graph representing the Markov chain describing the number of customers in the
queue (with states 0, 1, . . . , N ), and label the probability of each transition.
(b). Is the chain irreducible? Which states are recurrent?
(c). Suppose the queue starts in state 1. Write down a set of equations that yield the probaiblity
that exactly three customers are blocked before the queue empties. (Hints: Note that a customer is
blocked if and only if the queue is in state N , and a new arrival occurs. Also, by the Strong Markov
Property, conditional on one customer just having been blocked, the process behaves like a queue
starting with N customers.)
(d). Explain why there is a unique invariant distribution , and write down a set of equations
you can use to answer the following question: Assuming the states are distributed according to the
invariant distribution , what is the average number of customers in the queue? (You dont have to
solve the equations.)
Note: Next week, you will learn that this is also the long run average number of customers in
the queue.
Problem 4. (Service staffing) Consider the following model of a manager handling temp
staffing in a service organization.
Suppose that if a worker comes in to work today, he comes in to work tomorrow as well with
probability p, and otherwise shirksi.e., skips workwith probability 1 p. If the worker
shirks today, he receives a phone call from the manager, and this induces him to return to work the
next day with probability r, or to shirk the next day with probability 1 r. Finally, if the worker
shirks D days in a row, the manager gets fed up and replaces the worker with a new worker the
next day.
(a) Give a Markov chain description of the position from the point of view of the manager, with
states W (worker is working) and S1 , . . . , SD (where Si represents the worker having shirked for i
consecutive days). You may assume that a new worker always starts in state W.
(b) Show the chain has a unique invariant distribution , and compute it. Hint: You might find
the following identity useful:
1 + x + x2 + + xD1 = (1 xD )/(1 x).
Note: the remainder of the problem is OPTIONAL on this weeks problem set; I will be assigning parts (c)-(f) on Problem Set 3. Part (g) will remain optional even on Problem Set 3.
Now suppose that a cost c1 is incurred each day that the worker doesnt come in to work, and
that a replacement cost c2 is incurred if we have to hire a new worker (e.g., this might be the cost
we pay to a temp staffing firm to send us a new worker).
(c) (OPTIONAL this week) Define a function C(i) that gives, as a function of the state i, the
cost incurred in each state.
(d) (OPTIONAL this week) Determine the long run average cost (due to both shirking and
replacements).

(e) (OPTIONAL this week) Show that if c2 > c1 /r, then the average cost optimal decision (i.e.,
the decision which minimizes long run average cost) is to never replace the worker. (Hint: Show
that the long run average cost is decreasing in D.)
(f) (OPTIONAL this week) Show that there is a constant , such that if c2 /c1 > , then it is
optimal never to replace the worker; and if c2 /c1 , then it is optimal to replace the worker on
the first day he shirks.
(g) (OPTIONAL) Suppose that instead of the model described above, we were allowed to
choose each day whether to replace the worker; i.e., we are not required to fix D in advance, but
can make decisions dynamically. Are the policies of part (f) still optimal? Why or why not?
Problem 5. (Adapted from B&T) Consider the Markov Chain in Figure 1. We refer to a
transition that results in a state with a higher index as birth and that to state with a lower index as
death. Calculate the following quantities, assuming that when we start observing the chain, it is
already in steady-state.

Figure 1: Markov Chain for Problem 5

(a) For each state i, the probability that the current state is i.
(b) The probability that the first transition we observe is a birth.
(c) The probability that the first change of state we observe is a birth.
(d) The conditional probability that the process was in state 2 before the first transition that we
observe, given that this transition was a birth.
(e) The expected value and variance of the number of transitions to hit state 1.
Problem 6. (Doubly-Stochastic Matrices) In class, we defined a stochastic matrix P to be
one
P where each element is non-negative and each row sums to 1 (Formally, Pi,j > 0 (i, j), and
We say a matrix P is doubly-stochastic if, in addition, each column of P also
j Pi,j = 1 i). P
sums to 1 (that is, i Pi,j = 1 j).
(a) Show that the uniform distribution is an invariant distribution for any Markov chain with a
doubly-stochastic transition matrix.
3

(b) The above property is useful for designing randomized algorithms to uniformly sample
objects with certain properties. We now demonstrate this for the case of sampling independent sets
from a graph. An undirected graph G(V, E) is a collection of nodes V = {1, 2, . . . , n}, and edges
E = {(i, j)} connecting some pairs of nodes. An independent set of G is a set of nodes I V
such that no two edges in I are connected by an edge; see Figure 2 for examples.

Figure 2: Examples of graphs with their independent sets.

This is useful, for example, in modeling interference in wireless networks. The base stations of
the network are nodes of the graph, and any two stations that cannot both transmit simultaneously
due to interference (say, arising from proximity) are connected using an edge. Therefore, to avoid
interference, the set of base stations It transmitting at time t must be an independent set; and
choosing the independent set uniformly at random gives a simple scheduling strategy that has
reasonable fairness properties.
Consider the following Markov chain over the independent sets It of a graph G(V, E). To
construct It+1 from It , we first sample a node u uniformly from V . If u It , we remove it, so
It+1 = It \ {u}. Otherwise (that is, if u
/ It ) we add u to It only if the independent set property
can be retained; that is, It+1 = It {u} if It+1 is an independent set, else It+1 = It . Prove that
limt It is the uniform distribution over independent sets of G.
Argue that the above Markov chain is irreducible (Hint: given any two independent sets, how
can you change transform one into the other?) and aperiodic (Hint: are there any self-loops in the
graph?). Prove that the associated transition matrix is doubly-stochastic, and finally, using part (a)
argue that in steady state the above algorithm outputs a uniform-random independent set.