Markov Machines: CTMC Models & Control

• Markov Machines are computing resources modeled by CTMCs with binary and three-state formulations for dynamic resource allocation.
• They enable optimal job submission decisions by incorporating the age and staleness of state estimates into threshold or switching control policies.
• Advanced models use water-filling techniques for sampling allocation across multiple machines to balance monitoring quality and decision errors.

Searching arXiv for papers specifically on "Markov Machines" in the resource-allocation/monitoring sense. In the 2025 literature, a Markov Machine (MM) denotes a computing resource whose availability evolves stochastically as a continuous-time Markov process and whose sampled state is used for control decisions such as job submission or job assignment. The two principal formulations currently associated with the term are a binary free/busy model for utility maximization and a three-state model for joint monitoring and dispatch, both of which place partial observability at the center of the problem: the controller observes the machine only through samples or queries, maintains an estimate $\hat X(t)$, and acts under stale information rather than full state visibility (Liyanaarachchi et al., 30 Jul 2025, Liyanaarachchi et al., 30 Jan 2025).

1. Canonical stochastic models

The term has been used for two closely related CTMC models. In "Age of Estimates: When to Submit Jobs to a Markov Machine to Maximize Revenue" (Liyanaarachchi et al., 30 Jul 2025), a single MM is a binary CTMC with state $X(t)\in\{0,1\}$, where state $0$ is free and state $1$ is busy. The transition rates are $\alpha$ for free-to-busy and $\beta$ for busy-to-free, with transition matrix

$$P(t)= \begin{bmatrix} \frac{\beta}{\alpha+\beta}+\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}-\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} \ \frac{\beta}{\alpha+\beta}-\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}+\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} \end{bmatrix}.$$

This model treats the machine’s availability as Markovian and emphasizes that the job-submission decision must exploit both the current estimate and the staleness of that estimate.

In "Optimum Monitoring and Job Assignment with Multiple Markov Machines" (Liyanaarachchi et al., 30 Jan 2025), a single MM is a three-state CTMC with $X(t)\in\{0,1,2\}$, where $0$ is free, $1$ is busy with an internal job, and $X(t)\in\{0,1\}$0 is processing an external job. The internal dynamics are $X(t)\in\{0,1\}$1 at rate $X(t)\in\{0,1\}$2 and $X(t)\in\{0,1\}$3 at rate $X(t)\in\{0,1\}$4; when an external job is accepted while the machine is actually free, the machine enters state $X(t)\in\{0,1\}$5 and returns to $X(t)\in\{0,1\}$6 at rate $X(t)\in\{0,1\}$7. This extension makes the controller’s action state-changing rather than merely observational.

Paper	MM state space	Main optimization target
(Liyanaarachchi et al., 30 Jul 2025)	$X(t)\in\{0,1\}$8	Average revenue per job
(Liyanaarachchi et al., 30 Jan 2025)	$X(t)\in\{0,1\}$9	Sampling allocation for monitoring and job assignment

A plausible implication is that the two-state model isolates utilization under stale binary availability estimates, whereas the three-state model makes explicit the feedback loop created when accepted external work changes the machine state itself.

2. Observation, estimation, and stale information

Both formulations assume that the controller does not continuously observe the MM. In the binary model, jobs arrive according to a Poisson process of rate $0$0, the resource allocator has a unit-sized buffer, and the MM is sampled through queries arriving according to an exponential clock of rate $0$1 (Liyanaarachchi et al., 30 Jul 2025). A query instantly reveals the actual state $0$2 and updates the estimate $0$3. Between queries, the estimate remains fixed while the true state continues to evolve, creating stale information. When a job is submitted, the true state is revealed as a side effect and the estimate is updated to busy, $0$4; after each submission, the system restarts from state $0$5 in the query-sampling chain.

The paper formalizes staleness through the age of the estimate,

$0$6

defined as the time elapsed since the allocator last knew the exact state of the MM. On a job arrival, the controller knows the current estimate $0$7 and the current age $0$8, and chooses a waiting time $0$9, which may be $1$0, finite positive, or $1$1. The age variable is therefore not an auxiliary diagnostic but the core state variable driving the optimal policy.

In the three-state model, the monitor samples the MM at rate $1$2 and maintains $1$3, with initial condition $1$4 (Liyanaarachchi et al., 30 Jan 2025). The joint process

$1$5

is itself a finite-state Markov chain on

$1$6

Because $1$7 is finite, there is a unique stationary distribution $1$8 satisfying $1$9.

This shared emphasis on sampled rather than fully observed state is the defining technical feature of the MM framework as developed in these papers.

3. Freshness, decision quality, and utility criteria

The binary model takes revenue as the primary performance criterion. The average revenue per job is

$\alpha$0

where $\alpha$1 is the number of successfully processed jobs, $\alpha$2 is the number of jobs discarded because the MM was busy at submission time, $\alpha$3 is the total number of arrivals, $\alpha$4 is the reward for a successful submission, and $\alpha$5 is the penalty for submitting while the MM is busy (Liyanaarachchi et al., 30 Jul 2025). The paper notes that average revenue per unit time is simply $\alpha$6, so maximizing $\alpha$7 is equivalent to maximizing throughput-adjusted revenue over time.

The three-state model argues that state-tracking quality and control quality are distinct. It adopts binary freshness via the fresh when close notion and a similarity map $\alpha$8, with long-term average freshness

$\alpha$9

Under the special similarity map

$\beta$0

and zero otherwise, one obtains

$\beta$1

so the staleness is

$\beta$2

(Liyanaarachchi et al., 30 Jan 2025).

To evaluate the quality of accept/reject decisions, the paper introduces the false acceptance ratio (FAR) and false rejection ratio (FRR). With $\beta$3 and $\beta$4 denoting the total accepted jobs and false acceptances by time $\beta$5,

$\beta$6

With $\beta$7 and $\beta$8 denoting total rejections and false rejections,

$\beta$9

For the single-MM model, the relevant sets are

$$P(t)= \begin{bmatrix} \frac{\beta}{\alpha+\beta}+\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}-\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} \ \frac{\beta}{\alpha+\beta}-\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}+\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} \end{bmatrix}.$$0

and

$$P(t)= \begin{bmatrix} \frac{\beta}{\alpha+\beta}+\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}-\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} \ \frac{\beta}{\alpha+\beta}-\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}+\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} \end{bmatrix}.$$1

The paper then gives the closed-form simplifications

$$P(t)= \begin{bmatrix} \frac{\beta}{\alpha+\beta}+\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}-\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} \ \frac{\beta}{\alpha+\beta}-\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}+\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} \end{bmatrix}.$$2

and

$$P(t)= \begin{bmatrix} \frac{\beta}{\alpha+\beta}+\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}-\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} \ \frac{\beta}{\alpha+\beta}-\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}+\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} \end{bmatrix}.$$3

where

$$P(t)= \begin{bmatrix} \frac{\beta}{\alpha+\beta}+\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}-\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} \ \frac{\beta}{\alpha+\beta}-\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}+\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} \end{bmatrix}.$$4

For multiple heterogeneous MMs, the same paper defines system-level weighted metrics:

$$P(t)= \begin{bmatrix} \frac{\beta}{\alpha+\beta}+\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}-\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} \ \frac{\beta}{\alpha+\beta}-\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}+\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} \end{bmatrix}.$$5

with $$P(t)= \begin{bmatrix} \frac{\beta}{\alpha+\beta}+\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}-\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} \ \frac{\beta}{\alpha+\beta}-\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}+\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} \end{bmatrix}.$$6 and $$P(t)= \begin{bmatrix} \frac{\beta}{\alpha+\beta}+\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}-\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} \ \frac{\beta}{\alpha+\beta}-\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}+\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} \end{bmatrix}.$$7, and

$$P(t)= \begin{bmatrix} \frac{\beta}{\alpha+\beta}+\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}-\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} \ \frac{\beta}{\alpha+\beta}-\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}+\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} \end{bmatrix}.$$8

WAR measures weighted decision error cost, while WAF measures weighted monitoring quality.

4. Optimal utilization of a single Markov Machine

The submission problem in the binary MM model is formulated as a stationary waiting-policy problem: given estimate $$P(t)= \begin{bmatrix} \frac{\beta}{\alpha+\beta}+\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}-\frac{\alpha}{\alpha+\beta}e^{-(\alpha+\beta)t} \ \frac{\beta}{\alpha+\beta}-\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} & \frac{\alpha}{\alpha+\beta}+\frac{\beta}{\alpha+\beta}e^{-(\alpha+\beta)t} \end{bmatrix}.$$9 and age $X(t)\in\{0,1,2\}$0, select $X(t)\in\{0,1,2\}$1 to maximize average revenue (Liyanaarachchi et al., 30 Jul 2025). For a given stationary policy,

$X(t)\in\{0,1,2\}$2

where $X(t)\in\{0,1,2\}$3 is the random age of the estimate when a job is accepted, $X(t)\in\{0,1,2\}$4 is the probability that a job is accepted when the estimate is $X(t)\in\{0,1,2\}$5, $X(t)\in\{0,1,2\}$6 is the expected reward from one accepted-job episode, and $X(t)\in\{0,1,2\}$7 is the expected number of jobs lost while that episode lasts.

The paper linearizes the fractional objective via Dinkelbach’s method:

$X(t)\in\{0,1,2\}$8

with

$X(t)\in\{0,1,2\}$9

and the key property

$0$0

Thus $0$1 can be found by bisection.

The principal structural result is that the optimal policy is either a threshold policy or a switching policy, depending on the sign of

$0$2

The paper further defines

$0$3

and

$0$4

If $0$5, the optimal waiting times are

$0$6

and

$0$7

This is the threshold regime. The threshold is

$0$8

If $0$9, submission is immediate; if $1$0, the controller waits $1$1.

If $1$2, the optimal policy is a switching policy:

$1$3

and

$1$4

where

$1$5

In this regime, the controller either submits immediately or waits indefinitely for a new query to refresh the estimate.

The paper’s corollary reduces the optimization to the compact forms

$1$6

and

$1$7

which make the threshold-versus-switching dichotomy analyzable.

5. Monitoring and assignment with multiple Markov Machines

The multi-MM setting considers $1$8 heterogeneous machines, each with parameters $1$9, $X(t)\in\{0,1\}$00, $X(t)\in\{0,1\}$01, and $X(t)\in\{0,1\}$02, under a total sampling budget

$X(t)\in\{0,1\}$03

(Liyanaarachchi et al., 30 Jan 2025). Each MM is specialized for one job type, and the external job of type $X(t)\in\{0,1\}$04 arrives at rate $X(t)\in\{0,1\}$05. The design problem is to allocate sampling rates across machines so as to optimize either decision quality or monitoring quality.

For WAR minimization, the paper shows that the objective is convex and reduces to

$X(t)\in\{0,1\}$06

subject to

$X(t)\in\{0,1\}$07

The Lagrangian is

$X(t)\in\{0,1\}$08

with sufficient KKT conditions

$X(t)\in\{0,1\}$09

$X(t)\in\{0,1\}$10

and

$X(t)\in\{0,1\}$11

Since $X(t)\in\{0,1\}$12, the optimal allocation uses the full budget,

$X(t)\in\{0,1\}$13

For fixed $X(t)\in\{0,1\}$14, each $X(t)\in\{0,1\}$15 is obtained by bisection, and $X(t)\in\{0,1\}$16 is then found by another bisection search, giving what the paper describes as a water-filling structure.

For WAF maximization, the paper instead minimizes total weighted staleness,

$X(t)\in\{0,1\}$17

under the same budget constraint. This problem is generally non-convex. The paper establishes that if either

$X(t)\in\{0,1\}$18

or

$X(t)\in\{0,1\}$19

then the optimal solution must satisfy

$X(t)\in\{0,1\}$20

It then proposes projected gradient descent on the scaled simplex

$X(t)\in\{0,1\}$21

with iteration

$X(t)\in\{0,1\}$22

If $X(t)\in\{0,1\}$23 is $X(t)\in\{0,1\}$24-smooth and $X(t)\in\{0,1\}$25, the method converges to a stationary point; because the problem is non-convex, the algorithm is run from multiple initializations and the best result is retained.

6. Conceptual implications and nontrivial phenomena

A central conceptual result of both papers is that tracking quality cannot be separated from control quality. In the three-state model, the monitor’s decisions are not passive: if it assigns an external job to a machine, that assignment can change the machine’s state (Liyanaarachchi et al., 30 Jan 2025). In the binary model, the authors explicitly shift the focus from merely tracking a Markov machine to using it optimally, showing that the best revenue policy is not necessarily immediate submission under an estimated free state, but can instead require waiting based on the age of the estimate (Liyanaarachchi et al., 30 Jul 2025).

The most direct formal link between freshness and decision quality in the three-state model is the bound

$X(t)\in\{0,1\}$26

This shows that FAR + FRR acts as a proxy upper bound on estimator staleness, while also preserving the paper’s main distinction: a stale estimate is not necessarily bad if it still leads to correct job decisions, and a fresh estimate is not necessarily sufficient for good decisions under the chosen semantics.

The literature also identifies a counterintuitive behavior: freshness does not always increase with sampling rate. The paper proves that if

$X(t)\in\{0,1\}$27

then

$X(t)\in\{0,1\}$28

If

$X(t)\in\{0,1\}$29

then

$X(t)\in\{0,1\}$30

and eventually becomes negative as $X(t)\in\{0,1\}$31 increases (Liyanaarachchi et al., 30 Jan 2025). The paper interprets this by stating that an “ignorant decision maker can sometimes be better than an under-informed decision maker.” When

$X(t)\in\{0,1\}$32

however, freshness increases monotonically with $X(t)\in\{0,1\}$33.

A further structural simplification occurs when

$X(t)\in\{0,1\}$34

Then states $X(t)\in\{0,1\}$35 and $X(t)\in\{0,1\}$36 become effectively indistinguishable, as do $X(t)\in\{0,1\}$37 and $X(t)\in\{0,1\}$38, and the paper gives

$X(t)\in\{0,1\}$39

together with the corresponding closed form for $X(t)\in\{0,1\}$40 (Liyanaarachchi et al., 30 Jan 2025).

Taken together, these results define Markov Machines not merely as Markovian server abstractions, but as partially observed stochastic resources for which monitoring, inference, and action are jointly optimized. The current theory is organized around two linked propositions: first, that the age or freshness of a state estimate is an operative control variable rather than a descriptive statistic; and second, that optimal policies may be structurally simple—threshold, switching, or water-filling—even when the underlying coupling between observation and utilization is nontrivial.