Part I showed that the occupancy polytope $\mathcal{O}$ is a curved surface inside the simplex $\Delta(\mathcal{S}\times\mathcal{A})$, that the Fisher–Rao metric warps according to how often each state is visited, and that the Bellman equations couple the policy to its visitation through a drag term the natural policy gradient drops. This part asks a more algebraic question: what is the right metric on $\mathcal{O}$ in the first place? The answer depends on how one defines "right", and the most natural definition (invariance under structure-preserving maps of MDPs) does not single out a unique answer.
The Fisher–Rao metric on a probability simplex $\Delta(X)$ is usually introduced as the inner product of tangent vectors $u,v$ at $\omega$ given by $\sum_x u(x)v(x)/\omega(x)$. But the object that behaves well at the boundary, and that makes the probabilistic meaning transparent, is its inverse, the cometric. Fix a point $\omega \in \Delta(X)$; a function $c : X \to \mathbb{R}$ is then a random variable, and define the Fisher cometric through its variance:
$$g^{*}_{\omega}(c, c') \;:=\; \operatorname{Cov}_\omega(c, c') \;=\; \mathbb{E}_\omega[cc'] - \mathbb{E}_\omega[c]\,\mathbb{E}_\omega[c'].$$No logarithm appears. In coordinates the cometric is the covariance matrix of the coordinate indicators, $g^{*}_{ij}(\omega) = \omega_i\,\delta_{ij} - \omega_i\,\omega_j$, which is polynomial in $\omega$ with $\omega$ in the numerator: defined and smooth on the closed simplex, boundary included. The familiar metric is its pseudo-inverse: $g_{ij}(\omega) = \delta_{ij}/\omega_i$ in the mixture chart, with $\omega$ now in the denominator, blowing up at the boundary. The blow-up is not a property of the geometry; it is the singularity of the logarithm. The cometric degenerates at the boundary not by blowing up but by losing rank, and its rank is exactly $|\operatorname{supp}\omega| - 1$, the dimension of the face the boundary point sits on.
A function $c$ on $X$ induces a covector at every $\omega$: the map $\omega \mapsto \langle c, \omega \rangle = \sum_x c(x)\omega(x)$ is smooth, and its differential $\delta_\omega(c)$ is a linear functional on tangent vectors. On the full simplex, the kernel of this map is the constant functions: adding a constant to $c$ shifts $\langle c, \omega \rangle$ uniformly and changes no differential. A covector is a function modulo an additive constant.
On the occupancy polytope $\mathcal{O}$ the tangent space is smaller (only velocity fields consistent with the Bellman flow), so the kernel of $\delta_\omega$ is larger. The extra elements in the kernel are exactly the functions whose gradient is zero on $\mathcal{O}$: potential functions $V$ for which $c(s,a) = V(s) - \gamma \sum_{s'} P(s'\mid s,a) V(s')$, i.e. functions that can be written as the Bellman residual of a value function. Adding such a $c$ to the reward changes no occupancy-based expectation and hence no policy gradient. This is reward shaping, the gauge freedom of the occupancy view, and it is exactly the kernel of the shadow map on $\mathcal{O}$, not a separate theorem.
✎ Figures are planned for this post. The algebraic content is complete; interactive illustrations of the cometric at the boundary and the reward-shaping kernel are in preparation.