I’m a Ph.D. researcher at MPI CBS working on Reinforcement Learning that is safe, well understood, and ready for the real world.
Most RL breakthroughs live in simulators. Getting them to work under real-world constraints (safety requirements, limited data, deployment risk) is not just an engineering challenge, but the place where the hard and interesting theoretical problems are. I work on building a strong theoretical foundation for reinforcement learning for precisely these issues.
Education
-
Max Planck Institute for Human Cognitive and Brain SciencesNeural Data Science Lab
Ph.D. CandidateSep. 2022 - present -
University of Applied Sciences LeipzigM.S. in Electrical EngineeringSep. 2019 - Jul. 2021
Honors & Awards
-
Master's Thesis Award @ University of Applied Sciences Leipzig2021
News
Blog Posts
Selected Publications (view all )
Stochastic Decision Horizons for Constrained Reinforcement Learning
Nikola Milosevic, Leonard Franz, Daniel Haeufle, Georg Martius, Nico Scherf, Pavel Kolev
Preprint — arXiv:2602.04599 2026
We introduce a framework for constrained RL where constraint satisfaction is enforced at every step. A state-action continuation probability models constraint violations as effective horizon reductions. Building on Control as Inference, we develop two off-policy algorithms: max-entropy AS-SAC, which treats violations as absorbing states, and KL-regularized VT-MPO, which halts reward accumulation on violation. VT-MPO achieves competitive performance with 4× fewer environment steps on humanoid locomotion and identifies optimal reward–constraint tradeoff regimes in Safety Gymnasium.
Stochastic Decision Horizons for Constrained Reinforcement Learning
Nikola Milosevic, Leonard Franz, Daniel Haeufle, Georg Martius, Nico Scherf, Pavel Kolev
Preprint — arXiv:2602.04599 2026
We introduce a framework for constrained RL where constraint satisfaction is enforced at every step. A state-action continuation probability models constraint violations as effective horizon reductions. Building on Control as Inference, we develop two off-policy algorithms: max-entropy AS-SAC, which treats violations as absorbing states, and KL-regularized VT-MPO, which halts reward accumulation on violation. VT-MPO achieves competitive performance with 4× fewer environment steps on humanoid locomotion and identifies optimal reward–constraint tradeoff regimes in Safety Gymnasium.
The Geometry of Nonlinear Reinforcement Learning
Nikola Milosevic, Nico Scherf
Preprint — arXiv:2509.01432 2025
We present a unified geometric framework in which reward maximization, safe exploration, and intrinsic motivation emerge as aspects of a single optimization problem over the space of achievable long-term behavior. Classical RL techniques — policy mirror descent, natural policy gradient, trust-region methods — extend naturally to nonlinear utilities and convex constraints under this perspective, subsuming objectives related to robustness, safety, exploration, and diversity.
The Geometry of Nonlinear Reinforcement Learning
Nikola Milosevic, Nico Scherf
Preprint — arXiv:2509.01432 2025
We present a unified geometric framework in which reward maximization, safe exploration, and intrinsic motivation emerge as aspects of a single optimization problem over the space of achievable long-term behavior. Classical RL techniques — policy mirror descent, natural policy gradient, trust-region methods — extend naturally to nonlinear utilities and convex constraints under this perspective, subsuming objectives related to robustness, safety, exploration, and diversity.
Embedding Safety into RL: A New Take on Trust Region Methods
Nikola Milosevic, Johannes Müller, Nico Scherf
International Conference on Machine Learning (ICML) 2025
Reinforcement Learning (RL) agents can solve diverse tasks but often exhibit unsafe behavior. Constrained Markov Decision Processes (CMDPs) address this by enforcing safety constraints, yet existing methods either sacrifice reward maximization or allow unsafe training. We introduce Constrained Trust Region Policy Optimization (C-TRPO), which reshapes the policy space geometry to ensure trust regions contain only safe policies, guaranteeing constraint satisfaction throughout training. We analyze its theoretical properties and connections to TRPO, Natural Policy Gradient (NPG), and Constrained Policy Optimization (CPO). Experiments show that C-TRPO reduces constraint violations while maintaining competitive returns.
Embedding Safety into RL: A New Take on Trust Region Methods
Nikola Milosevic, Johannes Müller, Nico Scherf
International Conference on Machine Learning (ICML) 2025
Reinforcement Learning (RL) agents can solve diverse tasks but often exhibit unsafe behavior. Constrained Markov Decision Processes (CMDPs) address this by enforcing safety constraints, yet existing methods either sacrifice reward maximization or allow unsafe training. We introduce Constrained Trust Region Policy Optimization (C-TRPO), which reshapes the policy space geometry to ensure trust regions contain only safe policies, guaranteeing constraint satisfaction throughout training. We analyze its theoretical properties and connections to TRPO, Natural Policy Gradient (NPG), and Constrained Policy Optimization (CPO). Experiments show that C-TRPO reduces constraint violations while maintaining competitive returns.
Central Path Proximal Policy Optimization
Nikola Milosevic, Johannes Müller, Nico Scherf
The Exploration in AI Today Workshop at ICML 2025 2025
In constrained Markov decision processes, enforcing constraints during training is often thought of as decreasing the final return. Recently, it was shown that constraints can be incorporated directly in the policy geometry, yielding an optimization trajectory close to the central path of a barrier method, which does not compromise final return. Building on this idea, we introduce Central Path Proximal Policy Optimization (C3PO), a simple modification of PPO that produces policy iterates, which stay close to the central path of the constrained optimization problem. Compared to existing on-policy methods, C3PO delivers improved performance with tighter constraint enforcement, suggesting that central path-guided updates offer a promising direction for constrained policy optimization.
Central Path Proximal Policy Optimization
Nikola Milosevic, Johannes Müller, Nico Scherf
The Exploration in AI Today Workshop at ICML 2025 2025
In constrained Markov decision processes, enforcing constraints during training is often thought of as decreasing the final return. Recently, it was shown that constraints can be incorporated directly in the policy geometry, yielding an optimization trajectory close to the central path of a barrier method, which does not compromise final return. Building on this idea, we introduce Central Path Proximal Policy Optimization (C3PO), a simple modification of PPO that produces policy iterates, which stay close to the central path of the constrained optimization problem. Compared to existing on-policy methods, C3PO delivers improved performance with tighter constraint enforcement, suggesting that central path-guided updates offer a promising direction for constrained policy optimization.