TutORials

Sequential decision making under uncertainty arises across business,
engineering, and science. Markov Decision Processes (MDPs) offer a rich modeling language, but translating models into implementable policies requires confronting challenging state and action spaces and non-convex optimization landscapes. Approximations are essential, and making them accessible and performant without algorithmic expertise allows users to focus on formulating MDPs that capture real-world features rather than simplifying their models to facilitate solution. Obtaining a policy involves a three-stage COR cycle: Constructing a parametrized model that approximates the MDP, Optimizing for the parameters of this model, and Refining the model by learning from the solution. These stages involve design choices that burden the user, leading them to simplify or skip stages.

This tutorial provides an overview of methods that reduce the user burden using COR as its guide. Construction ranges from lightweight extensible models to automated structural reformulations. Optimization leverages mathematical programming and first-order methods that have matured in theory and off-the-shelf software. Refinement performs feature expansion and approximation tightening, feeding back into the construct stage. When a combination of techniques across COR stages results in both accessibility and performance, we refer to it as a self-adapting approximation. Such an approximation adapts to problem structure, instance data, or solution progress. We guide readers through these ideas across weakly coupled MDPs and general continuous state-action MDPs, providing generalizable foundations and a code base to get started on a simple example. The tutorial concludes by outlining the potential for foundation models and quantum computing to further impact this area.

Andre A. Cire is an Associate Professor of Operations Management and Analytics at the University of Toronto, cross-appointed between the Rotman School of Management and the Department of Management at the Scarborough campus. His research bridges methodological development and practical applications in optimization, with a focus on sequential decision-making, mathematical programming, dynamic programming, network models, and applications in scheduling, healthcare, and supply chains. Andre earned his PhD in Operations Research from Carnegie Mellon University, and his contributions have been recognized with awards such as the INFORMS Harvey J. Greenberg Award (2024), an honorable mention for the INFORMS Computing Society Prize (2023), and research and teaching awards at the University of Toronto. He has also held senior editorial and service roles for multiple journals and conferences including INFORMS Journal on Computing, AAAI, and NeurIPS.

Selvaprabu (Selva) Nadarajah is an Associate Professor of Information and Decision Sciences and Bielinski Family Endowed Scholar at the College of Business Administration, University of Illinois Chicago (UIC). He is also a UIC Global Scholar and previously served as the Decision Intelligence R&D Lead at the Discovery Partners Institute, the innovation hub of the University of Illinois System. He is an Associate Editor for Decision Sciences, Operations Research, and Production and Operations Management. His research develops self-adapting approximation methods for large-scale Markov decision processes. He applies these methods to energy investment and operations problems, including real options valuation of commodity and energy assets, renewable energy procurement, and capacity expansion. This work has been recognized with awards from INFORMS, the Commodity and Energy Markets Association, and NeurIPS. It has been funded by the Alfred P. Sloan Foundation, Argonne National Laboratory, and industry. He has received the UIC College of Business Teaching Excellence Award and the EnergyTech University Prize Faculty Explorer Award from the U.S. Department of Energy for educational initiatives. He holds a PhD in Operations Research from the Tepper School of Business at Carnegie Mellon University.

Parshan Pakiman is an Assistant Professor in the Department of Operations Management and Strategy at the University at Buffalo School of Management. His research focuses on approximate dynamic programming and data-driven optimization methods for sequential decision making, with an emphasis on large-scale Markov decision processes and structured relaxations. His work develops algorithms with strong theoretical guarantees that minimize the need for human intervention during deployment, including model selection and parameter tuning. These methods are designed to be scalable, interpretable, and accessible to practitioners, with applications in healthcare operations and revenue management. Prior to joining the University at Buffalo, he was a Principal Researcher at the Tolan Center for Healthcare at the University of Chicago Booth School of Business. He received his PhD in Information and Decision Sciences from the University of Illinois Chicago and his BSc in Applied Mathematics from the University of Tehran.

Negar Soheili is an Associate Professor in the Department of Information and Decision Sciences at the University of Illinois Chicago. She received her PhD in Operations Research from Carnegie Mellon University. Her research develops scalable optimization algorithms for machine learning and sequential decision-making under uncertainty. In particular, she designs efficient first-order methods for large-scale constrained optimization and the approximation of Markov decision processes, with applications in healthcare, supply chains, and business analytics. She launched the PhD program in Operations Research and Business Analytics at UIC. She serves as an Associate Editor for the INFORMS Journal on Computing and for Optimization Letters.