SA/BA/MA projects

The objective of this project is to study how Control Barrier Functions can be used together with flow-based generative models for safe trajectory and decision generation. The goal is to design methods that generate trajectories, actions, or state-action pairs that not only match a desired target distribution, but also satisfy safety requirements relevant to downstream control and robotic systems. One possible direction is to use a flow matching model as a generator of control-relevant outputs, and then incorporate barrier-based mechanisms to promote or enforce safety. This may involve safety filtering, projection-based corrections, or other ways of integrating barrier constraints while preserving as much as possible of the original generative behavior. More broadly, the project will investigate how modern generative models can be combined with control-theoretic safety tools in a way that is both practically useful and theoretically meaningful. Another possible direction is to incorporate safety more deeply into learning itself, for example by encouraging the model to generate outputs that are naturally compatible with safety constraints, by designing training objectives that penalize unsafe behavior, or by studying representations that are better aligned with downstream control tasks. The project may also compare different ways of introducing safety, such as post-processing generated outputs versus incorporating safety considerations directly during learning. From a methodological perspective, the project will investigate the trade-off between safety, generation quality, and computational efficiency. This is particularly important because one of the main promises of flow matching is fast and scalable inference, whereas safety mechanisms may introduce additional optimization or correction steps. The resulting methods will be evaluated on representative dynamical systems and motion generation tasks, with possible applications to robotics, safe planning, and control. Overall, this project sits at the intersection of generative modeling and control theory. It aims to combine the expressive power of modern generative models with the rigorous safety guarantees of barrier-based control, contributing toward trustworthy generative methods for safety-critical applications.

The goals of the project are as follows: 1. Review the literature on flow matching, diffusion and continuous-time generative modeling, and control barrier functions. 2. Formulate a framework for safe flow-based generation using CBF-inspired safety constraints. 3. Analyze the proposed method theoretically, including safety guarantees, feasibility, and the effect of safety mechanisms on generation quality and efficiency. 4. Validate the approach on numerical simulations and benchmark examples. 5. Write a report and prepare a presentation. If the final results are promising, they may be developed into a machine learning conference paper.

Please send your resume/CV (including lists of relevant publications/projects) and transcript of records in PDF format via email to Dr. Han Wang hanwang@control.ee.ethz.ch and Dr. Keyan Miao (Oxford) keyan.miao@eng.ox.ac.uk

Self-commissioning controllers for simple control devices regulate a single output variable using one actuator. When first connected, the controller learns the system’s dynamic behaviour and automatically designs suitable proportional–integral (PI) gains to ensure smooth and stable performance. For this learning-based approach to work effectively, key hyperparameters of the underlying learning and identification algorithms must be correctly selected during commissioning. In practice, these hyperparameters strongly influence convergence speed, robustness, and closed-loop performance. Misconfigured parameters, overly aggressive or slow responses, or changing operating conditions can therefore require systematic analysis and re-tuning. The goal of this project is to develop a program that can analyse a controller’s behaviour, identify the causes of undesired performance, and automatically adjust its hyperparameters to improve it. In particular, learning-based and system identification methods will be used to support structured hyperparameter identification and performance assessment. The system should reliably diagnose performance issues, tune hyperparameters of the learning algorithm, and ensure robust adaptation in real-world applications. Ultimately, the aim is to move closer to true plug-and-play functionality by enabling the controller to autonomously monitor its behaviour and take corrective actions. This project will be co-supervised with Belimo Automation AG. In addition to simulation-based development, the work will include hands-on experimentation in an industrial setting using real HVAC devices and embedded controllers. The student will gain practical experience with real hardware, data acquisition, controller implementation, and performance evaluation under realistic operating conditions. This provides direct exposure to industrial product development and validation processes. We are looking for a talented, motivated student interested in self-learning control systems and smart building technologies and HVAC systems with • Background/interest in control systems, automation, or mechatronics engineering. • Familiarity with PI control, system identification, and dynamic modelling. • Basic knowledge in data-based and machine learning methods. • Experience with MATLAB/Simulink or a similar simulation environment. • Strong analytical and problem-solving skills.

The tasks that will be carried out in this thesis are: 1. Identify the relevant hyperparameters that are preselected based on the application and analyse how different applications influence these parameters. 2. Define suitable performance metrics and develop a method to analyse the controller’s behaviour online to detect undesired responses 3. Design a robust and systematic program capable of debugging and automatically adjusting the parameters and performing other corrective actions to resolve potential issues 4. Numerical simulations to test the proposed strategy across different application scenarios. 5. Documentation and final presentation of the work

- Dr. Efe Balta efe.balta@inspire.ch - Dr. Varsha N. Behrunani varsha.behrunani@belimo.ch

Meta Reinforcement Learning (meta RL) studies machine learning algorithms that learn to reinforcement learn [1]. Rather than learning from scratch on a given task through a large number of repeated episodes, agents in meta RL first perform meta-learning on meta-learning data (outer loop) which is a collection of tasks/environmental realisations sampled from a given distribution. Then, during test-time or online play time, the algorithms fine-tune or adapt to the particular task at hand (inner loop) and ideally do so fast through a small number of episodes (few-shot learning), as visualized in the Figure [1]. Such a setting is especially relevant for robotics applications such as legged robotics [2] or quadrotors[3], where the meta-learning happens in simulation due to safety and cost considerations, and the adaptation to the specific instantiation of the environment happens during test-time or online. Moreover, for many such applications there is no availability to do more than a single episode for fine-tuning, and the agent needs to adapt to control online, during a single trajectory, aka in a zero-shot setting. RL2 [4] and VariBAD [5] are among the algorithms that operate in this zero-shot meta RL setting. In addition, there is significant literature on online learning methods for control that study the zero-shot RL setting often assuming no offline training availability[6, 7]. In contrast to meta-RL, these methods concentrate on linear dynamical systems and provide rigorous theoretical guarantees on stability and online reward performance in the form of regret. This thesis concentrates on the zero-shot meta RL setting and aims to bridge the gap between data-driven meta RL methods and theoretically grounded online/adaptive control. It leverages the theoretical foundations in online learning [8] and adaptive control theory [9], as well as the advancements in meta RL [10, 1, 4] to introduce no-regret (sublinear) algorithms for control-affine dynamical systems. Such systems arise frequently in practice and capture mild nonlinearities through a control-affine structure, where the system dynamics depend linearly on the control input and affinely on the state through a known functional form. A gray-box modeling approach is adopted: the structural form of the system dynamics is assumed known, while key parameters—including the system matrices and additive non-stochastic disturbances—remain unknown. This modeling choice balances expressiveness and tractability, enabling both practical applicability and rigorous theoretical analysis. In addition to reinforcement learning and online learning methods [11, 12], the thesis will draw on optimal control results for control-affine systems [13, 14]. The thesis will explore two fundamental directions of synthesizing RL algorithms: • Model-free policy gradient methods, where the control policy is updated directly online without explicitly identifying a system model. An example of this approach is given in [7], where an affine policy is adapted online via gradient- based updates. • Model-based indirect methods, where the system dynamics and uncertainty are first learned online, and the control policy is subsequently derived using a certainty-equivalence principle. A representative example is the PLOT algorithm [6], which employs a receding horizon controller based on recursively updated model estimates. **By unifying ideas from meta RL, online optimization, and adaptive control, this thesis aims to develop policies that adapt online in a zero-shot setting while providing strong theoretical guarantees on stability and regret.** [1] J. Beck, R. Vuorio, E. Zheran Liu, Z. Xiong, L. Zintgraf, C. Finn, and S. Whiteson, “A survey of meta-reinforcement learning,” arXiv e-prints, pp. arXiv–2301, 2023. [2] A. Belmonte-Baeza, J. Lee, G. Valsecchi, and M. Hutter, “Meta reinforcement learning for optimal design of legged robots,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 12134–12141, 2022. [3] S. Belkhale, R. Li, G. Kahn, R. McAllister, R. Calandra, and S. Levine, “Model-based meta-reinforcement learning for flight with suspended payloads,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 1471–1478, 2021. [4] Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel, “Rl2: Fast reinforcement learning via slow reinforcement learning,” arXiv preprint arXiv:1611.02779, 2016. [5] L. Zintgraf, K. Shiarlis, M. Igl, S. Schulze, Y. Gal, K. Hofmann, and S. Whiteson, “Varibad: A very good method for bayes-adaptive deep rl via meta-learning,” arXiv preprint arXiv:1910.08348, 2019. [6] A. Tsiamis, A. Karapetyan, Y. Li, E. C. Balta, and J. Lygeros, “Predictive linear online tracking for unknown targets,” in International Conference on Machine Learning, pp. 48657–48694, PMLR, 2024. [7] A. Karapetyan, D. Bolliger, A. Tsiamis, E. C. Balta, and J. Lygeros, “Online linear quadratic tracking with regret guarantees,” IEEE Control Systems Letters, vol. 7, pp. 3950–3955, 2023. [8] N. Cesa-Bianchi and G. Lugosi, Prediction, learning, and games. Cambridge university press, 2006. [9] A. M. Annaswamy, “Adaptive control and intersections with reinforcement learning,” Annual Review of Control, Robotics, and Autonomous Systems, vol. 6, no. 1, pp. 65–93, 2023. [10] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in Inter- national conference on machine learning, pp. 1126–1135, PMLR, 2017. [11] M. Simchowitz and D. Foster, “Naive exploration is optimal for online lqr,” in International Conference on Machine Learning, pp. 8937–8948, PMLR, 2020. [12] A. Wagenmaker and K. Jamieson, “Active learning for identification of linear dynamical systems,” in Conference on Learning Theory, pp. 3487–3582, PMLR, 2020. [13] Z. Aganovic and Z. Gajic, “The successive approximation procedure for finite-time optimal control of bilinear systems,” IEEE Transactions on Automatic Control, vol. 39, no. 9, pp. 1932–1935, 2002. [14] Z. Yuan and J. Cort´es, “Data-driven optimal control of bilinear systems,” IEEE Control Systems Letters, vol. 6, pp. 2479–2484, 2022

1. Perform literature review of state-of-the-art meta RL and online control methods, 2. Get familiar with the SIMULINK/Python simulation of a test environment to test and validate derived algorithms, 3. Implement existing methods, such as [4, 5, 7, 6] on the simulator and judge their performance, 4. Propose an improved algorithm that is suited for control-affine/bilinear systems, 5. Provide theoretical guarantees in the form of regret, under suitable assumptions, 6. Verify the proposed method in simulation, 7. Prepare the final report and presentation **Qualifications** • Good programming skills (Python, Matlab) • Good understanding of Control and RL foundations **Arrangements** The project will be co-supervised with Belimo Automation AG.

Please send an email to: aren.karapetyan@belimo.ch, and efe.balta@inspire.ch

Object manipulation with contact-rich interations involves switching between contact and no-contact modes. Open-loop execution of stiff position control fails under these hybrid dynamics by ignoring dynamic coupling, by being too slow to solve the combinatorial planning of optimal motion sequences, and by being inaccurate under occlusion and physical uncertainties. Thus, the motivation for robust, fast feedback controllers, ideally which address the impracticality of modeling changing environments. Several classes of feedback-based control strategies have been proposed. Compliance-based strategies, such as impedance control, avoid crashing by adjusting control magnitude based on resistance. Extending the idea, variable impedance control adapts online based on safety and dexterity (precision) requirements. Recent work [1] learns a general representation of variable impedance expert demonstrations through inverse reinforcement learning (IRL) to generate new policies with policy gradient reinforcement learning (RL), while [2] uses probabilistic ensembles as a variable impedance model within model predictive control (MPC). Beyond compliance based strategies, hybrid force-motion control explicitly jointly satisfies constraints needed to maintain contact during manipulation. [3] plans over a predefined skill set to specify force and velocity targets, which are then tracked using a low-level torque controller inconveniently requiring careful parameter tuning. [4] instead learn low-level force control parameters with RL, integrating RL into both hybrid force position control and admittance controllers. Optimization-based formulations such as MPC-based approaches offer the advantage of incorporating look-ahead planning over contact modes. [5] trains with supervised learning a classifier to select contact mode sequences offline, then solves a fast reactive controller online. [6] devises a mode-invariant, contact-implicit MPC formulation using linear complementarity contact formulation and bilevel trajectory planning. [1] X. Zhang, L. Sun, Z. Kuang, and M. Tomizuka, “Learning variable impedance control via inverse reinforcement learning for force-related tasks,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 2225–2232, 2021. [2] A. S. Anand, J. T. Gravdahl, and F. J. Abu-Dakka, “Model-based variable impedance learning control for robotic manipulation,” Robotics and Autonomous Systems, vol. 170, p. 104531, 2023. [3] J. Liang, X. Cheng, and O. Kroemer, “Learning preconditions of hybrid force-velocity controllers for contact-rich manipulation,” in Proceedings of The 6th Conference on Robot Learning (K. Liu, D. Kulic, and J. Ichnowski, eds.), vol. 205 of Proceedings of Machine Learning Research, pp. 679–689, PMLR, 14–18 Dec 2023. [4] C. C. Beltran-Hernandez, D. Petit, I. G. Ramirez-Alpizar, T. Nishi, S. Kikuchi, T. Matsubara, and K. Harada, “Learning force control for contact-rich manipulation tasks with rigid position- controlled robots,” IEEE Robotics and Automation Letters, vol. 5, no. 4, pp. 5709–5716, 2020. [5] F. R. Hogan, E. R. Grau, and A. Rodriguez, “Reactive planar manipulation with convex hybrid mpc,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), p. 247–253, IEEE Press, 2018. [6] S. Le Cleac’h, T. A. Howell, S. Yang, C.-Y. Lee, J. Zhang, A. Bishop, M. Schwager, and Z. Manch- ester, “Fast contact-implicit model predictive control,” IEEE Transactions on Robotics, vol. 40, pp. 1617–1629, 2024.

The project will investigate low-level control approaches for achieving desired carton configurations based on estimated state feedback. Cartons will be modeled as articulated structures with com- pliant joints, and control objectives will be defined in terms of target fold angles, panel poses and contact constraints. The student will explore model-based and optimization-based control formula- tions, such as hybrid force–position control, impedance control, or constrained optimal control (e.g. constrained MPC), to regulate interaction forces and motion under uncertainty. The developed con- trollers will be validated on a robotic packaging setup, with experiments conducted on-site at the Swiss Cobotics Competence Center (S3C) in Biel.

Please send your resume/CV (including lists of relevant publications/projects) and transcript of records in PDF format via email to Paula Stocco (pstocco@control.ee.ethz.ch), Dr. Efe Balta (efe.balta@inspire.ch), and Dr. Hakan Girgin (hakan.girgin@s3c.swiss)

The increasing integration of power-electronics-based resources is significantly altering power system dynamics. Modern grids are characterized by complex interactions between subsystems, including network dynamics and converter-based devices. Impedance-based analysis provides a powerful framework for both stability assessment and control design in such systems. In practice, the dynamic behavior of the power grid is continuously changing and is generally unknown from the perspective of converter nodes. Online grid impedance identification is therefore essential to enable real-time stability analysis and control design. Traditional single-port dynamic impedance identification methods focus on a single interconnection point of a converter-based resource. These methods assume an isolated identification scenario, where only one agent conducts the identification at a time, and do not consider the effects of simultaneous identification by multiple agents. Furthermore, multi-port scenarios, where grid data from multiple ports is used to model the interconnections between multiple sources, remain largely unexplored. This thesis addresses these gaps by investigating multi-agent grid impedance identification scenarios in three-phase power systems. Both global multi-port and locally simultaneous multi-agent single-port identification frameworks will be developed and systematically compared. The study will also explore downstream applications such as stability assessment and control design based on the identified dynamic grid impedances.

- Introduce the concept of dynamic grid impedance modeling and its relevance in three-phase power systems for stability studies and control design. - Analyze relationships between different grid impedances (global multi-port vs. multiple local single-port) and assess the influence of grid topology on these relationships. - Develop and study system identification frameworks for both global multi-port and multi-agent single-port scenarios. Investigate suitable excitation methods and compare the identification accuracy and reliability of the two approaches. Implementation will be conducted in MATLAB/Simulink. - Apply the identified dynamic grid impedances to a downstream task, such as stability analysis, oscillation damping, or control design. The application will be implemented and evaluated in MATLAB/Simulink.

Dr. Mohamed Abdalmoaty (mabdalmoaty@ethz.ch) Dr. Verena Häberle (verenhae@ethz.ch) _Remark:_ This project is intended as a master thesis and aims to strengthen collaboration between the Automatic Control Laboratory (IFA) and the Power Systems Laboratory (PSL) at ETH Zurich. Depending on the student's interest, we can provide help to publish the results in a conference paper. _Background:_ Competitive candidates will have a background in systems & control, signal processing, and/or power systems/electronics. A prior experience or coursework related to grid-connected converters or system identification would be highly beneficial. Proficiency in MATLAB/Simulink is required for implementing and testing the methods.