Reinforcement Learning in Acumen

What exactly is Acumen?

Acumen is an experimental Domain - Specific Language (DSL) written in Scala, a programming language that runs on the Java Virtual Machine (JVM). It is bundled as an Integrated Developing Environment (IDE) that provides a wide range of tools to the user such as a code editor, error reporting and 3D animations, to name but a few.

The purpose of this language is to model and simulate Cyber-Physical Systems (CPS), dynamical systems that can exhibit discrete, continuous or hybrid (discrete/continuous) behavior. Although small in size, the language provides a variety of expressions to the user allowing for simulation of larger systems, as well as the ability to model systems that contain multiple variables dynamically changing over time.

If you want to know more about the language, please follow the following link: Acumen Language.

Reinforcement Learning

Reinforcement Learning (RL) is an area of ML, where software agents experiment with an unknown environment by taking actions. During this process they are able to adjust and optimize their behavior through a feedback mechanism based on rewards.

This type of learning can be modeled as a Markov Decision Process (MDP) which consists of the following:

1. A set of states
2. A set of actions that the agent is allowed to make
3. The probability of transitioning to one state from another
4. The reward returned after transitioning to a new state
5. A set of rules describing the agent’s observation of the environment

Reinforcement Learning is used in many areas. Examples include the famous AlphaGo Zero which learnt to play GO, self-driving cars and autonomous driving, healthcare, robotic applications trading and finance and so on so forth. There are currently an uncountable number of future and current applications of Reinforcement Learning!

How RL can be used in Cyber-Physical Systems

My work focused around building a Reinforcement Learning example, in which an agent would learn to navigate a custom build maze. Initially I created this idea in a Python application, in order to showcase the approach and that it works flawlessly. The real question was “Can we implement such methods to a language built for Cyber-Physical Systems?” If so how, and would it be beneficial?

Citation

If you are using this source code in your research please cite:

• Sotirios Tzamaras, Stavros Adam, Walid Taha. Intelligent Techniques and Hybrid Systems Experiments Using the Acumen Modeling and Simulation Environment. Proceedings of the Artificial Intelligence Applications and Innovations (AIAI 2021).

@inproceedings{TzamarasAT21,
author    = {Sotirios Tzamaras and Stavros Adam and Walid Taha},
title     = {Intelligent Techniques and Hybrid Systems Experiments Using the Acumen Modeling and Simulation Environment},
booktitle = {Artificial Intelligence Applications and Innovations - 17th {IFIP} {WG} 12.5 International Conference, {AIAI} 2021,
            Hersonissos, Crete, Greece, June 25-27, 2021, Proceedings},
series    = {{IFIP} Advances in Information and Communication Technology},
publisher = {Springer},
year      = {2021},
}