ML is the art of making computer programs learned from experience. A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E [17]. For example, task T can be playing checkers, experience E is playing thousands of checkers games, and the performance P is the fraction of games won against human opponents. We can divide the learning problems into three classes:

· Learning is called supervised if the experience E takes the form of a labeled dataset (x,y), the task is to learn a function that maps x to y,

· Learning is called unsupervised if E takes the form of an unlabeled dataset. The task is to learn underlying structure,

· Reinforcement learning (RL) is when the experience E takes the form of state-action pairs and corresponding rewards. The task is to maximize future rewards over a number of time steps.

Tasks are usually described in terms of how ML should process a data item (i.e., an example). If the desired behavior is to assign the input data item to one category among several, this is a classification task, e.g., object recognition. Other examples of tasks are machine translation, transcription, anomaly detection, etc [18] [19].

Reinforcement Learning (RL) is particularly interesting for playing games since its task involves interaction with an environment, by committing actions and receiving rewards for these actions [20]. In RL, the experience is a set of episodes. Each episode is a sequence of tuples (State, Action, Reward, Next State), the performance measure is the discounted total reward, and the task basically consists of playing (Figure 2). A more sophisticated description of playing is adopting a policy that maps states of the game to actions. If this mapping takes the form of a neural network, a deep one, we refer to Deep Reinforcement Learning (DRL).

Given an agent that interacts with an environment through percepts (observations) and actions, the goal of reinforcement learning is to find an optimal policy π * that maximizes the expected total sum of rewards the agent receives during a run, while starting from an initial state s 0 ∈ S [22]. Usually, the performance of a given policy π is evaluated as shown in Equation (1):

eval ( π | s 0 ) = E ρ ( π ) [ ∑ t = 0 τ     γ t r ( s t + 1 ) ] = E ρ ( π ) [ R 0 | s 0 ] (1)

where γ is a discount factor, and the expectation is over all the possible runs (or traces) allowed by the policy π . R 0 is the total reward for t = 0 . Among the most popular algorithms to reach an optimal policy in this context are value iteration and Q-learning. Value iteration assumes that the reward model and the transition model are known a priori. Q-learning actively learns a utility function for (State, Action) pairs [9] as detailed in Equations (2) and (3):

Q ( s t , a t ) = E [ R t | ( s t , a t ) ] (2)

π * ( s ) = arg max a Q ( s , a ) (3)

By combining these ideas from reinforcement learning with the recently re-invented neural networks a new set of algorithms emerges and is dubbed

DRL. One of the seed contributions in this area is value learning. In [23], a Convolutional Neural Network (CNN) was trained to play Atari with a variant of Q-learning. The CNN approximates the utility function of Q-learning based on raw pixels for input and an estimation of future reward as output. The loss function for value learning is in Equation (4):

L = E [ ‖ Q real ( s , a ) − Q predicted ( s , a ) ‖ ] (4)

where Q predicted ( s , a ) is the output of the neural network and Q real ( s , a ) is the actual Q value is in Equation (5):

Q real ( s , a ) = r + γ Q ( s ′ , a ′ ) (5)

Another approach is policy learning, where the policy is learned directly through training a neural network and without passing through value learning. Policy learning is shown to be very successful at addressing challenges of (1) large or continuous action space such as in self-driving, and (2) stochastic transition and reward models. Policy learning is based on a set of policy gradient methods with the goal of learning a probability distribution over the actions given a state P ( a | s ) . The training is performed through continuous running of episodes and simply increasing the probability of actions that resulted in high reward, and decreasing the probability of actions that resulted in low reward. The loss function in Equation (6):

L = E [ − log P ( a | s ) R ] . (6)

ML, RL and DRL are heavily used in gaming to develop not only competitive agents but also collaborative agents and NPCs. Alpha Go beated the top human player at Go in 2016. DeepMind introduced AlphaZero in 2017, a single system that taught itself through self-play how to master the games of chess, shogi (Japanese chess), and Go [24] [25]. MuZero, a general-purpose algorithm, was able to master Go, chess, shogi and Atari without needing to be told the rules, thanks to its ability to plan winning strategies in unknown environments [26]. A summary of these algorithms as per [27] is depicted in Table 1.

Similarly, AlphaStar, a multi-agent RL system, was developed to play StarCraft

II at Grandmaster level [30]. OpenAI developed Dota 2 AI agents, called OpenAI Five, and made them learn by playing over 10,000 years of games against themselves. The agents demonstrated the ability to defeat world champions in Dota 2 [31]. Using the same RL model as OpenAI Five boosted with additional techniques, OpenAI trained a pair of neural networks to solve the Rubik’s Cube with a human-like robot hand. Facebook and Carnegie Mellon built the first AI-based game that beats pros in 6-player poker [32].

The Unity Machine Learning Agents Toolkit (ML-Agents) is an open-source project that enables games and simulations to serve as environments for training intelligent agents [33] [34]. The training of agents is performed using ML techniques including reinforcement learning, imitation learning, and neuroevolution [35]. There are 3 main kinds of objects in a learning environment in Unity ML-Agents:

· Agent: Each Agent can have a unique set of states and observations, take unique actions within the environment, and receive unique rewards for events within the environment. An agent’s actions are decided by the brain it is linked to.

· Brain: Each Brain defines a specific state and action space, and is responsible for deciding which actions each of its linked agents will take.

· Academy: Each environment contains a single academy which defines the scope of the environment, in terms of engine configuration, frameskip, and global episode length.

With the Unity ML-Agents toolkit, a variety of training scenarios are possible, depending on how agents, brains, and rewards are connected. Despite the lack of detailed studies on Unity ML-Agents, a few games have been implemented using Unity and its ML-Agents package where the training has been done using reinforcement learning including imitation learning and self-play. Figure 3 illustrates the Unity ML-Agents Learning Environment [36]. An AI-based agent has been implemented in Connect4 game using Unity ML-Agents [37]. The agent training was performed using the Proximal Policy Optimization (PPO) algorithm. Moreover, a RL model using Hierarchical Critics (RLHC) algorithm has been implemented in Unity ML-Agents which performance was compared with the PPO model using two different competitive games—Soccer and Tennis [38].

OpenAI is a research lab whose mission is to ensure that artificial general intelligence benefits all of humanity [21]. OpenAI provides various tools to support applications of RL and ML in scientific research and game design and development.

Petting Zoo is a python library for conducting research in multi-agent environments. Petting Zoo is a multi-agent version of OpenAI Gym. What OpenAI Gym has done with single agent reinforcement learning environments, Petting Zoo was developed with the goal of doing the same with multi-agent environments. PettingZoo’s API, while inheriting many features of OpenAI Gym, is unique amongst Multi Agent Reinforcement Learning (MARL) APIs. PettingZoo models environments as Agent Environment Cycle (AEC) games, in order to be able to cleanly support all types of multi-agent RL environments under one API and to minimize the potential for certain classes of common bugs. Petting Zoo includes 63 default environments [44].

Dopamine is a TensorFlow based research framework for the fast prototyping of reinforcement learning algorithms. Dopamine supports multiple agents like DQN, SAC and these are implemented using JAX which is a Python library for high-performance ML research. Dopamine supports Atari environments and OpenAI’s MuJoCo environments [45].

Portability in ML/RL toolkits is the usability of the same toolkit in different environments. The pre-requirement for portability is the generalized abstraction between the toolkit logic and its interfaces. When a ML/RL toolkit with the same functionality is developed for several environments, portability is the key issue for development cost reduction.

Interoperability refers to the capability of different ML/RL toolkits to communicate with one another and with game engines freely and easily. Toolkits that are interoperable exchange information in real-time, without the need for specialized or behind-the-scenes coding.

The training speed of agents in a ML/RL depends on the complexity and analysis of the algorithm used to train that agent. Booth et al. provide a comparison study of different algorithms including PPO in ML-Agents and A2C, ACKTR and PPO2 algorithms of OpenAI Baselines [46].

Multi-task learning is an ML/RL approach in which we try to learn multiple tasks simultaneously, optimizing multiple loss functions at once. Rather than training independent models for each task, we allow a single model to learn to complete all of the tasks at once. In this process, the model uses all of the available data across the different tasks to learn generalized representations of the data that are useful in multiple contexts.

An environment might contain a single agent or multiple agents. In the case of multiple agents, each agent might have a different set of actions to perform and the agents might need interaction between them as the training goes on [47]. This requires a different training methodology from training a single agent (see Figure 4).

Usability is a measure of how well a specific user in a specific context can use a ML toolkit to design and develop games effectively, efficiently and satisfactorily. Game designers usually measure a toolkit design’s usability throughout the development process’ from wireframes to the final deliverable’ to ensure maximum usability.

ML/RL toolkit documentation is written text or illustration that accompanies toolkits or is embedded in the source code. The documentation either explains how the toolkit operates or how to use it. Documentation is an important part of game design and development when using ML/RL toolkits. Types of documentation include; 1) Requirements—Statements that identify attributes, capabilities, characteristics, or qualities of a toolkit, 2) Architecture/Design—Overview of the toolkit design and includes relations to an environment and construction principles to be used, 3) Technical—Documentation of code, algorithms, interfaces, and APIs, 4) End user—Manuals for the end-user, administrators and support staff, and 5) Marketing—How to market the product and analysis of the market demand.

The learning strategies are the different techniques ML/RL toolkits and frameworks used to train the agents in game design and development. These strategies are translated through machine learning algorithms including:

· Naïve Bayes Classifier Algorithm (Supervised Learning—Classification) based on Bayes’ theorem and classifies every value as independent of any other value. It allows predicting a class/category, based on a given set of features, using probability.

· K Means Clustering Algorithm (Unsupervised Learning—Clustering) is a type of unsupervised learning, which is used to categorize unlabelled data, i.e., data without defined categories or groups. The algorithm works by finding groups within the data, with the number of groups represented by the variable K. It then works iteratively to assign each data point to one of K groups based on the features provided.

· Support Vector Machine Algorithm (Supervised Learning—Classification) analyzes data used for classification and regression analysis. It essentially filters data into categories, which is achieved by providing a set of training examples, each set marked as belonging to one or the other of the two categories. This algorithm then works to build a model that assigns new values to one category or the other.

· Linear Regression (Supervised Learning/Regression) is the most basic type of regression. Simple linear regression allows us to understand the relationships between two continuous variables.

· Logistic Regression (Supervised learning—Classification) focuses on estimating the probability of an event occurring based on the previous data provided. It is used to cover a binary dependent variable that is where only two values, 0 and 1, represent outcomes.

· Artificial Neural Networks (Reinforcement Learning) comprise “units” arranged in a series of layers, each of which connects to layers on either side. ANNs are essentially a large number of interconnected processing elements, working in unison to solve specific problems.

· Random Forests (Supervised Learning—Classification/Regression) is an ensemble learning method, combining multiple algorithms to generate better results for classification, regression and other tasks. Each individual classifier is weak, but when combined with others, it can produce excellent results. The algorithm starts with a “decision tree” (a tree-like graph or model of decisions) and an input is entered at the top. It then travels down the tree, with data being segmented into smaller and smaller sets, based on specific variables.

· Nearest Neighbours (Supervised Learning) The K-Nearest-Neighbour algorithm estimates how likely a data point is to be a member of one group or another. It essentially looks at the data points around a single data point to determine what group it is actually in.

Reward functions describe how the agent “ought” to behave. It is an incentive mechanism that tells the agent what is correct and what is wrong using reward and punishment. The goal of agents in RL is to maximize the total rewards. Sometimes we need to sacrifice immediate rewards in order to maximize the total rewards. Reward strategy depends on the parameters a game developer setup during the creation of a game environment.

Precision is one indicator of a machine learning model’s performance - the quality of a positive prediction made by the model. Precision refers to the number of true positives divided by the total number of positive predictions (i.e., the number of true positives plus the number of false positives). It helps us to measure the ability to classify positive samples in the model. Precision and recall are two important model evaluation metrics. While precision refers to the percentage of relevant results, recall refers to the percentage of total relevant results correctly classified by ML/RL algorithm. Recall helps measure how many positive samples were correctly classified by the model.

Visual observation extends ML/RL toolkits to allow both novice and expert game developers to quickly and easily build and deploy highly accurate and explainable ML/RL models for agents in games using image-based data. Observation gathers data through visual or technological means. Visual observation is ‘direct’ allowing game developers to witness the agents’ behaviors firsthand in their environment.