Our Approach to AI Safety at SafeAI
I started SafeAI more than seven years ago to accelerate the transition to an autonomous site. We knew AI would play a critical role in driving autonomous equipment at the work site, where safety is paramount, and this is why the name of our company, SafeAI.
AI safety is a complex and multifaceted challenge that requires a thorough and systematic approach. At SafeAI, we recognize that achieving robust AI safety is not a one-time effort but an ongoing process that involves continuous learning, monitoring, and adaptation.
This article will delve into the specific strategies and methodologies we employ to maintain safety standards in our AI systems.
From defining clear operational boundaries to incorporating domain-specific expertise and implementing rigorous safety checks, our approach is designed to create AI solutions that are not only effective but also reliable and safe. While we deploy our strategy in autonomous driving applications, the ideas and methodology apply broadly to AI applications, and safety is critical. We are excited about the recent advancement of AI and want to engage with the broader AI community to ensure AI safety in various applications.
Defining Boundaries for AI Output
For much of the article, we’ll review our approach to AI safety related to off-road autonomous vehicle operation. Examples of autonomous operations include autonomous driving or navigation along a path, loading and dumping operations, driving, compacting and rolling, spreading dirt, or excavating. These autonomous operations are configured to receive input from sensor data and other inputs, such as the map of the environment. Based on these inputs and the desired goal, the autonomous vehicle is designed to generate a plan of action to achieve the goal safely.
The autonomous solution may deploy various AI-based techniques as part of the processing to reach the end goal. For instance, a Convolutional Neural Network (CNN) or Transformer mode could identify objects in the path captured by one or more sensors. Building and using an AI model, such as one based on supervised learning, generally involves a training phase in which the AI model is built and trained using a training dataset and an inference phase during which the trained AI model is used to make inferences or predictions based upon real-time data (also referred to as inferring data or dataset).
We discussed in our previous articles that AI models could sometimes make unpredictable errors in their predictions during the inference phase. One factor for this is that the inferring dataset for which the AI model is predicting at the time of inference is different from the training dataset (e.g., the data used to train and/or validate the AI model during the training phase), resulting in suboptimal performance of the model at the time of the inference. To address this, we introduced the need for a multi-layered approach to AI safety, including domain-specific checks, robust validation mechanisms, and integrated safety modules to increase the reliability of AI systems in safety-critical contexts.
This article will share SafeAI’s validation mechanisms, including two key concepts: a Vehicle Safety Manager (VSM) and an Action Integrity Module (AIM).
A Vehicle Safety Manager approach to rate AI model prediction confidence
To account for the reliability problems of deploying AI models in unstructured environments, our system deploys the concept of a Vehicle Safety Manager (VSM). The VSM is configured to automatically control that the autonomous operations are being performed safely by checking how statistically similar (or dissimilar) an inferring data point is to the distribution of the training dataset.
The VSM generates a model confidence score that indicates how similar or dissimilar the inferring data point is to the training dataset. For example, a score indicative of a high degree of similarity may be generated where the inferring data point is similar to the training data set. Alternatively, a score indicative of a low degree of similarity may be generated where the inferring data point is different from or dissimilar to the training data set. The confidence score acts as a sanity check that measures how much the prediction made by the AI model for an inferring data point is to be trusted.
The autonomous vehicle system uses this confidence score to decide how the prediction made by the AI model is to be used. For example, in instances where the score for a certain inferring data point is low, which indicates a high measure of dissimilarity, the prediction made by the AI model based upon that inferring data point may be overridden or not used by the autonomous vehicle.
This improves the safety of autonomous vehicle operations by providing a boundary to the possible solution based on the degree of closeness to the training dataset.
Furthermore, suppose the inferring data point and the training data are dissimilar, as represented by the score. In that case, the inferring data point may be added to the training data to create updated data upon additional scrutiny and validation checks. The model may then be retrained using the updated training data.
This check can confirm that the underlying data is sufficiently similar to the training dataset. The Uncertainty Check can then flag the input data as belonging to the training data set distribution.
The observable state is developed from the suite of sensors of the Autonomous Kit, which is fed into our software perception module. This module implements object detection, object classification, object tracking, local world management, map information management, etc. For example, the perception module can generate a consistent internal map based on sensor data from the sensors. The consistent internal map can be a localized map of sensed entities/objects around the autonomous vehicle. These sensed entities/objects can be mapped in three dimensions (3D).
.
An autonomous kit consisting of several sensors and a computing unit is installed on the customer-provided vehicle to enable the vehicle to collect and process environmental data.
Our Reinforcement Learning patent, granted in September 2018, describes our framework of robust and safe reinforcement learning applications in real-world autonomous vehicle applications in more detail. The method includes obtaining an autonomous vehicle's observed state and generating an interruptible command based on the Safety Sanity Index (SSI) that monitors the safety performance of the RL model and the observed state.
Ensuring Domain-specific AI models with an Action Integrity Module (AIM)
The second concept introduced in this Patent is the Action Integrity Module (AIM). This ensures that AI-generated outputs comply with established safety guidelines and regulations within the mining domain.
This module continuously monitors and evaluates AI-generated outputs in real-time, cross-referencing them with established safety guidelines and regulations. AIM decides if the final action is okay to send to the environment; if not, it sends an overriding final action.
This ongoing scrutiny ensures that AI-driven operations remain within the prescribed safety parameters, mitigating risks and maintaining operational integrity. For instance, if the model's output were to accelerate vehicle speed above 50km/h, the AIM would identify this as violating the site norms and disregard the command.
Ensuring Safety in AI systems
This article outlined specific details of SafeAI’s multi-layered approach to AI Safety, which involves boundaries to the AI output with a safety sanity check and domain-specific knowledge, including norms and regulations that should be complied with. This framework demonstrates how it can provide robust and safe handling of AI model decision-making in real autonomous vehicle applications.
Our next article in this series will explore actionable steps companies can take to enhance their own AI safety. These include assembling interdisciplinary teams, collaborating with reputable vendors, and implementing rigorous testing and validation processes.
