Human Oversight (HO)

This compliance category contains details on the Human Oversight mechanisms required for FDA AI/ML based SaMD.

High-risk AI systems shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which the AI system is in use.

Human oversight shall be ensured through either one or all of the following measures:

1. identified and built, when technically feasible, into the high-risk AI system by the provider before it is placed on the market or put into service;

2. identified by the provider before placing the high-risk AI system on the market or putting it into service and that are appropriate to be implemented by the user.

The human oversight provisioned by the above integration/provisioning shall enable the individuals to do the following:

1. fully understand the capacities and limitations of the high-risk AI system and be able to duly monitor its operation, so that signs of anomalies, dysfunctions and unexpected performance can be detected and addressed as soon as possible;

2. remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (‘automation bias’), in particular for high-risk AI systems used to provide information or recommendations for decisions to be taken by natural persons;

3. be able to correctly interpret the high-risk AI system’s output, taking into account in particular the characteristics of the system and the interpretation tools and methods available;

4. be able to decide, in any particular situation, not to use the high-risk AI system or otherwise disregard, override or reverse the output of the high-risk AI system ;

5. be able to intervene on the operation of the high-risk AI system or interrupt the system through a “stop” button or a similar procedure.

This compliance category covers the following principles from the FDA GMLP:

Principle 7. Focus Is Placed on the Performance of the Human-AI Team: Where the model has a “human in the loop,” human factors considerations and the human interpretability of the model outputs are addressed with emphasis on the performance of the Human-AI team, rather than just the performance of the model in isolation.

Principle 8. Testing Demonstrates Device Performance During Clinically Relevant Conditions: Statistically sound test plans are developed and executed to generate clinically relevant device performance information independently of the training data set. Considerations include the intended patient population, important subgroups, clinical environment and use by the Human-AI team, measurement inputs, and potential confounding factors.

Principle 10. Deployed Models Are Monitored for Performance and Re-training Risks Are Managed: Deployed models have the capability to be monitored in “real world” use with a focus on maintained or improved safety and performance. Additionally, when models are periodically or continually trained after deployment, there are appropriate controls in place to manage risks of overfitting, unintended bias, or degradation of the model (for example, dataset drift) that may impact the safety and performance of the model as it is used by the Human-AI team.

Below is the list of the controls that are part of this compliance category:

• HO01 - Human Oversight Mechanism

• HO02 - Human Oversight Details

• HO03 - Human Oversight - Biometric Identification Systems