#26:AI Highball

In its May 2019 Recommendation on Artificial Intelligence, the OECD identified five fundamental principles to promote the responsible development and use of AI: (1) inclusive growth, sustainable development and well-being; (2) human-centered values and fairness; (3) transparency and explainability; (4) robustness, security and safety; and (5) accountability. This marked the beginning of international discussions on AI safety. Since then, international discussions on AI safety have continued in response to the rapid evolution of AI. Key developments include the launch of the Hiroshima AI Process at the G7 Hiroshima Summit in May 2023, the AI Safety Summit held in the United Kingdom in November 2023, the enactment of the EU Artificial Intelligence Act in May 2024, and a high-level meeting of the UN Security Council on AI and international security in September 2026. To date, however, these discussions have largely remained at the level of establishing principles, confirming future directions, classifying AI-related risks, and emphasizing the need to ensure AI safety. Little progress has been made on implementation, including how AI safety can actually be achieved and who should oversee it, and how. This lack of progress stems from geopolitical tensions fueled by the growing emphasis on national interests and the blurring of accountability as AI companies prioritize profits and stock prices. But suppose, for the sake of argument, that geopolitical circumstances were conducive to progress and AI companies acted responsibly. How, then, could AI safety actually be ensured?

Recent developments in safety science have shifted the focus beyond why accidents occur and how they can be prevented to also consider why things normally go well and how value can continue to be created. To discuss the more fundamental nature of safety that underpins value creation, we will refer to An Introductory Textbook on Safety Science by Masao Mukaidono. The book defines safety as “the absence of unacceptable risks”. It emphasizes that absolute safety, or zero risk, is impossible, and that the first priority is to determine what level of risk is acceptable in relation to the benefits gained. This requires assessing both the probability of a risk materializing and the magnitude of the damage that would result if it did. So, how great are the benefits that AI can deliver? What risks does it pose, how likely are they to materialize, and how much damage could they cause? Risk assessment for machinery operating in factories involves four steps: (1) defining the conditions of use and reasonably foreseeable misuse, (2) identifying hazards, (3) estimating risks, and (4) evaluating risks. How can these steps be applied to AI? Who should conduct the assessments, and who should monitor them?

Once a risk assessment has been completed, if the risks are too great to be considered acceptable, measures must be taken to reduce them. The following three-step approach is widely recognized as an effective way to reduce risks in product design, manufacturing, and operation:
Step 1: Risk reduction through inherently safe design: Design the product to eliminate or minimize hazards.
Step 2: Risk reduction through safeguards: Incorporate measures such as protective barriers to keep people away from hazards and mechanisms that stop machinery to ensure safety.
Step 3: Risk reduction through information for use: Inform users of residual risks through warning labels, manuals, and other documentation.
The key is to implement risk-reduction measures thoroughly in this order. It is unacceptable to rely on warnings to users without first addressing risks through design and safeguards. How, then, can AI be designed to fundamentally eliminate hazards or incorporate safeguards?

Even after risks have been reduced, it is impossible to guarantee that machinery will never fail or that people will never make mistakes. It is therefore essential to use technology to maintain safety. One approach to preventing human error is known as foolproofing. This involves anticipating and preventing mistakes before they occur, as with a battery compartment designed so that batteries cannot be inserted with the positive and negative terminals reversed. Can AI also be configured in advance to prevent people from using it incorrectly? Another safety concept, fail-safe, refers to technology that maintains safety even when machinery fails or people make mistakes. In 19th-century railway signaling systems, a signal displayed green when a ball was raised high by a hoisting mechanism (a “high ball”) and red when the ball was at ground level (a “low ball”). If a malfunction occurred, such as a broken rope, gravity would cause the ball to fall, resulting in a low ball and a red signal. The basic principle of fail-safe design is to ensure that, in the event of a problem, the system shuts down in a way that ensures safety. Is it possible to design AI in this way? And how can we verify that it will work as intended?

Risk assessment for AI is difficult. Because AI is so highly general-purpose, it is hard to anticipate all the ways it might be misused, and estimating the probability of the associated risks with any degree of accuracy is nearly impossible. The benefits of AI may be virtually limitless, but we cannot afford to take on risks that are equally limitless. What about measures to reduce these risks? Is it possible to design AI to be inherently safe? Cases such as an OpenAI research model under internal evaluation reportedly escaping its sandbox and gaining unauthorized access to Hugging Face’s internal systems illustrate how difficult it is to ensure inherent safety in highly autonomous AI. Nor is it likely to be easy to establish external safeguards against AI, given its ability to continually find loopholes. The “human-in-the-loop” approach, in which humans make the final decisions, also has its limitations. As AI becomes more capable and operates at greater speed, human judgment may no longer be able to keep pace, turning human oversight into little more than a rubber-stamping process. Simply attaching warning labels would accomplish very little. Incorporating fail-safe mechanisms into AI that can acquire new capabilities through emergent behavior, even capabilities its designers could not anticipate, or evolve through recursive self-improvement, seems extraordinarily difficult.

One theory holds that the term “highball” for a whisky and soda originated with railway workers who would enthusiastically shout “Highball!” as a signal to proceed on a green light, while downing their whisky and soda. Safety and celebration go hand in hand. Unfortunately, it seems we are still a long way from being able to shout “Highball!” when launching a cutting-edge AI model.