How AI Can Check Its Own Work for Errors

Software that is mathematically proven to be safe could soon become a routine part of development. A new approach uses an AI model to automatically check for and fix errors in code. This could reduce the need for human reviewers to spend time on complex proof generation. The process is designed to be fully autonomous, although the tools and foundations were built by humans.

The first point is that AI can now generate formal proofs of correctness. A team used Claude to prove that existing software code is correct. The model produced about 180 million lines of proof code in a few weeks. This was for 123 open-source libraries. The process involved translating crash tests into mathematical problems and analyzing compiled machine code.

The second point is that this method can find bugs that other methods miss. While generating proofs, Claude found hundreds of bugs. Many of these bugs would have been missed by conventional testing. This is because they only appear under very specific conditions. For example, a large input file or a particular sequence of accesses could trigger them.

The third point is that there are strict limits on this kind of verification. The proofs only hold under specific conditions. These include a certain processor architecture, single-threaded programs, and inputs under 64 MB. In real-world use, libraries run on different hardware and in different configurations. This means the proofs do not guarantee that software is error-free in all situations.

Proving that software runs without errors under all circumstances has always been very expensive. A team of specialists needed 20 person-years to formally verify a small operating system kernel. Because of this high cost, the method was reserved for extreme cases. One example is software for Airbus avionics. The new approach from Anthropic aims to make this kind of verification as routine as standard testing.

The project shows how AI can be used to write code and prove it is safe. This shifts the question from whether AI can program to whether it can prove its code is safe. The compute cost for this is high, with individual proofs running up hundreds of millions of tokens. However, if AI can verify its own code, developers can trust it more. This saves time and cost on the entire coding process.

The use of AI for formal verification could make it a common practice. Anthropic expects this to become as practical as standard testing. The roadmap for Theorem, a partner in the project, already targets critical infrastructure. This includes glibc, OpenSSL, and cURL. The ability to write and prove code is a major step for software automation.

The outlook for this technology is that it could have a major impact on software development. If AI can write code and guarantee its correctness, the entire process could change. The project strengthens the case for software automation. The ability to verify code is a key part of this process. This could lead to a major leap in what AI can do.