Anthropic has revealed that it identified several instances in which scientists used its AI models, particularly Claude, to assist with research that could be linked to the development of biological weapons. The company said it stopped these uses after detecting them.
In a new report, Anthropic documented five cases where its models were used in activities related to potential biological weapons development. The report also detailed the security measures that helped uncover these uses and how the company responded. The disclosure comes as advanced AI models become increasingly capable of helping researchers carry out sophisticated scientific tasks, raising the risk of misuse.
The Challenge of Distinguishing Dangerous from Legitimate Research
Anthropic emphasized that detecting this type of use is complex. Requests tied to dangerous research can resemble legitimate scientific work, such as vaccine development or the study of infectious diseases. Because of this overlap, the company said it has adopted a cautious approach that leans toward strict action when there are signs of misuse, given the potential consequences.
Documented Cases Involving Claude
Among the cases in the report, Anthropic’s biosecurity classifier flagged a request to use Claude for gain-of-function research on the chikungunya virus. According to the company, the research proposals included modifications aimed at increasing the virus’s ability to transmit and enhancing its ability to evade the immune response.
Anthropic noted that the nature of the research raised additional concerns because the proposed project was linked to a military research institute. The term “gain-of-function research” describes studies that explore how to change the characteristics of organisms or viruses. Such research can have scientific and medical applications, but it can also raise significant concerns when it involves increasing the danger of pathogens.
The company also detected another case involving gain-of-function research on avian influenza, as well as a case in which a researcher used Claude to help prepare an atlas of toxin peptides and develop a generative system aimed at improving the properties of toxins.
Misuse Beyond Biology
The cases in the report are not limited to biology. Anthropic also documented misuse of its models in other areas, including surveillance, the creation of exploit software for vulnerabilities, propaganda production, and the development of weapons systems.
Jacob Klein, Anthropic’s head of threat intelligence, said detecting harmful activity is not straightforward. He noted that real-world cases are more complex than the image of someone openly declaring a desire to develop a biological weapon.
Account Bans and Safety Improvements
According to the company, it has banned the accounts of users confirmed to have misused Claude or other Anthropic models. Findings from investigations are also used to develop protection systems and improve the models’ ability to detect and prevent dangerous uses in the future.
Anthropic declined to disclose the names of the researchers or institutions involved in the cases it investigated. It explained that the individuals concerned are scientists working in scientific research and that the company does not confirm they intended to harm others. Revealing their identities or laboratories could put them at risk.
A Growing AI Safety Dilemma
These cases reflect an increasingly important aspect of the AI safety dilemma. As models become more effective at assisting researchers and scientists, their potential to support dual-use research grows. This places pressure on AI companies to develop more precise mechanisms for balancing the enablement of legitimate scientific research with the prevention of uses that could threaten public safety.