In recent months, the use of artificial intelligence tools in daily life has gained mainstream popularity. Companies like OpenAI and Google have released generative AI tools such as ChatGPT and Bard. However, behind the scenes, the technology has been advancing for years, raising questions about evaluating and securing these new AI systems. Microsoft is now unveiling details about its AI red team, which has been tasked with discovering vulnerabilities in AI platforms since 2018.
Since its formation five years ago, Microsoft’s AI red team has grown from an experiment to a fully interdisciplinary team consisting of machine learning experts, cybersecurity researchers, and social engineers. The team shares its findings within Microsoft and the tech industry, using accessible language to ensure understanding among individuals and organizations without specialized AI knowledge. However, the team recognizes that AI security differs conceptually from traditional digital defense, necessitating a unique approach to their work.
Ram Shankar Siva Kumar, the founder of Microsoft’s AI red team, explains that it took time to establish the team’s dual focus on traditional red teaming and responsible AI. The team initially focused on developing traditional security tools, collaborating with organizations like MITRE and releasing open source automation tools for AI security testing named Microsoft Counterfit. In 2021, the red team published an AI security risk assessment framework.
As the urgency of addressing machine learning flaws and failures becomes more apparent, the AI red team has continued to evolve and expand. In one early operation, the team assessed a Microsoft cloud deployment service with a machine learning component. Exploiting a flaw, the team conducted a denial of service attack on other users of the service, strategically creating virtual machines to negatively impact the performance of other customers.