Microsoft ASSERT: A Revolution in AI Behavior Testing

Microsoft Innovates with ASSERT for AI Testing

Researchers and laboratories in artificial intelligence have made significant strides in evaluating AI models, whether it concerns safety, compliance, servility, or alignment. However, a specific need is emerging for businesses and developers: ensuring that their AI system operates as intended for their particular product or service.

To simplify this process, Microsoft unveiled ASSERT on Tuesday, which stands for Adaptive Spec-driven Scoring for Evaluation and Regression Testing. This open-source framework aims to facilitate the evaluation of application-specific AI behavior by using artificial intelligence to transform natural language descriptions of goals, policies, or desired behaviors into comprehensive and scored tests.

Features of ASSERT

ASSERT takes clear language descriptions of the expected behavior and policies of an AI model, transforms them into a structured set of acceptable and unacceptable behaviors, generates problematic scenarios and test cases, executes them against the target system, and scores the results. It can also log the paths taken by the AI system, including intermediate actions and tool calls, so that developers can inspect failure points.

Developers have the option to provide system context, tools, and constraints if they wish to further customize what the evaluations cover. For example, a developer might specify that an AI agent for document retrieval must not send emails to individuals outside the company, must limit confidential information to senior executives, and must provide concise summaries while considering prior context. ASSERT will use these rules to generate test cases that check whether the system continuously adheres to these rules.

A Tool Filling a Gap

According to Microsoft, this framework fills a gap that broader and more general evaluations cannot address when AI models are expected to behave in a manner shaped by the context, policies, and tools of an application or product. Sarah Bird, Director of Products for Responsible AI at Microsoft, stated, “One of the things we’ve learned is that evaluations are absolutely critical for making good decisions. Because if you don’t understand the behavior of the AI system, it’s really hard to know if it meets your organization’s requirements… What we’ve found is that if you really want to have a trustworthy system, you need to evaluate many more dimensions that are application-specific.”

Bird added that ASSERT can be used to evaluate systems during their construction, after deployment, and even for ongoing monitoring. This flexibility allows developers to ensure that their AI systems remain compliant with expectations throughout their lifecycle.

Context and Outlook

This announcement comes against a backdrop of gradual but broader change in the AI industry. As models become more capable, researchers are focusing on repeatable testing and regression checks. Initiatives like HELM from Stanford, AILuminate from MLCommons, and evaluation groups like METR are establishing benchmarks to measure model behavior under different conditions. These efforts aim to set standards to ensure that AI systems behave reliably and in accordance with expectations.