Google Tests Gemini in Double-Blind AI Evaluation

A double-blind evaluation pilot relies on Google’s confidential computing to prevent prior exposure to test prompts. The system targets sensitive cases and allows third parties to assess a model without revealing either the data or the weights.

Third-party testing without access to data or weights

Double-blind evaluations enable independent organizations to test advanced models without compromising data sovereignty or security. This method is deemed particularly suitable for sensitive assessments, such as those conducted in cybersecurity or by government agencies. A cryptographic proof helps avoid reference contamination and protects sensitive information.

A trust issue related to reference contamination

If a model has previously accessed the evaluation questions, its results may be artificially inflated. To ensure that scores accurately reflect a model's capabilities and security, it is essential that the test prompts remain unknown to the model prior to evaluation. Decision-makers, researchers, and companies expect references that faithfully represent actual performance.

Double-blind and confidential computing to overcome the old compromise

High-stakes external evaluations previously required a choice between sharing test prompts or model weights, which exposed sensitive information. The double-blind approach eliminates this compromise through Google’s confidential cloud computing portfolio. A cryptographic verification ensures that the evaluation data and the model remain private to their respective owners. In this setup, the evaluator does not access the weights of the Gemini model, and Google does not see the test prompts, as the evaluations are confined within a cryptographic box and a privacy-preserving environment.

A pilot underway with Gemini Flash Lite and partners

A first double-blind evaluation of an advanced AI model is currently underway, testing Gemini Flash Lite with confidential references. The initiative brings together the AI Security Institute of Singapore, OpenMined, AVERI, and MLCommons. Google specifies that it evaluates its systems throughout development and deployment, also relying on external partners such as research labs, civil society, and AI security and safety institutes for thorough testing. Zero-logging protocols and contractual guarantees were already in place to protect test prompts; the addition of technical and cryptographic safeguards is presented as a significant advancement. This pilot aims to establish a new benchmark in model oversight to encourage safer, more reliable, and trustworthy AI systems.