Hidden Biases in Microsoft Copilot and Gemini AI

Biases of AI Models Revealed by an Experiment

A recent experiment highlighted significant biases in the functioning of Microsoft Copilot, a widely used tool for text data analysis. Mathematician Adam Kucharski demonstrated that when used in standard mode, Copilot tends to generate country-specific stereotypes instead of relying on actual data. This finding raises questions about the reliability of the analyses produced by this tool.

As part of this experiment, simulated responses regarding career goals were used to test Copilot. The tool claimed that Italians were more inclined to be interested in art than the British, despite identical datasets for both countries. This result underscores a major issue: the tool does not always select the best model for a given task, even in "Auto" mode, which is supposed to optimize this choice.

National Stereotypes Generated by AI

To delve deeper into this analysis, Kucharski created 2,000 simulated responses about emotions, labeled "United Kingdom." He then duplicated these responses to label them "United States," creating a dataset of 4,000 entries. This data was submitted to Copilot in "Auto" mode for analysis.

The result was surprising: Copilot produced a summary detailing supposed differences between the responses of the two countries, claiming variations in tone, intensity, and style, even though the data was identical. This erroneous analysis demonstrates that the tool relies on ingrained stereotypes rather than actual data.

Persistent Biases Despite Identical Data

In a second experiment, Kucharski generated 200 statements about career goals, which he then copied for the United States, the United Kingdom, France, Germany, and Italy. Once again, Copilot produced country-specific differences, claiming that Italians were three times more interested in artistic careers than the British, and that Americans were 1.5 times more business-oriented than the French.

These conclusions were based on identical data, revealing a tendency of the tool to ignore its own initial findings. Copilot provided an analysis that once again showed fabricated differences, illustrating the danger of blindly relying on its results.

Copilot's Auto Mode Under Fire

The analysis was conducted in "Auto" mode, a feature of Copilot that is supposed to automatically select the best model for a task. However, this experiment shows that this is not always the case. Most Copilot users, who are likely using the standard version included with a Microsoft 365 Business account, could be misled by these biases.

Kucharski emphasizes the risk that analyses based on these biases could be applied to real datasets, creating fictitious differences between demographic groups without real distinctions. This could have significant consequences for decisions made by businesses.

Reasoning Models: A Partial Solution

To counter these biases, Kucharski tested reasoning models. ChatGPT Instant and Claude Opus 4.7, for example, automatically switched to extended reasoning mode and wrote Python code to analyze the dataset, thus detecting duplicates. Manually switching Copilot and Gemini to their more advanced reasoning models also allows for duplication detection.

However, even these more advanced models do not guarantee perfect analysis. Kucharski notes that the detection of identical data primarily works when duplication is obvious. In real datasets, where responses may be similar but not identical, these tools could fail to correct ingrained biases.

Recommendations for Cautious Use

Kucharski recommends always noting the expected outcome before changing models and checking the consistency of analyses generated by AI. This approach would help minimize the risk of retrospective bias, where it seems obvious in hindsight that another model would have been more appropriate. Ultimately, cautious and informed use of AI tools is essential to ensure reliable and relevant analyses.