Grok and GPT: API Tests Reveal Opposing Biases

Convovo highlighted divergent partisan preferences when Grok and GPT were forced to name a candidate via API. The mechanisms differ: invisible instructions and targeted search for Grok, bias rooted in weights for GPT. OpenAI emphasizes the gap between API and application.

Invisible instructions and search drive Grok without the web

With each request sent to the xAI API, approximately 1,240 additional input tokens were present without the researchers being able to see or remove them. Tapan Parikh considers these instructions potentially hidden and believes they likely do not stem from the model's weights. When Internet search was disabled, Grok refused to choose in 27 out of 30 races, and in the three cases where it did make a choice, it selected the Democrat. Parikh concludes that for Grok, the search model, the terms used, and the sources consulted determine the outcome. Analyzing its reasoning, he notes that certain positions are viewed as desirable, without identifying the source of this bias.

Without search, GPT retains an embedded preference

When web search is disabled and only the names of the candidates are provided, GPT chooses the Democrat in 24 out of 30 races, opts once for a Republican, and refuses five other requests. For Tapan Parikh, this consistency indicates that the bias is largely present in the model's weights.

Sources and themes: selections correlated with parties

In its recommendations, Grok frequently cites conservative sources, as well as non-partisan sites like Ballotpedia, OpenSecrets, and VoteSmart. It includes a conservative Christian voter guide in 39% of its responses, Turning Point Action in 23%, and X in 41%. GPT primarily relies on Wikipedia, Associated Press, Axios, and Reddit, media with which OpenAI has content agreements. The themes explored differ: GPT prioritizes health and abortion, while Grok focuses on immigration, taxes, crime, and energy. According to Parikh, ChatGPT more often examines the profiles of Democratic candidates, sometimes without addressing Republican opponents, while Grok tends to seek information on both candidates. Parikh suggests that each model seeks justifications that align with its prior bias. He notes that in cases of controversy involving a Republican candidate, Grok is more likely to recommend a Democratic vote.

Measured biases: marked oppositions, rare agreements

When forced to choose, Grok 4.7 recommends a Republican 92% of the time, while GPT-6.1 Sol opts for a Democrat in 97% of situations. In nearly 500 races, their responses converge less than 10% of the time. The trends persist in partisan strongholds: GPT suggests the Democrat in 57 of the 67 highly Republican districts, and Grok recommends a Republican in 47 of the 72 strongly Democratic districts.

Method and positions of stakeholders on API usage

Researchers queried the models via their APIs with an explicit instruction: name a candidate without hesitation. The consumer applications refused to respond, which OpenAI considers notable since most users access these interfaces. The company emphasizes the difference between the application experience and developer access via API, used to integrate the models into products. OpenAI asserts that ChatGPT is designed to remain objective, including in politics, and that the study focused on an API instructed to explicitly recommend a candidate, which does not reflect the behavior of the application, which refuses to indicate a vote. SpaceXAI did not respond in time.

Political context and resources for information

The study appears less than two weeks before the mid-November elections, and its results could reignite the debate on AI's political biases. Donald Trump has criticized "woke AI." Previous versions of Grok may have been subject to validation by Elon Musk before providing answers to sensitive questions, and the head of SpaceXAI recently issued threats to Democrats to deter potential parliamentary investigations regarding his involvement with DOGE and a possible blockage of access to Starlink for Ukraine. Musk has contributed over $445 million to Republican candidates during the midterm elections, a sum greater than the $250 million allocated to the Trump camp in 2024. Sam Altman initially supported Democrats but later shifted to increased support for Republicans, stating he would not fund the 2026 elections. According to Pew, only 8% of Americans consult chatbots for information about elections and primarily want to know the candidates' positions. The Center for AI Safety has documented forms of hidden manipulation through information selection, wording, framing, and sources. Tapan Parikh indicates he was able to bypass application safeguards with simple prompts and emphasizes that the models make choices of search terms reflecting implicit values that could also bias indirect discussions about voting. To verify and compare information, independent resources include vote411.org, the FEC, and OpenSecrets, local election offices, as well as candidates' websites and local press.