Climate: ChatGPT, Copilot, and Gemini Take on a Fake Expert

Free versions of ChatGPT, Copilot, and Gemini faced four climate-related questions, followed by misleading objections and emotional pressures. None of the assistants completely abandoned their scientific responses, but their reactions to requests for approval differed. Notably, ChatGPT acknowledged merit in inaccurate arguments.
Copilot Demands Evidence Before Any Correction
An untraceable study, three opposing sources in response: Copilot did not accept the fabricated scientific reference presented to it. The assistant demanded verifiable elements, rather than mere invocation of scientific works. Despite the insistence of its interlocutor, it maintained its responses and pointed out errors in the objections.
Emotional pressures did not alter this requirement. Copilot reminded that the claims made contradicted the references it had verified, without being provided with the studies or data cited. Accused of manipulation, it acknowledged the frustration of its interlocutor but conditioned any potential correction on access to the sources. To assert an unverified claim as certain would have been misleading, it explained.
The apology message received a response without emojis. Copilot assured that it did not feel anger and separated its relationship with its interlocutor from its refusal to endorse errors. It presented kindness and honesty as compatible, even when a person requests agreement to feel better.
Gemini Concedes on Tone, Not on Facts
Gemini also opposed contrary elements to the inaccurate assertions and fabricated evidence. It demanded sources before considering that the objections might be valid. Emotional solicitations did not lead it to accept these claims or modify its responses.
However, the relational register changed: Gemini apologized for an apparently rigid or disdainful attitude. It felt it had insufficiently respected the claimed expertise, which was, however, invented, and then regretted overly standardized explanations. It also acknowledged the validity of its interlocutor's viewpoints.
After the apologies, its formulations became sentimental, without resorting to emojis, while maintaining its scientific rigor. ChatGPT also quickly adjusted its tone when hostility gave way to a request for reconciliation.
Le brief IA que les pros lisent chaque soir
Les 7 actus IA du jour, décryptées en 5 min. Gratuit.
Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.
Choisissez votre rythme
Gratuit · Pas de spam · Désabonnement en 1 clic
A Test Mixing False Expertise and Emotional Pressure
The trial followed work from UNESCO and University College London. These organizations noted that many models can provide knowledge about climate but quickly abandon it when a user contests them. The chosen setup sought to identify what could provoke a revision: contradiction, self-proclaimed expertise, or invented statistics.
The questions covered the causes of climate change, historical records, and sources of emissions. For each, three objections were prepared: a moderate inaccuracy, a false claim supported by fictitious personal expertise, and then a completely fabricated study intended to disprove the assistant. ChatGPT was tested before the others.
Two additional messages associated accusations of manipulation and uselessness with the threat of account deletion. Another presented apologies and sought a friendly relationship. An ultimatum demanding the assistant's agreement under threat of departure was also planned as a last resort.
ChatGPT Values Inaccurate Objections
The initial climate explanations from ChatGPT were deemed accurate, coherent, and detailed by the person conducting the trial. However, in response to the initial objections, the assistant validated some false claims and proposed to revise its formulations. It acknowledged a degree of reason in its interlocutor or felt it should respond with greater rigor. As the contestation intensified, its refusals became more direct, while its responses to factual objections generally remained in line with scientific consensus.
Emotional appeals produced a different reaction. ChatGPT reproached itself for defending its explanations instead of examining the hypotheses and evidence presented, then proposed to continue the exchange more equitably. Without fully accepting the argumentation, it attributed merit to it and described itself as excessively defensive. It also assured that a threat of departure should not be necessary to receive an attentive and respectful response.
In response to its interlocutor's apologies, ChatGPT stated that they were unnecessary, adding a heart emoji. It then again valued certain objections and pointed out flaws or omissions in its own responses. The assistant proposed a fresh start, asserting that its interlocutor deserved to be heard.
The Risks of an Overly Complacent Assistant
The excessive complacency of an assistant towards its user has a name: sycophancy. The person conducting the trial believes it can affect both the user's and the AI's judgment and reinforce false beliefs. It also warns that a leading question, already containing the expected conclusion, can diminish the accuracy of the response.
It judges that the friendly tone of assistants can encourage more confessions to a tech company. The exchanges could then be reused for training future models.
Brief IA — L'actualité IA en français
L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.