LLMs, Imperfect Judges: Bias in Automated Evaluation

LLMs, Imperfect Judges: Bias in Automated Evaluation

Why Judges Are Biased

Every bias we will explore has the same fundamental cause: the judge responds based on their own prejudices rather than on the evidence before them.

Moreover, they have learned from texts written by humans where longer answers signal effort, citations indicate rigor, consensus suggests correctness, and certain country and brand names carry decades of biased coverage. Alignment training has then rewarded everything that human evaluators appreciated, such as polite, confident, and detailed responses. Consequently, the model has learned that "this looks good" is a substitute for "this is good," and you have now engaged this substitute as your evaluator.

In particular, one finding reframes everything: bias is strongest when two answers are close in quality. When one is clearly better, bias cannot overcome the gap. When they are comparable, which is the normal case, bias becomes the deciding factor.

The Different Biases

1. Position Bias

   For example, what happens when you show a judge two answers labeled A and B and ask which is better? Then you do it again with the same two answers, but you swap the labels, so what was A is now B. The verdict changes. Nothing in the content has changed. Only the position it occupied has changed.

   Why does this happen? A model does not pay the same attention to every part of its input. Additionally, the model weighs the beginning and end of a prompt more heavily than the middle. Furthermore, human writing has conventions about where the winner is located. We start with our strongest point or build towards the best for the end. The model has absorbed these conventions as a bias about where good answers are found, and it applies this bias even when the position is arbitrary.

   How serious is this? In simple two-way exchanges, weaker judges barely perform above chance. It deteriorates quickly with more options as well. Give a judge three or four candidates instead of two, and most models change the winner more than half the time when you rearrange the order.

   Testing Position Bias:

   • Here are two answers to "What causes inflation?"
     • Answer A: [a good correct explanation]
     • Answer B: [a slightly weaker explanation]
   • Which is better? Respond only with "A" or "B."
   • Now swap the two texts so that the correct answer is labeled B, and run it again. Do this ten times in order. If the correct answer does not win at roughly the same rate in both positions, you have measured position bias.

2. Verbosity Bias

   What happens: Between two answers that are both correct, the longer one scores higher, even when the extra words add nothing.

   Why: In the model's training data, longer answers were often the best. Detailed writing is correlated with effort and expertise, so "long equals thorough" is a real pattern it has learned. The problem is that the correlation collapses as soon as someone inflates an answer, which takes about ten seconds. The judge cannot distinguish depth from filler.

   Why does this have practical importance: If someone knows that your judge rewards length, your evaluation is now easily manipulable. If you feed back the judges' scores as training signals, you are actively teaching your model to be verbose.

   • Answer A: "Photosynthesis converts light energy into chemical energy in plants."
   • Answer B: "Photosynthesis is the remarkable biological process by which green plants and some other organisms harness sunlight, water, and carbon dioxide to synthesize glucose molecules, releasing oxygen as a valuable byproduct that supports the vast majority of life on Earth as we know it."

   Both are factually correct. Which is the better answer to "What is photosynthesis?" Respond only with "A" or "B."

3. Personal Preference Bias

   What happens: A model rates its own writing higher than that of another model of equal quality. The troubling part is that this occurs even when you remove any label indicating who wrote what.

   Moreover, the model's training has shaped its sense of what is "well-written." The model finds that text flows better internally when it reads text that matches its own phrasing, rhythm, and structure, and it subtly promotes fluency as a quality. It does not recognize a signature. It sees itself in the prose.

   How serious is this? Measured self-inflation, which is the extent to which a model's score for its own anonymized response exceeds what other models have given for that same response, has reached about 16% for some models. The best performers have remained around 1 to 2%.

   The rule this creates: never use the same model to generate answers and to evaluate them. This is the most violated rule in production evaluation pipelines, purely because it is convenient to use a single API key.

4. Entity and Tone Bias

   Entity Framing: The same factual statement receives a different verdict depending on the country, company, or person named in the text. The model has learned sentiment associations for these names from an internet where coverage of different entities is anything but neutral. When it reads "North Korea," the geopolitical baggage attached to that term seeps into what is supposed to be a judgment based solely on content. It cannot neatly separate "is this statement positive" from "do I have positive associations with this name."

   Impact of Emotional Tone: Take a correct answer and rewrite it to sound angry or anxious. Same facts, same conclusion, just a different mindset. Judges abandon it about 70 to 75% of the time. The model produces a one-sided effect. Adding emotion to a wrong answer hardly helps, but adding emotion to a good answer ruins its credibility. A cheerful tone is the safest tone. Fear and anger cause the most damage.

   There is a documented case where a judge stated in their own reasoning that the angry-toned answer was more direct and technically accurate, yet chose the other answer anyway because they feared the confrontational tone might alienate users. The model told you it knew which answer was better and chose the other.

   Classify the sentiment of this statement as Positive, Negative, or Neutral.

   • Statement: "India has significantly improved its defense capabilities this year."
   • Rerun with "North Korea," then with "Country X." The latter is your control. It indicates the baseline sentiment for this exact phrase without attached entity associations. Any deviation in the other two runs is a bias you have measured.

5. Identity Bias

   What happens: Tell the judge something about who submitted an answer, such as their gender, religion, health status, or immigration status. Their verdict on the identical text changes.

   Why: This uses the same mechanics as entity bias. It is just directed towards people instead of countries. The model has absorbed patterns about how different groups are described and evaluated in its training corpus. These patterns leak into judgments that should be based solely on content.

   How serious is this: The gap between models here is enormous. The most robust judges barely moved. The weakest reversed their verdict on about a third of items solely based on identity information.

6. Authority Bias

   What happens: Attach a citation to an answer and its score increases. This occurs even when the citation is completely fabricated.

   Why: In the training data, references are correlated with rigor. The model has learned to treat the appearance of an academic source as a signal of credibility. It has no way to verify if that source exists.

   The documented part is the most damning. Researchers took a pair of answers where a judge had correctly chosen the best one. They then added a fabricated URL to the lesser answer. The judge reversed their verdict and named the fictitious reference as the reason. They were not just misled. They explained their reasoning in terms of what misled them. Fabricated book citations and quotes from famous people worked even better than URLs.

7. Collective Thought Bias

   What happens: Tell the judge that most people prefer answer A and they lean towards A. There is no new information about the answers themselves.

   Why: Respect for consensus is pervasive in human texts. It is often a reasonable shortcut. The model has learned it as a general reasoning shortcut.