How Businesses Can Measure LLM "Bullshit": Diagnosing LLM-as-a-Judge Reliability

Technical Context

The original request references arXiv:2602.07432 and a "bullshitmeter" concept. As of today (2026-02-10), this paper does not exist in arXiv—it is likely a typo or a reference to an internal document. However, the business need is clear: companies need a measurable way to assess how much they can trust an LLM that evaluates other responses (LLM-as-a-Judge)—in ranking, moderation, quality assurance, compliance, and automated scoring.

The most relevant work regarding this context is arXiv:2602.00521v1 on diagnosing LLM-as-a-Judge reliability via Item Response Theory (IRT), specifically the Graded Response Model (GRM). This is crucial: instead of saying "I think the model judges well," we gain instrumental diagnostics—like a measuring tool with calibration, sensitivity, and stability metrics.

What Exactly Are We Measuring?

When an LLM acts as a "judge," it often breaks in unexpected ways:

• Prompt Sensitivity: A slight change in instructions causes the scoring scale to drift significantly.
• Scale Abuse: The model avoids extreme values, sticks to 7/10 or 0.7, and fails to distinguish between closely related options.
• Instability on Repetition: The same case gets different scores across different runs—especially at temperature > 0.
• Hidden Criterion Shift: Instead of judging "correctness," the judge starts evaluating "similarity to a reference style."

The IRT/GRM Approach: Decomposing the Score

In educational testing, IRT separates "student ability" from "question difficulty." Applying this to LLM-as-a-Judge, we want to separate:

• Latent Quality of the Answer: (conceptually θ — what we are trying to measure);
• Task/Prompt Effect: (how much a specific formulation, scale, or context distorts the measurement);
• Scale Discrimination: (how well the judge distinguishes between quality levels).

The described framework suggests a two-phase diagnosis. A key practical point: in Phase 1, you can assess judge reliability without human ground truth—meaning it's cheap, fast, and uses your own data.

Phase 1: Intrinsic Reliability (No Humans)

Idea: Take a pool of cases and run them through the LLM judge with controlled prompt variations (rephrasing, changing criteria order, scale format, etc.). Then, GRM is fitted to determine if the judge is a stable measuring instrument.

• C_V (Intrinsic Consistency): A metric of score stability under prompt variations. If C_V is poor, the judge is not a tool but a generator of random "opinions."
• ρ (Scale Discrimination): An indicator of the scale's distinguishing power. If ρ is low, the judge cannot differentiate quality levels well, making automated ranking/control noisy.
• Item Effects: Which specific "question variants" or response formats break the judge (diagnosing the source of instability).

Practically, this means before "putting the judge in prod," you can conduct a mini-audit: does it withstand instruction changes, does the scale drift, and where are its "blind spots"?

Phase 2: Human Alignment

If the judge passes Phase 1 (is stable and discriminates the scale), it then makes sense to spend money on human eval: compare the judge's decisions with experts and measure agreement/bias. Important detail: skipping Phase 1 can lead to false conclusions about "low model quality," when the problem is actually in the prompt/scale/temperature, not the "intelligence."

Limitations and Engineering Nuances

• No Ready-Made "Button": The source mentions code, but a public repository may not exist. You will need your own pipeline implementation.
• Experiment Design Required: Prompt variations must be controlled; otherwise, you are measuring chaos.
• Temperature/Seed: For a "judge," generation parameters are usually fixed (temp=0 or close), otherwise metrics mix sampling instability with judgment instability.

Business & Automation Impact

For business, the value of a "bullshit detector" lies not in academic beauty, but in managing risks and the cost of quality. In real processes, LLM-as-a-Judge often becomes a hidden decision center: it decides what answer to show a client, what to send to an operator, what counts as a "violation," and which cases to escalate. If the judge is unstable, you are automating randomness.

Direct ROI Areas

• Generation Quality Control: Auto-checking assistant/bot responses before sending to a client (gating). A reliable judge reduces the cost of manual sampling and incidents.
• Ranking Options: Choosing the best of N generated answers. If the judge has poor scale discrimination, you pay for N generations but quality doesn't improve.
• Ticket Scoring & Routing: Determining criticality, topic, and escalation probability. A judge's error turns into an SLA problem.
• Compliance & Moderation: If the "court" depends on policy rephrasing, you face regulatory risk.

Changing AI Solution Architecture

In a mature AI solution architecture, LLM-as-a-Judge is a separate component with its own quality metrics, just like a recognition model or an ML classifier. The IRT approach disciplines the architecture:

• Separating "Model Quality" from "Prompt Quality": You can provably show that the issue lies in the prompt/scale, not the base model.
• Calibration Before Deployment: A "judge diagnostic stand" is introduced as a mandatory CI/CD step for prompts.
• Observability: Monitoring now includes stability metrics, not just accuracy on rare manual labels.

In practice, companies often try to build AI automation "fast": they take an LLM, write an evaluator prompt, put it in the pipeline, and are surprised by quality degradation a week later. The reason is usually that the judge was not verified as a measuring instrument. Professional AI implementation is distinguished by the fact that evaluator reliability is proven experimentally and fixed in processes.

Winners and Losers

• Winners: Contact centers, e-commerce, fintech, and insurance, where there are many repeatable decisions and a high cost of error; teams building employee assistants (internal copilots) with quality control.
• At Risk: Companies relying on "LLM courts" for disputed/regulated decisions without provable stability (loans, KYC/AML, legal advice, medical hints). Here, instability is not a "margin of error" but a source of liability.

Expert Opinion Vadym Nahornyi

The most expensive mistake in LLM projects is considering quality evaluation "subjective" and therefore unmeasurable. At Nahornyi AI Lab, we regularly see the same pattern: business invests in generation (RAG, tools, agency) but saves on measurement. Without measurement, you manage neither quality nor risk.

The IRT approach is important because it shifts the conversation from "is the model good/bad" to "is the tool stable/unstable and why." This is directly applicable to industrial cases:

• When changing providers/model versions, you can quickly understand if the judge got worse or if the scale just "shifted";
• Evaluator prompts can be standardized as artifacts with tests (prompt unit tests + diagnostic runs);
• Management decisions can be justified: where automation is acceptable and where a human-in-the-loop is needed.

My forecast: the hype around "LLM-as-a-Judge replacing annotators" will move into a utilitarian phase. Judges will remain but will become "controlled instruments"—with calibration, stability metrics, and update protocols. The main implementation trap is trying to use the same judge for all domains and all scales. In practice, domain calibrations, different scales for different tasks, and explicit applicability boundaries are needed.

If you need a "bullshit detector" not as a meme but as a manageable product component, at Nahornyi AI Lab we usually start with a small diagnostic sprint: we design prompt variations, collect a scoring matrix, calculate stability/discrimination, and only then decide whether to scale automation.

Theory is good, but results require practice. Want to understand if you can trust your LLM evaluator and how to safely embed it into processes? Let's discuss your case at Nahornyi AI Lab: we will design metrics, a test loop, and an AI automation implementation plan. I take responsibility for the quality and architecture — Vadym Nahornyi.