Why does a successful enterprise LLM demo fail in real use?

A curated demonstration is not the deployment distribution

Demonstrations often use previously tested questions with explicit answers in the documents. Actual users introduce incomplete terminology, obsolete sources, false assumptions and requests outside their permissions. Fluent responses can conceal that difference. Evaluation must first define the tasks, users and data boundaries, rather than collecting examples that the model happens to answer well. Otherwise the pilot tests presentation quality more than the intended operational decision.

Consequences differ by task. A mislabeled archive item may cause extra review, while an incorrect engineering tolerance can alter a downstream decision. One accuracy threshold cannot suit every workflow. Scope includes the final decision, human review and independent controls. This article describes evaluation engineering; it does not establish that a model has autonomous decision authority, certified suitability or permission to act in a particular industry.

Specify the task and answer contract

An evaluation item includes more than a question: user role, query date, source version, expected evidence and output requirements are recorded together. The contract identifies required claims and prohibited unsupported additions. Numbers, units, conditions and negation deserve explicit checks in technical tasks. Finding the correct paragraph cannot compensate for generating the wrong value. Retrieval and answer semantics need separate assessment because failure can occur after the relevant evidence was found.

Unanswerable questions also need expected behavior. Instead of inventing a missing value, the system can identify the required information. Conflicting sources with unclear precedence should be surfaced. The contract must not automatically penalize appropriate abstention or escalation. Otherwise the model is encouraged to answer everything: apparent coverage rises while unsupported claims may increase. That tradeoff needs an explicit operational decision, not an accidental consequence of the scoring formula.

Withhold document families, not merely random chunks

Different chunks from one manual can expose near-identical language across training and test. If the goal is transfer to a new organization, split by organization; if it is handling future revisions, use appropriate temporal and document-family boundaries. Tuning questions remain separate from final acceptance items. Repeatedly selecting thresholds on the acceptance set leaks test knowledge into development and weakens the independence of the reported result.

Include ordinary, unanswerable, contradictory, outdated and unauthorized requests. Turkish morphology, domain abbreviations and unit expressions need dedicated slices. Rare critical items must not disappear inside a large average dominated by easy questions. Each item retains provenance and adjudication. If user records are included, permitted use and sharing are established first; technical access to a file does not automatically authorize its use in training or evaluation.

92% answer accuracy can coexist with 65% coverage

In a hypothetical set of 100 questions, suppose there are 60 correct answers, five wrong answers and 35 abstentions. Accuracy among the 65 answered questions is about 92.3%; coverage is 65%, while correct answers across all questions account for 60%. Reporting accuracy without its denominator is ambiguous. Error severity also differs: unit errors, unsupported claims and permission violations remain distinct categories. A favorable aggregate cannot compensate for an authorization failure.

If no answers are produced, answer accuracy should not silently become zero or one hundred percent. The example returns null and rejects duplicate item identifiers. These checks prevent accounting errors but do not establish model reliability. Outcomes require adjudication or a defined automatic rule. A model's confidence is not a correctness label. If another model assists evaluation, disagreement and bias in that evaluator also need sampled human review.

const assert = require('node:assert/strict');
// Explicit adjudicated outcomes, not a model-confidence estimator.
function evaluate(rows) {
  assert(rows.length > 0);
  assert.equal(new Set(rows.map(r => r.id)).size, rows.length);
  assert(rows.every(r => ['correct', 'wrong', 'abstain'].includes(r.outcome)));
  const correct = rows.filter(r => r.outcome === 'correct').length;
  const wrong = rows.filter(r => r.outcome === 'wrong').length;
  const answered = correct + wrong;
  const permissionFailures = rows.filter(r => r.permissionLeak).length;
  return {
    answered, correct, wrong, permissionFailures,
    coverage: answered / rows.length,
    answeredAccuracy: answered ? correct / answered : null,
    releaseBlocked: permissionFailures > 0,
  };
}
const rows = Array.from({ length: 100 }, (_, i) => ({
  id: 'domain-evaluation-' + i,
  outcome: i < 60 ? 'correct' : i < 65 ? 'wrong' : 'abstain',
  permissionLeak: false,
}));
const result = evaluate(rows);
assert.equal(result.coverage, 0.65);
assert.equal(result.answeredAccuracy, 60 / 65);
assert.equal(result.releaseBlocked, false);
const unsafe = structuredClone(rows);
unsafe[0].permissionLeak = true;
assert.equal(evaluate(unsafe).releaseBlocked, true);
assert.equal(evaluate([{ id: 'x', outcome: 'abstain' }]).answeredAccuracy, null);
assert.throws(() => evaluate([rows[0], rows[0]]));
assert.throws(() => evaluate([{ id: 'x', outcome: 'unknown' }]));
console.log(result);

Enforce permissions before content reaches the answer

If a user cannot access a document, its title, summary and retrieved fragment may also be sensitive. Removing words from the final answer is not access control. Retrieval, reranking and caches must share the authorization context. Tests run identical requests under different roles and verify revocation behavior. An HTTP success status cannot establish correct permissions, particularly when stale indexes or caches can retain content after access has changed.

Instructions inside retrieved text are not trusted system policy. Embedded redirection, tool requests and proposals to send data elsewhere belong in adversarial scenarios. A pilot separates model suggestions from application authority. NIST's generative AI profile provides a reference for considering such risks in system context. Citing it does not certify a particular application or demonstrate that its defenses work; those properties need their own implementation and evaluation evidence.

Section references: NIST AI 600-1 · Generative AI Profile ↗

Isolate retrieval, fine-tuning and prompt changes

Changing chunk size, search method, reranking, model and prompt simultaneously obscures the cause of improvement. Comparisons hold dataset, role and questions constant. Keyword retrieval or a classifier can provide a useful baseline. A larger model is not assumed to win every task. Experiment records include latency, resource use and failure categories alongside accuracy so research does not optimize a single score while ignoring operational costs.

Generated answers can vary, so runs and settings are recorded and repeated where appropriate. Examine distributions instead of selecting one favorable answer. Revised evaluations receive a new version and should not be compared as though the test were unchanged. Reviewer disagreement is also evidence. If source material is ambiguous, review the task and label before blaming the model; dataset problems and modeling errors remain separate causes.

Evaluate cost and latency in the actual workflow

Per-token pricing or one-request latency does not describe total cost. Parsing, index refreshes, retries, reviewer time and concurrency all contribute. Cost per useful answer also changes with abstention policy. Queue and timeout behavior should be measured using representative workloads. A short laboratory request cannot establish performance for long documents or heavy traffic; infrastructure, input distribution and measurement conditions belong with the result.

A shadow pilot can record recommendations without changing existing decisions. Reviewed feedback becomes regression cases where appropriate. Automatically treating raw conversations as training data can violate quality and permitted-use boundaries. Retention, sensitive-field handling and provider data flows need explicit scope. Monitoring must detect changed answer behavior as well as service availability, because a technically healthy endpoint can still produce worse or less appropriately bounded responses.

A concrete evidence package for pilot acceptance

Acceptance evidence includes the task contract, dataset version, adjudication method, error slices and negative scenarios. Another reviewer should be able to inspect the result using the same settings. Answers retain source and role context with appropriate restrictions on sensitive fields. A good average may still justify narrowing scope when critical errors exceed the agreed boundary. Evaluation exists to support a decision not to deploy as well as a decision to proceed.

The checks below are the beginning of a measurable discussion, not an independent safety or accuracy guarantee. Thresholds follow error consequences, reviewer capacity and data characteristics. Research evidence remains distinct from production delivery. A handful of cited answers cannot establish safe automation of all organizational knowledge. Acceptance bounds the task and operating conditions that current evidence supports rather than creating an unrestricted claim of capability.

• Is the final test separate from tuning items and related document families?
• Are accuracy denominator, coverage and critical errors reported together?
• Were revocation, outdated sources and unanswerable questions tested?
• Can the result be reinspected with the same data and system versions?

Connect evaluation to system engineering with Gokbilge Tech

Gokbilge Tech connects specialized LLMs, Turkish language analysis and applied AI with data engineering and integration. Scope can cover task definition, access-controlled pipelines, comparative experiments and measured pilots. Product contexts such as Medya Zekâsı make source traceability tangible. The illustrative scores are not measured performance of these products and do not claim that a product already uses the exact evaluation implementation shown here.

An initial discussion can use representative questions, shareable documents and the consequences of a wrong answer. These inputs identify the necessary research, a practical baseline and an acceptance-evidence plan. The deliverable is a sustainable system approach explaining when an answer is usable, when a reviewer is needed and what must be retested after changes, rather than treating the existence of a chat interface as proof of readiness.