Jev Citation Verifier

04 / HOW TO READ THE RESULT

Relation, probabilities, confidence, then review status

Jev returns the selected relation, its probability distribution, and a separate confidence value for this question. The probabilities describe the four relations. Only valid provider confidence drives this demo’s local review policy: at or above your chosen threshold, the verdict is marked Auto; otherwise it needs Human Review. Missing or invalid confidence also requires Human Review, even if a relation has a high probability. This threshold is an application choice, not a TypeSafe-recommended cutoff. Moving it never calls the model again.

03 / WHY VERIFY THE QUOTE FIRST

Use each tool for the job it does well

05 / REAL-WORLD PATTERN

The full system can start much earlier

The public Paper Trellis Citation Verifier demonstrates a broader workflow. It imports manuscripts and matches citation markers to references, resolves DOI, PMID, arXiv, or title against PubMed, OpenAlex, and Crossref, then tries full text or a PDF before falling back to an abstract. Claude can transcribe references and propose quotes; code checks quote location and numeric consistency; Jev judges support; a person makes the final call. This demo starts after a user has supplied the evidence.

Paper Trellis uses bring-your-own provider keys stored in the browser and a same-origin TypeSafe proxy. This demo uses Jev Playground’s server-side key through its public evaluation API; it does not request your provider key or read your mailbox or manuscripts.

Attributed example: Paper Trellis Citation Verifier on GitHub ↗ (opens in a new tab).

06 / IMPLEMENTATION

The same two-stage check in TypeScript

This original example uses the same Vercel AI Gateway integration as Jev Playground. It stops before the model when the quote is absent, asks one Choice question when it is present, and applies a local confidence threshold afterward. The distribution remains visible independently of that policy.

import { createGateway, experimental_evaluate as evaluate } from 'ai';

const gateway = createGateway({
  apiKey: process.env.AI_GATEWAY_API_KEY,
});

const normalizeQuote = (text: string) => text
  .replace(/[“”]/g, '"')
  .replace(/[‘’]/g, "'")
  .replace(/\s+/g, ' ')
  .trim();

export async function verifyCitation(
  claim: string,
  quote: string,
  source: string,
) {
  const normalizedQuote = normalizeQuote(quote);
  const quoteFound = normalizedQuote.length > 0 &&
    normalizeQuote(source).includes(normalizedQuote);

  if (!quoteFound) throw new Error('Quote not found after typography normalization');

  const result = await evaluate({
    model: gateway.evaluationModel('typesafe-ai/jev'),
    state: `CLAIM:
${claim}

VERIFIED QUOTE:
${quote}

SOURCE:
${source}`,
    questions: {
      relation: {
        type: 'choice',
        instructions: 'How does the verified evidence relate to the claim?',
        criteria: {
          SUPPORTS: 'The evidence directly supports or clearly entails the claim.',
          CONTRADICTS: 'The evidence states or clearly implies the opposite.',
          INSUFFICIENT: 'Relevant, but not enough to establish the claim.',
          UNRELATED: 'The evidence does not address the claim.',
        },
      },
    },
    maxRetries: 0,
    abortSignal: AbortSignal.timeout(30_000),
    providerOptions: { gateway: { zeroDataRetention: true } },
  });

  const answer = result.answers.relation;
  const confidence = result.providerMetadata?.typesafe?.confidence?.relation;
  const reviewThreshold = 0.8; // Your application policy, not Jev's.
  const usableConfidence = typeof confidence === 'number' &&
    Number.isFinite(confidence) && confidence >= 0 && confidence <= 1
    ? confidence : null;
  const status = usableConfidence !== null && usableConfidence >= reviewThreshold
    ? 'AUTO'
    : 'HUMAN REVIEW';

  return { relation: answer.choice, probabilities: answer.probabilities, confidence: usableConfidence, status };
}

For request contracts, production error handling, and server-side credentials, continue to the Jev API guide.

07 / LIMITATIONS

What this verification cannot prove

• The source itself is trustworthy.
• The source is current.
• The claim is universally true.
• The quote is interpreted correctly across the full document.
• An academic citation meets a journal’s standards.

The demo evaluates only the relationship between the supplied claim and supplied evidence.