Skip to content
Evalite

Scorers

Scorers are used to score the output of your LLM call.

Autoevals is a great library of scorers to get you started.

You can create your own using createScorer:

import { createScorer } from "evalite";


const containsParis = createScorer<string>({
  name: "Contains Paris",
  description: "Checks if the output contains the word 'Paris'.",
  score: ({ output }) => {
    return output.includes("Paris") ? 1 : 0;
  },
});


evalite("My Eval", {
  data: async () => {
    return [{ input: "Hello" }];
  },
  task: async (input) => {
    return input + " World!";
  },
  scorers: [containsParis],
});

The name and description of the scorer will be displayed in the Evalite UI.

Score Properties

The score function receives three properties on the object passed:

import { createScorer } from "evalite";


const containsParis = createScorer({
  name: "Contains Paris",
  description: "Checks if the output contains the word 'Paris'.",
  score: ({ input, output, expected }) => {
    // input comes from `data`
    // expected also comes from `data`
    // output is the output of `task`
    return output.includes("Paris") ? 1 : 0;
  },
});

Type Arguments

In TypeScript, createScorer takes two type arguments, the input and output type:

import { createScorer } from "evalite";


const containsParis = createScorer<
  string, // Input type
  string // Output type
>({
  name: "Contains Paris",
  description: "Checks if the output contains the word 'Paris'.",
  score: (output) => {
    // output is typed as string!
    return output.includes("Paris") ? 1 : 0;
  },
});

Scorer Metadata

You can provide metadata along with your custom scorer:

import { createScorer } from "evalite";


const containsParis = createScorer<string>({
  name: "Contains Paris",
  description: "Checks if the output contains the word 'Paris'.",
  score: (output) => {
    return {
      score: output.includes("Paris") ? 1 : 0,
      metadata: {
        // Can be anything!
      },
    };
  },
});

This will be visible along with the score in the Evalite UI.