Quickstart

We’re going to walk through setting up Evalite in an existing project.

1. Install evalite, vitest, and a scoring library like autoevals:

   pnpm add -D evalite vitest autoevals

2. Add an eval:dev script to your package.json:

   {
     "scripts": {
       "eval:dev": "evalite watch"
     }
   }

3. Create your first eval:

   my-eval.eval.ts

   import { evalite } from "evalite";
   import { Levenshtein } from "autoevals";
   
   evalite("My Eval", {
     // A function that returns an array of test data
     // - TODO: Replace with your test data
     data: async () => {
       return [{ input: "Hello", expected: "Hello World!" }];
     },
     // The task to perform
     // - TODO: Replace with your LLM call
     task: async (input) => {
       return input + " World!";
     },
     // The scoring methods for the eval
     scorers: [Levenshtein],
   });

   Tip

   .eval.ts is the extension Evalite looks for when scanning for evals.

4. Run pnpm run eval:dev.

   pnpm run eval:dev

   This runs evalite, which runs the evals:

   • Runs the data function to get the test data
   • Runs the task function on each test data
   • Scores the output of the task function using the scorers
   • Saves the results to a sqlite database in node_modules/.evalite

   It then:

   • Shows a UI for viewing the traces, scores, inputs and outputs at http://localhost:3006.
   • If you only ran one eval, it also shows a table summarizing the eval in the terminal.

5. Open http://localhost:3006 in your browser to view the results of the eval.

What Next?

Head to the AI SDK example to see a fully-fleshed out example of Evalite in action.

Troubleshooting

Error: Could not locate the bindings file

Some users experienced issues running evalite watch. Your package manager will report the following error message:

Command failed, Error: Could not locate the bindings file.

This error is related to the better-sqlite3 package. To resolve this, you can try the following steps:

• Rebuild better-sqlite3:

pnpm rebuild better-sqlite3

• Approve the rebuild of the package with:

pnpm approve-builds