Quickstart
We’re going to walk through setting up Evalite in an existing project.
-
Install
evalite
,vitest
, and a scoring library likeautoevals
:Terminal window pnpm add -D evalite vitest autoevals -
Add an
eval:dev
script to your package.json:{"scripts": {"eval:dev": "evalite watch"}} -
Create your first eval:
my-eval.eval.ts import { evalite } from "evalite";import { Levenshtein } from "autoevals";evalite("My Eval", {// A function that returns an array of test data// - TODO: Replace with your test datadata: async () => {return [{ input: "Hello", expected: "Hello World!" }];},// The task to perform// - TODO: Replace with your LLM calltask: async (input) => {return input + " World!";},// The scoring methods for the evalscorers: [Levenshtein],}); -
Run
pnpm run eval:dev
.Terminal window pnpm run eval:devThis runs
evalite
, which runs the evals:- Runs the
data
function to get the test data - Runs the
task
function on each test data - Scores the output of the
task
function using thescorers
- Saves the results to a sqlite database in
node_modules/.evalite
It then:
- Shows a UI for viewing the traces, scores, inputs and outputs at http://localhost:3006.
- If you only ran one eval, it also shows a table summarizing the eval in the terminal.
- Runs the
-
Open http://localhost:3006 in your browser to view the results of the eval.
What Next?
Head to the AI SDK example to see a fully-fleshed out example of Evalite in action.