---
title: LLM rules
description: Rules written in prose and judged by a model, file by file or as a set.
sidebar:
  order: 3
---

A `standard` rule states a standard in prose. `lawbook check` sends each
selected file to a model with that text and asks for a decision: the
probability that the file meets the standard, and why. A file whose
probability falls below the rule's `threshold` (default `0.5`) is a finding.

```yaml
version: 1
llm:
  provider: claude-code
rules:
  - id: errors-are-actionable
    description: Users can act on every error
    files: ["src/**/*.ts"]
    standard: |
      Every error message shown to a user says what went wrong and what to
      do next. Internal assertions that users never see are exempt.
```

```txt
FAIL errors-are-actionable
  src/cli.ts: "unexpected state" on line 41 says what broke but not what to do. (noul 0.12)

0 passed, 1 failed, 0 warned, 0 errored, 0 skipped
```

## What the model sees

For each file, one request carries a fixed system prompt, the standard, the
file's path, and its full content; under `--changed-lines` the content comes
with its lines numbered and the changed ones named, see [Changed
lines](#changed-lines). The model answers with structured JSON,
`{ "noul": number, "reason": string }`, plus a `line` when the lines are
numbered and one shows the shortfall, so there is nothing to parse from
prose. The system prompt tells the model to judge only what the standard
says, to treat files the standard does not apply to as meeting it, to
calibrate the number so that `0.5` means the file gives no way to tell, and
to cite its evidence.

Files are judged up to `llm.concurrency` at a time (default `4`, see
[Providers](/lawbook/providers)), each with no context from other files, and the
results are reported in path order whatever order the model answers in.
Write standards that one file can satisfy on its own.

## Decisions

lawbook records each answer as a decision in the
[Jev decision schema](https://docs.typesafe.ai/api). A `standard` rule is a
yes/no question, which Jev calls a *noul*:

```json
{ "type": "noul", "noul": 0.12 }
```

`noul` is the probability, from `0` to `1`, that the file meets the standard.
There is no separate confidence: the probability already says how sure the
model is, and `0.5` means it cannot tell.

JSON output lists every judged file's decision under `decisions`, and each
failing finding carries its own `decision`. The model's reason is kept for
findings; pass `--explain` to keep it for passing files too, which is the
quickest way to see why a borderline file landed where it did while a
standard's wording settles. See [Commands](/lawbook/commands#json-output).

## Threshold

`threshold` sets the probability a file must reach to pass. It ranges from
`0` to `1` and defaults to `0.5`, so by default a file the model cannot judge
either way passes.

```yaml
rules:
  - id: errors-are-actionable
    files: ["src/**/*.ts"]
    threshold: 0.8
    standard: Every error message says what went wrong and what to do next.
```

Raise it for standards that must be met beyond doubt; lower it to flag only
clear violations. A `threshold` of `1` fails anything but a certain pass.
While the wording and threshold settle, give the rule `level: warn` so its
findings print without failing the run (see
[Configuration](/lawbook/configuration#rules)).

## Writing a good standard

- Say what the file must do, not what reviewers dislike. "Every exported
  function has a doc comment that states what it returns" is checkable;
  "code is well documented" is not.
- Name the exemptions. Without them the model has to guess.
- Keep `files` narrow, and use `exclude` for fixtures and generated code.
  Every selected file but an empty one is one model request, so a `**/*`
  glob on a large repository is slow and costly. `lawbook check --dry-run` lists the files
  and counts the requests before you pay for them.
- Run `lawbook check --only <id>` while you refine the wording, and pin the
  rest of the suite with `--no-llm`.

## Testing a standard

Writing a standard is writing a prompt, and a prompt wants a test. Give a
rule `fixtures`: files the standard must pass and files it must fail. Then
`lawbook test` judges each one and reports the ones the model put on the
wrong side of the threshold:

```yaml
rules:
  - id: errors-are-actionable
    files: ["src/**/*.ts"]
    standard: Every error message says what went wrong and what to do next.
    fixtures:
      pass: ["fixtures/errors/good.ts"]
      fail: ["fixtures/errors/vague.ts", "fixtures/errors/silent.ts"]
```

```sh
lawbook test
```

```txt
FAIL errors-are-actionable
  fixtures/errors/vague.ts: expected fail, judged pass (noul 0.72): The messages name the failing step.

3 fixtures, 1 misclassified
```

The reason shows what the model read into the standard, which is usually the
exemption it needs spelled out. Tighten the wording, raise or lower
`threshold`, and run again; verdicts are cached, so only changed fixtures
and changed standards cost a request. Keep the fixtures out of `files`, or
`exclude` them, so `check` does not judge them as code. A standard with
fixtures that still misclassify is telling you the standard, not the code,
needs work; see [Commands](/lawbook/commands#lawbook-test-root).

## Repeatability

Models can give different probabilities for the same file on different runs,
so a `standard` rule is not as repeatable as a `forbid` rule. The number makes
borderline files visible: a file that flips between `0.48` and `0.52` is
telling you the standard's wording is unclear, not that the model is. Where a
regular expression can express the standard, prefer `forbid` or `require`.
Reserve `standard` for judgment calls, and treat a failure as a prompt for a
human to look at the file.

## Judging files together

By default a `standard` rule judges each file in its own request, with no
context from other files. A rule with `scope: set` sends every selected file
in one request, each as a `File: <path>` block, and asks whether they meet
the standard together, which fits standards about consistency across a
module:

```yaml
rules:
  - id: consistent-naming
    files: ["src/models/*.ts"]
    scope: set
    standard: Every model exports one class named after its file, in PascalCase.
```

```txt
FAIL consistent-naming
  src/models/user.ts exports `UserModel` while the others match their file names. (noul 0.20)
```

A set has one verdict: the finding names no file, and the JSON result
carries it as `decision` instead of the per-file `decisions`. Suppressed
files are left out of the set and listed as skipped, and
[blank files](#empty-files) are left out unlisted. The set is sized as a
whole against `llm.maxBytes`; a set that does not fit is reported as an
`ERROR` without a request, since nothing was judged. A set with no files
passes without a request. `--dry-run` counts one request for the rule.

## Reference files

A standard often leans on a document the model has not seen: the team's
style guide, an architecture note, the list of approved loggers. `context`
names files, relative to the checked directory, that every request of the
rule carries as reference material:

```yaml
rules:
  - id: follows-style-guide
    files: ["src/**/*.ts"]
    context: ["docs/style.md"]
    standard: The file follows the conventions in the style guide.
```

The model is told to read the reference files to understand the standard and
to judge only the files in the message, never the references themselves. The
context sits with the standard in the instructions, ahead of the files, so a
CLI that caches its prompt reads it from the cache after the first file in a
rule. Context files are
read once per rule; a path that is missing, or a binary file, is a config
error with exit code `2` before any request is made. They are not counted
against `llm.maxBytes`, since the author chose them, and `--dry-run` does not
list them. A changed reference file changes every verdict's
[cache key](#verdict-cache), so the rule is judged again.

## Changed lines

`--since` and `--changed` pick the files a change touched, but a `standard`
rule still judges each of them in full, so a one-line change to an old file
is judged on every line of it, and a finding on code the change never
touched fails the run. `--changed-lines` scopes the verdict to the change:

```sh
lawbook check --since origin/main --changed-lines
```

For each selected file, lawbook takes the added and modified lines from the
same diff the selector runs: the commits since the merge base for `--since`,
the working tree against `HEAD` for `--changed`, and the working tree against
the merge base when both are given. An untracked file counts as changed in
full. The request carries the whole file with every line numbered and the
changed lines named, and the model is told to judge those lines and to read
the rest as reference material, the way it reads [reference
files](#reference-files). Its answer may name the line that shows the
shortfall. A finding whose line lies outside the change is dropped and does
not count against the rule; the text output says `2 findings outside the
change` under the rule, and JSON carries the count as `outside` on the rule
and in the summary. A finding that names no line stays.

```txt
PASS tests-describe-behavior
  1 finding outside the change

1 passed, 0 failed, 0 warned, 0 errored, 0 skipped
```

A selected file with no changed lines, such as a mode change, or one named
by `--files` that the change left alone, costs no request and is left out
the way [empty files](#empty-files) are; `--dry-run` leaves it out too. A
`scope: set` rule sends the changed files with their lines in its one
request. The ranges are part of the [cache key](#verdict-cache), so a verdict
scoped to one change never answers for another or for the whole file.
`forbid`, `require`, `exists`, and `absent` are unchanged: `forbid` already
reports one line, and `require` asks about the file as a whole.

Without `--changed` or `--since` there is no change to scope to, so
`--changed-lines` alone exits `2`. A [baseline](/lawbook/baseline) still fits a rule
adopted on an old codebase, where every file carries old findings;
`--changed-lines` answers the per-change question, whether this change
lowered the bar, without a file to maintain.

## Per-rule model

The run's `llm` section picks the model for every `standard` rule. A rule can
override it with its own `llm`, so a subtle check can go to a stronger model
while the rest keep the default:

```yaml
llm:
  provider: claude-code
rules:
  - id: has-doc-comments
    files: ["src/**/*.ts"]
    standard: Every exported function has a doc comment.
  - id: errors-are-actionable
    files: ["src/**/*.ts"]
    standard: Every error message says what went wrong and what to do next.
    llm: { model: claude-sonnet-5-5 }
```

A rule's `llm` takes `provider` and `model`. A rule that keeps the run's
provider inherits its model and replaces it only when it names one. A rule
that names another provider starts from that provider's default model
instead, since a Claude Code model id means nothing to Codex; `codex` and
`kiro` have no default, so such an override must name a `model`.
`concurrency`, `maxBytes`, and `cache` always come from the run's `llm`. The
merged settings are checked when the config loads, so a `codex` override
without a model is an error naming the rule.

Rules with the same provider and model share one judge, and the
[verdict cache](#verdict-cache) keys on the model, so the same file judged by
two models is cached twice.

## Prompt caching

The standard and any reference material come before the files in every
request, so a CLI that caches its prompt reads them from the cache after
the first file in a rule. Claude Code and Codex report what they read from
the cache, and the JSON output carries it under `usage`; Kiro meters
credits rather than tokens, so its usage shows zero.

## Verdict cache

Lawbook also caches verdicts itself. Every verdict is stored under
`node_modules/.cache/lawbook/` in the checked directory, keyed by a hash of
the model, the system prompt, the standard, the file's path, and its
content. A later run answers from the cache for any file whose key is
unchanged, without a request, and prints it as cached in the usage line.
So a rerun on an unchanged tree is free and gives the same verdicts. For a
`scope: set` rule the key covers every file in the set, in order. A lawbook
release that changes the system prompt invalidates every entry; the next run
simply asks again.

- `--no-cache` asks the model again for every file and stores nothing, so
  it also serves when the cache directory cannot be written. To forget the
  stored verdicts, delete the cache directory; the next run fills it again.
- `--cache-dir <dir>` keeps the cache elsewhere; the path is relative to the
  current directory, like `--config`.
- `llm.cache: false` in the config turns the cache off for every run.

In CI, persist the cache directory between jobs with a cache action keyed on
the lockfile or the branch; a stale entry only costs one request. A cache
file lawbook cannot write is reported as that file's error, like a provider
error, with the two flags to try.

## Budget

A `**/*` glob on a large repository can turn one run into thousands of
requests. `llm.maxRequests` caps a run: before any client is built, lawbook
counts the requests the way `--dry-run` does and stops with exit code `2`
when the count is over the cap, naming both numbers and what to do:

```yaml
llm:
  maxRequests: 200
```

```txt
error: the run would make 412 model requests, over llm.maxRequests 200; narrow files, pass --only, or raise the limit
```

The count is the plan's, so it leaves out empty files but includes files the
[verdict cache](#verdict-cache) would answer, whitespace-only files, and files
the size and suppression guards would drop; it is an upper bound on what the
run would send. `--max-requests <n>` overrides the
config for one run, and `--no-llm` makes no requests, so the cap never
applies to it.

## Large files

A `standard` rule never sends a file larger than `llm.maxBytes` (default
`131072`, see [Providers](/lawbook/providers)). Such a file is listed under the rule
as `path: skipped, <bytes> bytes over llm.maxBytes <limit>`, in text and in
JSON under `skipped`, and does not affect the rule's status, since nothing
was judged. Raise `llm.maxBytes`, which applies to every rule, when one
must see a large file, or narrow `files` so the file is not selected. Binary files are skipped by every rule
without being listed; see [Configuration](/lawbook/configuration#selecting-files).

## Empty files

A file that is empty or holds only whitespace has nothing in it to break a
standard, so a `standard` rule leaves it out without a request and without
listing it, the way binary files are left out. An empty `__init__.py` costs
nothing however many standards select it. `--dry-run` leaves empty files out
too; it tells them by size, so a file of only whitespace still appears there.
`forbid` and `require` still read such a file, so a standard that a file must
have content to meet, such as a license header, belongs in a `require` rule.

## Provider errors

When the provider rejects a request, for instance with a credential, quota,
or network error, or answers without a verdict or with a probability outside
`0` to `1`, lawbook records the error
against that file instead of stopping. The rule stops sending new requests,
lets the ones already in flight finish, and reports the rest as
`not judged after an earlier error`. The rule's status is `ERROR`, its
judged files keep their verdicts, every other rule still runs, and the run
exits `2` after printing the report:

```txt
ERROR errors-are-actionable
  src/cli.ts: "unexpected state" on line 41 says what broke but not what to do. (noul 0.12)
  src/init.ts: claude-code: Not logged in. Run claude auth login
  src/report.ts: not judged after an earlier error

0 passed, 0 failed, 0 warned, 1 errored, 0 skipped
```

An `ERROR` rule is never a pass, whatever its `level`. A CLI that is not
installed is reported the same way, on the first request.

## Skipping

`lawbook check --no-llm` reports every `standard` rule as `SKIP` without
touching a CLI, so a run on a machine with none installed still checks the
deterministic rules and exits on their result alone. A config with no
`standard` rule never touches a provider either.

Which provider answers, and how it is configured, is described in
[Providers](/lawbook/providers).