LLM rules

Rules written in prose and judged by a model, file by file or as a set.

A standard rule states a standard in prose. lawbook check sends each selected file to a model with that text and asks for a decision: the probability that the file meets the standard, and why. A file whose probability falls below the rule’s threshold (default 0.5) is a finding.

version: 1
llm:
  provider: claude-code
rules:
  - id: errors-are-actionable
    description: Users can act on every error
    files: ["src/**/*.ts"]
    standard: |
      Every error message shown to a user says what went wrong and what to
      do next. Internal assertions that users never see are exempt.

FAIL errors-are-actionable
  src/cli.ts: "unexpected state" on line 41 says what broke but not what to do. (noul 0.12)

0 passed, 1 failed, 0 warned, 0 errored, 0 skipped

What the model sees

For each file, one request carries a fixed system prompt, the standard, the file’s path, and its full content; under --changed-lines the content comes with its lines numbered and the changed ones named, see Changed lines. The model answers with structured JSON, { "noul": number, "reason": string }, plus a line when the lines are numbered and one shows the shortfall, so there is nothing to parse from prose. The system prompt tells the model to judge only what the standard says, to treat files the standard does not apply to as meeting it, to calibrate the number so that 0.5 means the file gives no way to tell, and to cite its evidence.

Files are judged up to llm.concurrency at a time (default 4, see Providers), each with no context from other files, and the results are reported in path order whatever order the model answers in. Write standards that one file can satisfy on its own.

Decisions

lawbook records each answer as a decision in the Jev decision schema. A standard rule is a yes/no question, which Jev calls a noul:

{ "type": "noul", "noul": 0.12 }

noul is the probability, from 0 to 1, that the file meets the standard. There is no separate confidence: the probability already says how sure the model is, and 0.5 means it cannot tell.

JSON output lists every judged file’s decision under decisions, and each failing finding carries its own decision. The model’s reason is kept for findings; pass --explain to keep it for passing files too, which is the quickest way to see why a borderline file landed where it did while a standard’s wording settles. See Commands.

Threshold

threshold sets the probability a file must reach to pass. It ranges from 0 to 1 and defaults to 0.5, so by default a file the model cannot judge either way passes.

rules:
  - id: errors-are-actionable
    files: ["src/**/*.ts"]
    threshold: 0.8
    standard: Every error message says what went wrong and what to do next.

Raise it for standards that must be met beyond doubt; lower it to flag only clear violations. A threshold of 1 fails anything but a certain pass. While the wording and threshold settle, give the rule level: warn so its findings print without failing the run (see Configuration).

Writing a good standard

• Say what the file must do, not what reviewers dislike. “Every exported function has a doc comment that states what it returns” is checkable; “code is well documented” is not.
• Name the exemptions. Without them the model has to guess.
• Keep files narrow, and use exclude for fixtures and generated code. Every selected file but an empty one is one model request, so a **/* glob on a large repository is slow and costly. lawbook check --dry-run lists the files and counts the requests before you pay for them.
• Run lawbook check --only <id> while you refine the wording, and pin the rest of the suite with --no-llm.

Testing a standard

Writing a standard is writing a prompt, and a prompt wants a test. Give a rule fixtures: files the standard must pass and files it must fail. Then lawbook test judges each one and reports the ones the model put on the wrong side of the threshold:

rules:
  - id: errors-are-actionable
    files: ["src/**/*.ts"]
    standard: Every error message says what went wrong and what to do next.
    fixtures:
      pass: ["fixtures/errors/good.ts"]
      fail: ["fixtures/errors/vague.ts", "fixtures/errors/silent.ts"]

lawbook test

FAIL errors-are-actionable
  fixtures/errors/vague.ts: expected fail, judged pass (noul 0.72): The messages name the failing step.

3 fixtures, 1 misclassified

The reason shows what the model read into the standard, which is usually the exemption it needs spelled out. Tighten the wording, raise or lower threshold, and run again; verdicts are cached, so only changed fixtures and changed standards cost a request. Keep the fixtures out of files, or exclude them, so check does not judge them as code. A standard with fixtures that still misclassify is telling you the standard, not the code, needs work; see Commands.

Repeatability

Models can give different probabilities for the same file on different runs, so a standard rule is not as repeatable as a forbid rule. The number makes borderline files visible: a file that flips between 0.48 and 0.52 is telling you the standard’s wording is unclear, not that the model is. Where a regular expression can express the standard, prefer forbid or require. Reserve standard for judgment calls, and treat a failure as a prompt for a human to look at the file.

Judging files together

By default a standard rule judges each file in its own request, with no context from other files. A rule with scope: set sends every selected file in one request, each as a File: <path> block, and asks whether they meet the standard together, which fits standards about consistency across a module:

rules:
  - id: consistent-naming
    files: ["src/models/*.ts"]
    scope: set
    standard: Every model exports one class named after its file, in PascalCase.

FAIL consistent-naming
  src/models/user.ts exports `UserModel` while the others match their file names. (noul 0.20)

A set has one verdict: the finding names no file, and the JSON result carries it as decision instead of the per-file decisions. Suppressed files are left out of the set and listed as skipped, and blank files are left out unlisted. The set is sized as a whole against llm.maxBytes; a set that does not fit is reported as an ERROR without a request, since nothing was judged. A set with no files passes without a request. --dry-run counts one request for the rule.

Reference files

A standard often leans on a document the model has not seen: the team’s style guide, an architecture note, the list of approved loggers. context names files, relative to the checked directory, that every request of the rule carries as reference material:

rules:
  - id: follows-style-guide
    files: ["src/**/*.ts"]
    context: ["docs/style.md"]
    standard: The file follows the conventions in the style guide.

The model is told to read the reference files to understand the standard and to judge only the files in the message, never the references themselves. The context sits with the standard in the instructions, ahead of the files, so a CLI that caches its prompt reads it from the cache after the first file in a rule. Context files are read once per rule; a path that is missing, or a binary file, is a config error with exit code 2 before any request is made. They are not counted against llm.maxBytes, since the author chose them, and --dry-run does not list them. A changed reference file changes every verdict’s cache key, so the rule is judged again.

Changed lines

--since and --changed pick the files a change touched, but a standard rule still judges each of them in full, so a one-line change to an old file is judged on every line of it, and a finding on code the change never touched fails the run. --changed-lines scopes the verdict to the change:

lawbook check --since origin/main --changed-lines

For each selected file, lawbook takes the added and modified lines from the same diff the selector runs: the commits since the merge base for --since, the working tree against HEAD for --changed, and the working tree against the merge base when both are given. An untracked file counts as changed in full. The request carries the whole file with every line numbered and the changed lines named, and the model is told to judge those lines and to read the rest as reference material, the way it reads reference files. Its answer may name the line that shows the shortfall. A finding whose line lies outside the change is dropped and does not count against the rule; the text output says 2 findings outside the change under the rule, and JSON carries the count as outside on the rule and in the summary. A finding that names no line stays.

PASS tests-describe-behavior
  1 finding outside the change

1 passed, 0 failed, 0 warned, 0 errored, 0 skipped

A selected file with no changed lines, such as a mode change, or one named by --files that the change left alone, costs no request and is left out the way empty files are; --dry-run leaves it out too. A scope: set rule sends the changed files with their lines in its one request. The ranges are part of the cache key, so a verdict scoped to one change never answers for another or for the whole file. forbid, require, exists, and absent are unchanged: forbid already reports one line, and require asks about the file as a whole.

Without --changed or --since there is no change to scope to, so --changed-lines alone exits 2. A baseline still fits a rule adopted on an old codebase, where every file carries old findings; --changed-lines answers the per-change question, whether this change lowered the bar, without a file to maintain.

Per-rule model

The run’s llm section picks the model for every standard rule. A rule can override it with its own llm, so a subtle check can go to a stronger model while the rest keep the default:

llm:
  provider: claude-code
rules:
  - id: has-doc-comments
    files: ["src/**/*.ts"]
    standard: Every exported function has a doc comment.
  - id: errors-are-actionable
    files: ["src/**/*.ts"]
    standard: Every error message says what went wrong and what to do next.
    llm: { model: claude-sonnet-5-5 }

A rule’s llm takes provider and model. A rule that keeps the run’s provider inherits its model and replaces it only when it names one. A rule that names another provider starts from that provider’s default model instead, since a Claude Code model id means nothing to Codex; codex and kiro have no default, so such an override must name a model. concurrency, maxBytes, and cache always come from the run’s llm. The merged settings are checked when the config loads, so a codex override without a model is an error naming the rule.

Rules with the same provider and model share one judge, and the verdict cache keys on the model, so the same file judged by two models is cached twice.

Prompt caching

The standard and any reference material come before the files in every request, so a CLI that caches its prompt reads them from the cache after the first file in a rule. Claude Code and Codex report what they read from the cache, and the JSON output carries it under usage; Kiro meters credits rather than tokens, so its usage shows zero.

Verdict cache

Lawbook also caches verdicts itself. Every verdict is stored under node_modules/.cache/lawbook/ in the checked directory, keyed by a hash of the model, the system prompt, the standard, the file’s path, and its content. A later run answers from the cache for any file whose key is unchanged, without a request, and prints it as cached in the usage line. So a rerun on an unchanged tree is free and gives the same verdicts. For a scope: set rule the key covers every file in the set, in order. A lawbook release that changes the system prompt invalidates every entry; the next run simply asks again.

• --no-cache asks the model again for every file and stores nothing, so it also serves when the cache directory cannot be written. To forget the stored verdicts, delete the cache directory; the next run fills it again.
• --cache-dir <dir> keeps the cache elsewhere; the path is relative to the current directory, like --config.
• llm.cache: false in the config turns the cache off for every run.

In CI, persist the cache directory between jobs with a cache action keyed on the lockfile or the branch; a stale entry only costs one request. A cache file lawbook cannot write is reported as that file’s error, like a provider error, with the two flags to try.

Budget

A **/* glob on a large repository can turn one run into thousands of requests. llm.maxRequests caps a run: before any client is built, lawbook counts the requests the way --dry-run does and stops with exit code 2 when the count is over the cap, naming both numbers and what to do:

llm:
  maxRequests: 200

error: the run would make 412 model requests, over llm.maxRequests 200; narrow files, pass --only, or raise the limit

The count is the plan’s, so it leaves out empty files but includes files the verdict cache would answer, whitespace-only files, and files the size and suppression guards would drop; it is an upper bound on what the run would send. --max-requests <n> overrides the config for one run, and --no-llm makes no requests, so the cap never applies to it.

Large files

A standard rule never sends a file larger than llm.maxBytes (default 131072, see Providers). Such a file is listed under the rule as path: skipped, <bytes> bytes over llm.maxBytes <limit>, in text and in JSON under skipped, and does not affect the rule’s status, since nothing was judged. Raise llm.maxBytes, which applies to every rule, when one must see a large file, or narrow files so the file is not selected. Binary files are skipped by every rule without being listed; see Configuration.

Empty files

A file that is empty or holds only whitespace has nothing in it to break a standard, so a standard rule leaves it out without a request and without listing it, the way binary files are left out. An empty __init__.py costs nothing however many standards select it. --dry-run leaves empty files out too; it tells them by size, so a file of only whitespace still appears there. forbid and require still read such a file, so a standard that a file must have content to meet, such as a license header, belongs in a require rule.

Provider errors

When the provider rejects a request, for instance with a credential, quota, or network error, or answers without a verdict or with a probability outside 0 to 1, lawbook records the error against that file instead of stopping. The rule stops sending new requests, lets the ones already in flight finish, and reports the rest as not judged after an earlier error. The rule’s status is ERROR, its judged files keep their verdicts, every other rule still runs, and the run exits 2 after printing the report:

ERROR errors-are-actionable
  src/cli.ts: "unexpected state" on line 41 says what broke but not what to do. (noul 0.12)
  src/init.ts: claude-code: Not logged in. Run claude auth login
  src/report.ts: not judged after an earlier error

0 passed, 0 failed, 0 warned, 1 errored, 0 skipped

An ERROR rule is never a pass, whatever its level. A CLI that is not installed is reported the same way, on the first request.

Skipping

lawbook check --no-llm reports every standard rule as SKIP without touching a CLI, so a run on a machine with none installed still checks the deterministic rules and exits on their result alone. A config with no standard rule never touches a provider either.

Which provider answers, and how it is configured, is described in Providers.