KeelDeployed

The problem

A small business or a public body has years of policy documents and staff records sitting in folders, and a question about any of it gets answered by whoever remembers where things are. The obvious fix is to ask a model, which means handing the whole shelf to a company overseas, where the documents and the questions both live from then on. Payroll bands would sit beside the roster the front desk reads. That trade is where most of these projects stop.

What it does

Keel runs retrieval and tool-calling under policy, on Azure or entirely on-premise where the data stays in the building. Documents go in carrying access tags, and answers come out citing the exact passages they were drawn from. One Python codebase is the whole appliance: the same code serves a quantised local model on hardware the business owns and an Azure tenancy running Azure OpenAI and AI Search under a managed identity with no key anywhere, and it behaves the same way in both. The profile is decided in one place, and everything above the provider contracts is written as though only one profile existed.

Permission filtering sits inside retrieval rather than after it. Every chunk carries the tags of the document it came from, every request carries a user with tags, and chunks outside those tags leave both candidate lists before the two lists are fused, so a document outside a reader’s entitlement stays clear of the reranker and clear of the context window. When the best entitled passage falls below the relevance line, the appliance answers with one fixed sentence from the gate alone, at zero model calls, which makes the quiet outcome a refusal rather than a guess.

The agent side calls typed tools under a written policy: an allowlist per deployment, argument rules, a call budget, and schema validation of every argument before a tool runs. A tool marked as a write is stored pending and reported to the model as queued, then runs once after a person approves it. Every request, retrieval set, tool call, approval, ingest and quarantine change lands in a hash-chained ledger, beside an inference log holding one row per request with the user, the tags, the chunk ids retrieved, the answer, the citations, latency and tokens. A golden-set evaluation with a regression gate ships in the same repository, so answer quality is measured on the deployment’s own model rather than asserted.

How it works

One request travels the same path in both profiles, from a file on disk to a row in the ledger. Everything persistent lives in one SQLite file, so a backup is one consistent copy of it and a restore is putting it back followed by a ledger verify.

• Ingest and the injection screenLoaders read PDF, DOCX, Markdown, HTML and plain text into section-aware chunks, each stamped with its source, heading, page, character span, checksum and access tags; re-ingesting the same bytes adds nothing, because the document checksum is what makes ingest idempotent. Every chunk is screened at ingest against weighted patterns, with an optional model judge, and a flagged chunk is stored quarantined with its reason where an operator can read it.
• Retrieval under the caller’s tagsBM25 over SQLite FTS5 and vector search over embeddings held in the same file each run with the tag list inside the query, then a second entitlement check drops anything the store let through, then reciprocal rank fusion combines what remains. A cross-encoder reranks the survivors and maps its score into the unit interval, so the gate downstream has a relevance estimate it can compare against a number.
• The relevance gate and the answerBelow the configured line, 0.15 by default, the engine writes a refusal and reaches no model, which is why a refusal is the fastest response the appliance gives. Above it, the entitled passages go to the model numbered from one, the bracketed markers in the reply resolve back to real chunk ids, and JSON mode validates the reply against a caller-supplied schema with one retry before reporting an error.
• The agent loop and the approval queueA proposed tool call meets the deployment policy, then the registry validates its arguments against the tool schema, then the tool’s own guard runs: arithmetic only in the calculator, one SELECT over allowlisted tables in the SQL tool, an allowlisted host for HTTP. Write tools stop at the approval queue and wait for a named person, and the approval re-checks the live policy at execution time so a decision cannot outlive the rules that permitted it.
• The ledger and the inference logEach ledger row stores a SHA-256 over a canonical JSON array of the previous hash, the row kind, the request id and the payload, computed in Python inside an immediate transaction under a per-connection lock. The inference log runs alongside it as the readable record, feeding the admin page totals, a fourteen-day trend, the request detail view and the exported log.
• The two deploy profilesOn-premise, native runner scripts start the model server and the app without Docker, and a Compose stack runs the app image beside a llama.cpp container with the air-gap guard on and that model service as its only permitted destination. On Azure, one Bicep template raises Container Apps, Azure OpenAI deployments, AI Search, Key Vault, a user-assigned managed identity and private endpoints behind a flag, with local key auth disabled on the two services that would otherwise accept a key.

What was hard

Safe to leave running

Permission filtering runs before generation and again at the generation boundary. A chunk outside the caller’s tags leaves both candidate lists before fusion, and the answer engine re-checks every hit it was handed, so a retriever carrying a bug of its own still yields only what the reader is entitled to. Below the relevance line the response is one fixed sentence at zero model calls, which makes the unattended outcome a refusal rather than an invention.

Retrieved text is treated as data rather than instruction. Every chunk is screened at ingest, a flagged chunk is stored quarantined with its reason and stays out of both retrieval paths and out of the prompt, and releasing one is an operator action that writes its own ledger row. Two heuristic gaps stay open on purpose: a paraphrase written around the words the heuristics key on, covered by the model judge available at ingest, and one benign passage the patterns flag that an operator releases from the quarantine list. Both are written as strict expected-failure tests, so the suite fails on the day either one starts passing.

Whatever the model proposes meets the deployment policy before a tool sees it. The SQL tool takes one SELECT over allowlisted tables through a read-only connection with an authoriser, a step budget, a row cap and a size cap on any single value; the calculator walks an abstract syntax tree, accepts arithmetic and bounds the size of a result before a power is computed; HTTP needs an allowlisted host and is refused outright under air-gap. Write tools reach a person or they wait, and every transition in the queue is a ledger row carrying the decider.

With the air-gap flag set, the process refuses outbound connections to any host beyond loopback and its allow list at the socket, asyncio, urllib and httpx layers, before a packet leaves. Name resolution is guarded on the same terms, since a DNS lookup is an exfiltration channel of its own and was one of the review findings. Admin routes are open on loopback and require a token past it. Beyond loopback the appliance takes the user and the tags only from a reverse proxy holding a shared token, and every other caller runs as the public role. The hosted demo of the fixture corpus sets a flag that honours its own user picker instead, and that flag stays off for a real deployment.

The trail is checkable by the operator rather than by the vendor. Verifying the ledger recomputes the chain and names the first broken link, and an exported chain verifies offline with nothing but the Python standard library, against a hash recipe the architecture document writes out in full. The ingest ledger row is written inside the ingest transaction, so a ledger failure rolls the ingest back and the store and the trail stay in step.

The published claims are machine-checked, and the limits are published beside them. Repository tests read a live pytest collection and fail the build when a test count or a review row drifts from the number printed on a page, and another fails the build if any page claims every medium finding was fixed while one stays open. The security policy names what the appliance leaves to the operator: it decides what a user may retrieve rather than what they do with an answer on screen, the database file holds the corpus in plain SQLite and wants an encrypted disk, the egress guard covers this process while the host firewall covers the machine, and identity in this build comes from the machine’s own login or from the operator’s proxy, with per-person login named as a later release.

Evidence

• 105 attack tests
  The three red-team files define 105 test functions, which pytest expands into 174 parametrised cases, and they run in the default suite rather than in a job of their own. A repository test counts both from a live pytest collection and holds them against the numbers printed in the README, so the figure and the code move together or the build fails. The same counts appear on the appliance overview page.
• Hash-chained ledger
  Each row stores a SHA-256 over a canonical JSON array of the previous hash, the row kind, the request id and the payload. The verify command recomputes the chain, names the first broken link and exits 1 on a break, which a CLI test asserts; separate tests tamper with a stored payload and with an exported file and require both to be detected. Timestamps and sequence numbers sit outside the hash, so order is bound by the chain itself.
• One-command deploy
  From a fresh clone, keel up finds an OpenAI-compatible chat server already answering on the machine, writes .env, loads the fixture corpus, starts the appliance on 127.0.0.1:8400 and opens it, skipping each step that has already happened. The container route is one docker compose command against the host-model stack, which reaches the model runtime on the host and keeps the air-gap guard on with that bridge as its only permitted destination. A test parses both Compose files and asserts that air-gap declaration, and the README records that no Compose run has happened on the reference machine, where Docker is absent, while the same Dockerfile is what the hosted demo runs. The Azure route is one Bicep template driven by a deploy script; it is code-complete and unit-tested against mocked SDK clients, and the README states plainly that no live Azure deployment has run yet.
• 687 tests across 21 files
  Counted by pytest --collect-only, of which 174 are the adversarial cases. A repository test reads the number out of the README and out of the overview page and holds both to what pytest collects, so a rounder, friendlier number fails the build. Continuous integration runs the same suite on every push with the integration tests deselected, alongside ruff, a Bicep build and lint, and the evaluation harness against fakes.
• 27 findings, 25 fixed, 2 open by choice
  The published review table carries a row per finding with its severity and the test that proves the fix. Hygiene tests count the table rows against the prose, require every test named in the proof column to be a test pytest actually collects, and count the strict expected-failure markers in the red-team files against the two findings declared open.
• hit@3 1.00, refusal correctness 1.00, judged correctness 0.94
  One evaluation run of the 22-item golden set through the production answer path on the reference machine on 2026-08-18, judged by the deployment’s own 3B model on CPU, printed in the README, which names the report JSON they were copied from and records that the reports directory stays out of version control. Latency per item came in at 1,951 ms median and 2,947 ms at the 95th percentile. It is one run on one machine with a small model, which the README says in the same breath, and the comparison against a 9B model is the next measurement.
• Three to nine seconds for a cited answer over the public URL
  Five questions timed with curl against the hosted demo on 2026-08-20 across two different roles, recorded in the Railway deploy notes and in the audit document in the tree. A refusal returned in about two seconds at zero output tokens, since the relevance gate answers before any model call. The first answer after a deploy takes longer while the model loads.

See it running

The hosted demo at keel.flow-through.com.au runs a fixture corpus through the same code paths an installed appliance uses, where one click asks the same restricted question as a public reader and as an HR officer so the two answers sit side by side.