Human in the loop vs human on the loop: open source decides it per call

Human in the loop and human on the loop are not two settings you pick once for a system. In open source, the answer belongs to each action, in a policy file the runtime reads at call time. Kortix is the open-source AI Operating System.

Who holds the call

Elementum calls human in the loop "an AI operating model in which a human must review, approve, or act on AI output before any automated system proceeds with execution" (Elementum). Torch.AI puts the pair in one sentence: "in the loop requires a person to approve each action before the system acts, while on the loop lets the system act on its own under active human supervision with the authority to intervene or stop it" (Torch.AI).

Read together, the difference is a default: in the loop nothing proceeds without a decision, on the loop the action proceeds unless someone stops it.

The pages agree on the unit. Flowable: "It is not a choice you make once when deploying an AI system. It is a decision that should be made for every meaningful step in a workflow" (Flowable). Decagon's glossary says it plainly: "The choice is made per action, not per system" (Decagon).

The open question is whether anything checks that choice at run time. A per-action decision written in a design document is still a per-system one in production, because the runtime never reads the document.

Two ways a system-level choice fails

"Human-in-the-loop fails through fatigue," Decagon writes, because "reviewers who approve a long run of correct proposals stop reading closely, and approval turns into rubber-stamping that adds delay without adding safety" (Decagon). One mode for the whole agent produces exactly this. The agent lists a shared drive at nine and refunds a customer at ten, and both arrive as the same card. By the fortieth card of the week, the recipient field gets a glance.

The second failure is quieter: an audit log nobody reads. An agent running on the loop records no decision until after the action lands, so the first report of a wrong send is the customer's reply. Torch.AI draws the line there: "A person who cannot perceive the action, or cannot stop it before it takes effect, is on the loop in name only" (Torch.AI). Decagon adds that on-the-loop oversight "only protects actions that can be undone" (Decagon).

Both failures trace back to where the mode was set: once for the agent instead of once per call.

Six call classes, three modes

A call class groups calls that share a consequence, and the mode follows from what a wrong call costs: a bad read costs a retry, a bad email is already in an inbox.

The table is the design work. Rows three and four are the same action, gmail.send_email, with different modes, and nothing in the name separates them. The recipient argument does.

The policy that encodes the table

Kortix stores policy as YAML in the project repo (connectors). These two rules are rows three and four.

policy:
  default_mode: risk

policies:
  - match: gmail.send_email
    action: always_run
    conditions:
      - arg: to
        match: "*@example.com"
  - match: gmail.send_email
    action: require_approval

Read the rules in order: always_run is Allow, require_approval is Ask, block is Block. The first lets a send to the company's own domain run. The second matches the same action with no condition, so every other recipient waits. First match wins, so the narrow rule sits above the wide one.

The condition is where a per-action decision becomes real. arg: to walks into the call's arguments by dot path, so message.channel reaches a nested field, and match takes a glob or a /regex/. A rule holds up to ten conditions, all must hold, and negate: true flips one. A malformed condition fails closed.

default_mode: risk sets the floor for anything unnamed: reads run, writes and destructive calls wait. allow_all runs whatever no rule matched. Project-wide rules speak first, then the connector's own, then sensitive: true, then the default mode.

Approval gates you set. Off until you set them. The table is work you do.

What a held call actually is

An agent in a support project writes a reply and calls gmail.send_email. The rule on that action is Ask, so the call stops before anything is sent.

What comes back is a receipt for a call that did not happen: a 202 carrying status: "pending_approval", an approval_url, an approval_summary and an execution_id. Nothing ran.

The person with authority reads the arguments the provider would see, redacted, then approves or denies. The turn picks up where it stopped.

The scope of the yes matters more than the click: this connector, this action, these arguments, this account. Change the recipient and the old approval does not follow; the new address needs its own decision. The same call repeated inside fifteen minutes runs on that approval, so a retry is not a second email.

A held call expires after 24 hours, or after approvals.ttl when the project sets one (Kortix clamps it to 5 minutes through 30 days) (manifest). The link stops working and the session does not resume, so the agent asks again with a new call. No approval covers a whole session, so an action that must run unattended needs an Allow rule.

Authority sits with a project manager or whoever started the session; a session started with an API token, the CLI or a service account goes to project managers only.

Every Ask spends one decision from a named person and waits.

Where supervision earns its keep

Human on the loop is right for a large class of work: the system acts on its own under active supervision, and the person watching can stop a call before it takes effect. That fits the table's Allow rows: reads, internal notes, and a message to a known domain.

A third position sits above both. Human over the loop is where a person sets the goal and the constraints, then reads the outcome and changes the rule when it drifts. That is real oversight too; it just does not decide individual calls, while the on-the-loop supervisor still can.

The mode is still a policy question: something has to say what runs unattended. Under default_mode: risk, reads run and writes wait, so attention goes to the exceptions, and widening one action to always_run stops the asking.

Supervision with no lever is watching. If the mode lives in a meeting and a wiki page, a supervisor who spots a bad pattern cannot tighten anything without a release. If the mode is a file, the change is one rule and one diff.

What to write down

For each action, write three things: the mode, who may decide it, and the condition that separates it from a lookalike. Those three lines are the oversight model.

Then decide who may change them. A rule an agent writes in a session reaches main through a change request, so a reviewer reads the new rule beside the work that prompted it. A change a project manager makes to connector policy in the dashboard, or with kortix connectors policy set --default risk --apply, takes effect at once, unreviewed. Both doors are legitimate; know which one is open.

For the credential boundary, read giving AI agents tool access safely. For the review each platform ships, read the open-source comparison. For the closed alternative, Kortix vs Claude Cowork.