An open-source screen recording an AI agent can query

You recorded the bug, then handed the file to an agent and watched it guess. Most coding agents cannot take a video at all, and a transcript of someone saying "click that one" stops resolving once the screen is gone. Clipy has done the most careful work on this problem. Its agent-readable recorder turns one recording into a structured document, and its own blinded study scores that document 11.8/12 against 5.8/12 for a transcript. That settles what one recording is worth.

The question people actually ask an agent spans days, people and machines, and no single document answers it. Kortix is the open-source CompanyOS: your agents, their skills, your company memory, and every connector in one git repo you own, with the agents working on real cloud computers. Kortix Capture is the part of it that records a member's screen, input actions and audio into one searchable timeline the account owns.

An agent queries a recording

Clipy's published comparison measures the format problem directly. The study ran three arms on one landing-page task, with five fresh sessions per arm and twenty primary trials. Hand a coding agent the raw MP4 and it scores 2.4 out of 12, at 417,487 input tokens and 147 seconds. Hand it a timestamped transcript and the score rises to 5.8 at 78,257 tokens, because the words are cheap and the pointing is gone. The AREC document, with intent, extracted requests, key moments carrying frames and click targets, diagnostics and capture gaps, scores 11.8 at 175,172 tokens. The structured document used 58% fewer input tokens than the raw video and finished 82 seconds sooner.

Those numbers are worth taking seriously. An agent reads, searches and cites, and nothing in that loop needs a video player. A recording that arrives as a document can be read in one pass. A document that states its own capture gaps lets the agent ask instead of inventing the missing screen.

One recording is one moment

The format argument ends at the edge of a single file. The questions people bring to an agent rarely fit there. A support lead asking what happened to a refund from last Tuesday needs every recording from that day, from the support person's machine, plus the billing screen someone opened in another window. Someone asking which screen showed an error needs a search across a week of frames, not a link to the one clip somebody remembered to share.

That is a retrieval problem, and retrieval is decided by the index. A per-recording document still requires the agent to know which recording to open before it can read anything. Loom answers the scope question with MCP tools over a video library and Rovo search inside it. Screenpipe answers it with continuous local capture and a searchable history on the device. Cap hands an agent a CLI and a local MCP server, so it can record, upload and read a transcript without opening a dashboard. Each answers a different part of the scope question: which recordings exist, and how far back a query can reach.

What a queryable record looks like

Kortix Capture records three layers of a member's computer. The screen layer holds video, the app, the window title, the URL and the on-screen text. Input actions, meaning clicks and typing, each carry a screenshot. Audio carries a transcript when the account's policy turns it on. Kortix uploads the recordings and indexes them into one timeline per person. Capture belongs to the account rather than to a project, so one account has one Capture workspace, one switch and one policy.

An agent in a member's private session reads that member's timeline through /v1/capture/me/search, /v1/capture/me/timeline and /v1/capture/me/frames/<frame_id>, or with kortix capture search, timeline and frame from the CLI. A search hit comes back as { kind, id, ts, user_id, device_id, chunk_id, display_id, display_name, app, title, url, snippet }, newest first. Each screen hit covers one chunk and window, and a request returns up to 100. The frame route returns the full on-screen text, a one-frame-per-second chunk video to seek through, and the nearest action screenshot. Each comes back as a signed URL that expires in five minutes.

The account route is the same index with a wider scope. With a kortix_pat_ key and the capture.read_all permission, a script or an agent can search every member's timeline:

curl -s -H "Authorization: Bearer $KORTIX_API_KEY" \
  "https://<your Kortix API>/v1/accounts/$ACCOUNT/capture/search?q=invoice&limit=5"

Add kinds=screen,actions,audio to narrow the layers, from and to to bound the window, and scope=account where the route takes it. The timeline route covers a window of at most 31 days, which is the shape of the question a support lead asks on a Monday.

The policy is the part a recorder cannot give you

A record of everyone's screen is a governance problem before it is a search problem. Kortix Capture puts one policy on the account and publishes it to orgs/<account_id>/policy.json in the capture store, where every device reads it within a minute. A device cannot write a policy of its own, and Kortix enforces the layers again when it indexes, so a device never widens the policy.

The default is an exclusion policy: record everything except the apps, websites, categories and private browsing windows you exclude. Password managers, messaging and banking are categories you can exclude. An inclusion policy records only what you list and blacks out the rest. Screen frames run at one every 0.5 to 10 seconds at 720p to 4K. Retention is a ceiling: remote_days defaults to 90, and 0 keeps recordings without a limit. A person can keep less on their own computer, never more.

Reads are audited by name. An agent's read writes capture.agent_read, a read of another member's data writes capture.member_view, and an account-wide read writes capture.account_view. Access is Kortix IAM, and Capture has no roles of its own. Every member reads their own devices and timeline, capture.read_all reads everyone's, and capture.manage holds the switch and the policy.

Where Capture sits in Kortix

Capture ships inside the account rather than beside it. It is off by default, and while it is off every Capture route answers 403 with capture_disabled, a device cannot sign in, and Kortix indexes nothing new. An owner or admin turns it on from /capture, or with PATCH /v1/accounts/<account_id>/capture and {"enabled": true}.

The permission model is narrower than the account search suggests. An agent reads only the timeline of the person its private session acts for. A trigger run or a shared session acts for no person and gets 403 capture_no_human. A person with capture.read_all can search a colleague's timeline through the account route with their own key; an agent in a session reads only its own person.

Run it on Kortix Cloud, in your VPC, or on your own on-prem network. Self-host is free. Open source. Read it, fork it, audit it.

Start with one machine

Turn Capture on for one person, record a day, and ask the agent something you would normally ask a colleague. The answer will tell you more about your retrieval than any benchmark. If you want the longer argument about the record an agent leaves behind, read the audit trail post and what an agent team shares. Then wire the search route into a session.

Try Kortix