How we QA every pull request automatically
The QA agent we run on Kortix — connected to GitHub and our test environment. It checks out each PR, runs the suite, exercises the change, and posts the result.
Code review catches problems a person can find by reading a diff. It misses the ones you only find by running the change: a test that passes locally but flakes in CI, a path that works in the happy case and 500s on an edge case, a migration that reads fine but locks a table under load. These tend to surface in staging or production, after the PR is approved.
We run a QA agent on Kortix that does this checking when the PR opens. This is how we QA our own changes, including the connections and guardrails involved.
The problem
Some bugs don't show up in a diff. A reviewer reads the code and approves, but no one has checked out the branch, run the suite, hit the new endpoint, and watched what happens. CI runs the tests the author wrote; it doesn't cover the paths they missed.
The common fixes are incomplete. Green CI shows the existing tests pass, not that the change is correct. A manual QA pass is thorough but slow and lands after the review. A generic AI reviewer only sees the diff; it can't run the branch, so it misses failures that only appear at runtime.
What we built
On Kortix, each PR triggers an agent. On open and on every push, the PR spawns an isolated session (a cloud sandbox) with scoped access to the branch, the test suite, a deployable test environment, and the Cloudflare edge in front of it. The agent checks out the change, runs the suite, exercises the new behavior on a live deploy, and posts a pass/fail result on the PR.
How it works
Connect GitHub as the trigger
A signed GitHub webhook points at the project. Every PR opened or updated fires it, and each firing spawns a fresh session in its own sandbox, seeded with the branch under test. One PR maps to one session on one disposable machine, so nothing carries over between runs and concurrent PRs run in parallel.
Give the agent the branch and the test playbook
The session checks out the branch and installs it clean. Our QA conventions live as skills and memory that travel with the agent: how to run the suite, which flows are critical, edge cases that have caused problems before, and what a result should contain. When a bug slips through, we write it down and the agent picks it up on the next run.
Connect the systems QA needs
Through scoped connectors, brokered server-side so no raw token reaches the model, the agent can:
- Run the full suite — unit, integration, and e2e inside the sandbox, with the failure output captured in full.
- Deploy to the test environment — it stands the change up on an ephemeral deploy and exercises it end-to-end against the new paths.
- Reach the edge via Cloudflare — it checks behavior through the edge (routing, headers, caching, redirects), not just localhost.
- Report on GitHub — the pass/fail result and any reproduction post as a check and a comment on the PR.
Set the guardrails
The agent operates against the test environment only; production is out of scope. It does not merge or deploy to prod. Its output is a result, and a human owns the merge. Credentials are encrypted in the Secrets Manager and injected at runtime, never shown to the model or written to logs.
Let each PR arrive pre-QA'd
With that in place, a new PR checks itself out, runs the suite, deploys to the test environment, exercises the change through Cloudflare, and posts a result: green when it passes, or a red check with the failing command, the logs, and steps to reproduce. A flaky test is flagged with the evidence. A broken edge route is caught before staging.
Summary
A trigger on every PR spawns a session with scoped connectors into the branch, the test suite, the test environment, and Cloudflare. The QA playbook is encoded as skills and memory. The agent stays on the test environment and a human owns the merge.
Guardrails
The agent has access to the test environment and the edge, so the access is scoped and contained:
- Isolation. Every PR runs in its own microVM sandbox on its own branch. The session can install, deploy, and exercise the change to reproduce a failure; only the reported result leaves the sandbox.
- Scoped secrets. The GitHub, test-environment, and Cloudflare credentials are encrypted in the Secrets Manager and injected into the sandbox at runtime, never exposed to the model or the logs.
- Test environment only. The agent's access stops at the test plane: no production access, no prod deploy, no merge. It reports; the team decides.
- Everything is code. The agent's configuration, skills, and permissions are files in the repo, versioned and changed through a reviewed change request rather than a dashboard setting.
The outcome
Runtime failures that a diff can't show now surface on the PR when it opens, with the failing command, the logs, and the repro attached. Reviewers spend their time on design rather than checking out branches by hand, and the changes that reach staging have already been run against the test environment.
Read more
How we keep our docs in sync with the code
The docs agent we run on Kortix — connected to GitHub and our codebase. Once a day it checks the code that landed since its last run and updates the docs those changes affected, opening a PR for review.
How we run operations from Slack
A single agent reachable from Slack, with scoped access to our database, Stripe, Linear, and GitHub. Ask in a thread and it runs the task across whatever platforms it needs.
How we run customer support with an AI agent
The support agent we run on Kortix — connected to Plain, our codebase, and Stripe. It triages and resolves inbound threads, and stops for human approval on anything sensitive.
Engineer your first loop
Give your company a workforce of agents that run on a schedule, ship real deliverables, and improve one reviewed change at a time. Free to self-host, managed cloud from $20.
Open source · SSO · RBAC · on-prem · no lock-in