Bright Data — Scraper Studio

• bdata scraper create <url> <description> — describe what you want in plain English; Bright Data's AI Flow generates a scraper template and returns a collector_id.
• bdata scraper run <collector_id> <url> — run that collector (or any existing one from the Bright Data web UI) against a URL and get the extracted data back.

Setup gate (run first)

if ! command -v bdata >/dev/null 2>&1; then
    echo "bdata CLI not installed — see bright-data-best-practices/references/cli-setup.md"
elif ! bdata zones >/dev/null 2>&1; then
    echo "bdata not authenticated — run: bdata login  (or: bdata login --device for SSH)"
fi

Pick your path

Action 1 — scraper create

bdata scraper create <url> "<description>" [--name <name>] \
    [--deliver-webhook <url>] [--timeout <seconds>] \
    [--json | --pretty] [-o <path>] [--timing] [-k <api-key>]

# Minimal
bdata scraper create https://example.com/product/1 \
    "Extract title, price, currency, image URL, and availability \
     from this product page. If the price has a strike-through \
     original price, capture both as price and original_price."

# Save the full AI output for inspection
bdata scraper create https://example.com/product/1 \
    "Extract title, price, and image URL" \
    --name product-scraper-v1 \
    --pretty -o create.json

What happens under the hood

1. POST /dca/collector — creates an empty scraper template with a stub webhook delivery target (https://example.com/webhook by default). Returns a collector_id like c_mp3tuab31lswoxvpws.
2. POST /dca/collectors/{collector_id}/automate_template — triggers Bright Data's AI Flow with the description + URL.
3. GET .../automate_template/progress (polled) — waits for status: "done". Generation typically takes 5–10 minutes for moderately complex pages.

Critical: hold the collector_id

• Open https://brightdata.com/cp/scrapers/{collector_id} to finish or inspect it in the web UI.
• Re-trigger generation programmatically against the same id.
• Delete it from the UI if they want a clean slate.

--timeout — default 600 s

bdata scraper create https://complex-site.com/page \
    "Extract all 30 fields …" \
    --timeout 1200

--deliver-webhook — placeholder by default

Action 2 — scraper run

bdata scraper run <collector_id> [url] \
    [--urls "u1,u2,..." | --input-file <path>] \
    [--sync [--sync-timeout 25-50]] \
    [--timeout <seconds>] [--name <name>] [--version <version>] \
    [--json | --pretty] [-o <path>] [--timing] [-k <api-key>]

Choosing the run mode

                ┌─────────────────────────────────────────────────┐
                │ Expected to finish in ≤ ~50 seconds?            │
                └─────────────────────────────────────────────────┘
                 yes                                       no
                  │                                         │
                  ▼                                         ▼
       ┌──────────────────────┐                ┌────────────────────────┐
       │   --sync             │                │  default (no flag)     │
       │   /dca/crawl         │                │  trigger_immediate +   │
       │   one-shot, 25–50 s  │                │  poll get_result       │
       └──────────────────────┘                └────────────────────────┘
                  │
                  │  on 202 timeout: response_id is printed —
                  │  re-run WITHOUT --sync to poll for it
                  ▼
            (cleanly fall back to async)

# Default — async + poll (recommended for most cases)
bdata scraper run c_mp3tuab31lswoxvpws https://www.amazon.com/dp/B08N5WRWNW

# Pretty-printed, saved to disk
bdata scraper run c_mp3tuab31lswoxvpws https://www.amazon.com/dp/B08N5WRWNW \
    --pretty -o product.json

# Sync mode for a fast page
bdata scraper run c_mp3tuab31lswoxvpws https://example.com/p/1 --sync

# Sync with a tighter server timeout
bdata scraper run c_mp3tuab31lswoxvpws https://example.com/p/1 \
    --sync --sync-timeout 30

# Multi-URL via comma-separated list — one batch call
bdata scraper run c_mp3tuab31lswoxvpws \
    --urls "https://example.com/p/1,https://example.com/p/2,https://example.com/p/3" \
    --pretty -o products.json

# Multi-URL via file (one URL per line; # comments and blanks ignored)
bdata scraper run c_mp3tuab31lswoxvpws --input-file urls.txt -o products.json

# Multi-URL via JSON array
echo '["https://example.com/p/1","https://example.com/p/2"]' > urls.json
bdata scraper run c_mp3tuab31lswoxvpws --input-file urls.json

--sync-timeout is bounded to 25–50

--sync timeout recovery

Silent auto-fallback to batch (paginated / large pages)

--name and --version

• --name <name> — tag this run in the Bright Data dashboard. Useful when you're scripting many runs and want to find one later.
• --version <version> — pin a specific scraper version (commonly dev when iterating in the web UI). Omit to use the latest published version.

Action 3 — scraper heal

bdata scraper heal <collector_id> "<what's wrong>" [--url <verify-url>] \
    [--timeout <seconds>] [--max-retries <n>] [--no-retry] \
    [--json | --pretty] [-o <path>] [--timing] [-k <api-key>]

What happens under the hood

1. POST /dca/collectors/{collector_id}/refactor_template with {prompt, custom_input: []} — triggers the AI self-healing job.
2. GET .../refactor_template/progress (polled) — waits for status: "done", same job shape and timing as automate_template.

Output + the verify loop

• preview_result — sample rows the fixed scraper would produce, so you can judge whether the fix is correct before committing it.
• diff_summary — a summary of what changed in the template.
• next_step — points at bdata scraper approve <collector_id> to commit (or --reject to discard).

{
  "collector_id": "c_mp3tuab31lswoxvpws",
  "status": "awaiting_approval",
  "preview_result": [{"price": "29.99", "currency": "USD"}],
  "diff_summary": "Updated price selector from .price to span[data-testid='price']",
  "next_step": "bdata scraper approve c_mp3tuab31lswoxvpws"
}

Failure is non-destructive

Action 4 — scraper approve

bdata scraper approve <collector_id> [--reject] [--url <verify-url>] \
    [--timeout <seconds>] [--json | --pretty] [-o <path>] [-k <api-key>]

• Approves by default (POST /dca/collectors/{id}/resume_automation_job {"message": true}), then polls to done and hands back a next_step = bdata scraper run <id> <url> to verify.
• --reject sends {"message": false} to discard the proposed fix; re-heal with a sharper prompt to try again.
• If a heal needs multiple approvals, approve may stop at awaiting_approval again — just run it again.

Full create-then-run workflow

# 1. Create the scraper, save the AI output, extract the collector_id
bdata scraper create https://example.com/product/1 \
    "Extract title, price, currency, image URL, and availability" \
    --pretty -o create.json

# 2. Pull the collector_id (or copy from the human-readable summary)
COLLECTOR_ID=$(jq -r '.collector_id // .id' create.json)
echo "Built: $COLLECTOR_ID"

# 3. Run it on every URL in one batch — single API call, merged result array
bdata scraper run "$COLLECTOR_ID" --input-file urls.txt \
    --pretty -o out/results.json

Common mistakes

1. Inventing command names. The commands are exactly bdata scraper create and bdata scraper run. There is no bdata generate, no bdata scrape-batch, no bdata data, no bdata build. If you're tempted to use one of those, you're hallucinating — run bdata scraper --help to verify.
2. Re-running create after a timeout. Generation creates a fresh collector every time. If polling times out, the half-built collector is still printed in the error output. Resume it in the web UI or wait and re-poll — don't burn another collector.
3. Defaulting to --sync for everything. Sync caps at 50 s server-side. Any paginated page or heavy SPA will time out. Default async is the right choice unless you specifically know the page is small.
4. Skipping Scraper Studio for known platforms. If the target is Amazon, LinkedIn, TikTok, Instagram, YouTube, Reddit, etc., the data-feeds skill exposes pre-built scrapers that are faster, cheaper, and more reliable. Only build a custom scraper when no pre-built one exists for the site, or when the user explicitly wants a custom output shape.
5. Treating sync timeout as failure. A --sync 202 timeout returns a valid response_id — the job is still running on the backend. Re-run the same command without --sync to pick it up.
6. Throwing away collector_id / response_id on error. The CLI prints them in every failure path on purpose. Both are recoverable via the API and the web UI. Always surface them to the user.
7. Trying to disable the batch auto-fallback. It has no flag. The fallback is the correct behavior when a URL expands past the realtime page limit. Let it run.
8. Hand-rolling a for url in $(cat urls.txt); do bdata scraper run ... loop. That's N API calls, N snapshot IDs, and N poll loops for what the API natively treats as one batch. Use --input-file urls.txt or --urls "u1,u2,..." — the CLI posts the whole array to /dca/trigger in a single request and returns one merged result array. This mirrors the canonical triggerWithUrls / trigger_with_urls helpers from the Scraper Studio reference SDKs.
9. Vague descriptions in create. A description like "scrape the page" produces a generic scraper. Name every field, name conditions ("if there's a sale price, capture both"), name disambiguators ("the price near the title, not in the recommendations sidebar"). See references/prompts.md [blocked].
10. Re-running create to fix a broken scraper. That builds a new collector and orphans the old one. To fix an existing scraper, use bdata scraper heal <collector_id> "<what's wrong>" — it mutates the scraper in place so your saved collector_id keeps working and improves.
11. Treating awaiting_approval as a failure. It is the normal end state of a heal — the fix is computed and waiting for your decision. Review preview_result, then bdata scraper approve <id> (or --reject). Use heal --auto-approve to skip the gate.

Troubleshooting

Reference files

• references/prompts.md [blocked] — How to write a description for scraper create that the AI Flow can act on. Examples of strong vs. weak descriptions, field-listing patterns, and conditional rules.
• references/recipes.md [blocked] — End-to-end recipes: capture collector_id from create, batch run a list of URLs, handle sync→async fallback, recover from a failed create, list scrapers from the dashboard.
• references/api-flow.md [blocked] — Exact REST endpoints, payloads, and status sentinels the CLI uses (/dca/collector, /dca/collectors/{id}/automate_template, /dca/trigger_immediate, /dca/get_result, /dca/crawl, /dca/trigger, /dca/dataset). Read this when debugging unexpected CLI output or when the user wants to hit the API directly.