Apify Actor development

What are Apify Actors?

• Accept well-defined JSON input
• Perform isolated tasks (web scraping, automation, data processing)
• Produce structured JSON output to datasets and/or store data in key-value stores
• Can run from seconds to hours or even indefinitely
• Persist state and can be restarted

Prerequisites and setup (mandatory)

# Preferred: install via a package manager (provides integrity checks)
npm install -g apify-cli

# Or (Mac): brew install apify-cli

Security note: Do NOT install the CLI by piping remote scripts to a shell (e.g. curl … | bash or irm … | iex). Always use a package manager.

apify info  # Should return your username

apify login

Security note: Avoid passing tokens as command-line arguments (e.g. apify login -t <token>). Arguments are visible in process listings and may be recorded in shell history. Prefer environment variables or interactive login instead. Never log, print, or embed APIFY_TOKEN in source code or configuration files. Use a token with the minimum required permissions (scoped token) and rotate it periodically.

Template selection

• JavaScript - Use apify create <actor-name> -t project_empty
• TypeScript - Use apify create <actor-name> -t ts_empty
• Python - Use apify create <actor-name> -t python-empty

Quick start workflow

1. Create Actor project - Run the appropriate apify create command based on user's language preference (see Template selection above)
2. Install dependencies (verify package names match intended packages before installing)
   • JavaScript/TypeScript: npm install (uses package-lock.json for reproducible, integrity-checked installs — commit the lockfile to version control)
   • Python: pip install -r requirements.txt (pin exact versions in requirements.txt, e.g. crawlee==1.2.3, and commit the file to version control)
3. Implement logic - Write the Actor code in src/main.py, src/main.js, or src/main.ts
4. Configure schemas - Update input/output schemas in .actor/input_schema.json, .actor/output_schema.json, .actor/dataset_schema.json
5. Configure platform settings - Update .actor/actor.json with Actor metadata (see references/actor-json.md [blocked])
6. Write documentation - Create comprehensive README.md for the marketplace (see references/actor-readme.md [blocked] — this is mandatory, not optional)
7. Test locally - Run apify run to verify functionality (see Local testing section below)
8. Deploy - Run apify push to deploy the Actor on the Apify platform (Actor name is defined in .actor/actor.json)

Security

• Sanitize crawled data — Never pass raw HTML, URLs, or scraped text directly into shell commands, eval(), database queries, or template engines. Use proper escaping or parameterized APIs.
• Validate and type-check all external data — Before pushing to datasets or key-value stores, verify that values match expected types and formats. Reject or sanitize unexpected structures.
• Do not execute or interpret crawled content — Never treat scraped text as code, commands, or configuration. Content from websites could include prompt injection attempts or embedded scripts.
• Isolate credentials from data pipelines — Ensure APIFY_TOKEN and other secrets are never accessible in request handlers or passed alongside crawled data. Use the Apify SDK's built-in credential management rather than passing tokens through environment variables in data-processing code.
• Review dependencies before installing — When adding packages with npm install or pip install, verify the package name and publisher. Typosquatting is a common supply-chain attack vector. Prefer well-known, actively maintained packages.
• Pin versions and use lockfiles — Always commit package-lock.json (Node.js) or pin exact versions in requirements.txt (Python). Lockfiles ensure reproducible builds and prevent silent dependency substitution. Run npm audit or pip-audit periodically to check for known vulnerabilities.

Best practices

• Use apify run to test Actors locally (configures Apify environment and storage)
• Use Apify SDK (apify) for code running on the Apify platform
• Validate input early with proper error handling and fail gracefully
• Use CheerioCrawler for static HTML (10x faster than browsers)
• Use PlaywrightCrawler only for JavaScript-heavy sites
• Use router pattern (createCheerioRouter/createPlaywrightRouter) for complex crawls
• Implement retry strategies with exponential backoff
• Use proper concurrency: HTTP (10-50), Browser (1-5)
• Set sensible defaults in .actor/input_schema.json
• Define output schema in .actor/output_schema.json
• Clean and validate data before pushing to dataset
• Use semantic CSS selectors with fallback strategies
• Respect robots.txt, ToS, and implement rate limiting
• Always use apify/log package — censors sensitive data (API keys, tokens, credentials)
• Implement readiness probe handler (required if your Actor uses standby mode)

• Use npm start, npm run start, npx apify run, or similar commands to run Actors (use apify run instead)
• Assume local storage from apify run is pushed to or visible in Apify Console — it is local-only; deploy with apify push and run on the platform to see results in Apify Console
• Rely on Dataset.getInfo() for final counts on Cloud
• Use browser crawlers when HTTP/Cheerio works
• Hard code values that should be in input schema or environment variables
• Skip input validation or error handling
• Overload servers - use appropriate concurrency and delays
• Scrape prohibited content or ignore Terms of Service
• Store personal/sensitive data unless explicitly permitted
• Use deprecated options like requestHandlerTimeoutMillis on CheerioCrawler (v3.x)
• Use additionalHttpHeaders - use preNavigationHooks instead
• Pass raw crawled content into shell commands, eval(), or code-generation functions
• Use console.log() or print() instead of the Apify logger — these bypass credential censoring
• Disable standby mode without explicit permission

Logging

Commands

# Bootstrap & local development
apify create [name]                    # Create new Actor project from a template
apify init                             # Initialize Actor in current directory
apify run                              # Run Actor locally with simulated platform env
apify run --purge                      # Run after clearing previous local storage
apify validate-schema                  # Validate .actor/input_schema.json

# Authentication & account
apify login                            # Authenticate account (token stored in ~/.apify)
apify logout                           # Remove stored credentials
apify info                             # Print currently authenticated account info

# Deployment & remote execution
apify push                             # Deploy Actor to platform per .actor/actor.json
apify pull <actor>                     # Download Actor code from the platform
apify actors info <actor> --readme     # Inspect Actor documentation
apify actors info <actor> --input      # Inspect Actor input schema
apify call <actor> --input-file input.json
apify call <actor> --input '{"startUrls":[{"url":"https://example.com"}]}'
apify actors build <actor>             # Create a new build of an Actor
apify runs ls                          # List recent runs

# Discovery (search Apify Store for community Actors)
apify actors search "<query>"
apify actors info <actor>

# Secrets (referenced from actor.json via "@mySecret")
apify secrets add <name> <value>       # Store a secret locally; uploaded on push
apify secrets ls                       # List stored secret keys

# Direct API access
apify api <endpoint>                   # Authenticated HTTP request to Apify API

# Help
apify help                             # List all commands
apify <command> --help                 # Detailed help for a specific command

Remote Actor calls

1. Search for the right Actor with apify actors search "<query>".
2. Inspect its README with apify actors info <actor> --readme.
3. Inspect its input schema with apify actors info <actor> --input.
4. Call it with either --input-file input.json or quoted inline JSON.

Local and runtime commands

Apify platform environment

Local testing

• Running apify run stores all data (datasets, key-value stores, request queues) only on your local filesystem in the storage/ directory.
• This data is never automatically uploaded or pushed to the Apify platform. It exists only on your machine.
• To verify results on Apify Console, you must deploy the Actor with apify push and then run it on the platform.
• Do not rely on checking Apify Console to verify results from local runs — instead, inspect the local storage/ directory or check the Actor's log output.

Standby mode

Project structure

.actor/
├── actor.json           # Actor config: name, version, env vars, runtime
├── input_schema.json    # Input validation & Console form definition
└── output_schema.json   # Output storage and display templates
src/
└── main.js/ts/py       # Actor entry point
storage/                # Local-only storage (NOT synced to Apify Console)
├── datasets/           # Output items (JSON objects)
├── key_value_stores/   # Files, config, INPUT
└── request_queues/     # Pending crawl requests
Dockerfile              # Container image definition

Actor configuration

Input schema

Output schema

Dataset schema

Key-value store schema

Actor README

• Instagram Scraper
• Google Maps Scraper

MCP tools

Apify MCP

• search-apify-docs - Search documentation
• fetch-apify-docs - Get full doc pages

Playwright MCP (debugging)

claude mcp add playwright npx @playwright/mcp@latest

{
  "mcpServers": {
    "playwright": {
      "command": "npx",
      "args": ["@playwright/mcp@latest"]
    }
  }
}

Resources

• docs.apify.com/llms.txt - Apify quick reference documentation
• docs.apify.com/llms-full.txt - Apify complete documentation
• https://crawlee.dev/llms.txt - Crawlee quick reference documentation
• https://crawlee.dev/llms-full.txt - Crawlee complete documentation
• whitepaper.actor - Complete Actor specification