WebCrawlerAPI
WebCrawlerAPI provides crawling, page scraping, AI web agents, and monitored website feeds through a unified API.
10 actionsAPI key
What your bot can do in WebCrawlerAPI
Looks up
- Get Crawl JobGet a crawl job's status, configuration, per-page results, content URLs, errors, and observed costs.
- Get FeedGet one feed's configuration, lifecycle status, and recent run history, including per-run crawl counts and cost.
- Get Organization CostsReturn current spendable balance plus request count and USD usage for a date range.
- List Feed ChangesReturn one page of detected feed changes as structured JSON, with an opaque continuation cursor for older pages.
- List FeedsList active and paused feeds for the connected organization, newest first.
- Scrape PageScrape one web page synchronously and return requested content or structured extraction.
Takes action
- Create CrawlStart a metered asynchronous crawl over a site and return its job ID.
- Create FeedCreate a recurring website-change feed.
- Delete FeedPermanently cancel a feed so it cannot be resumed.
- Set Feed StatePause an active feed's future scheduled runs or resume a paused feed.
