Crawlbase
Crawlbase provides APIs for crawling web pages, managing crawler queues, storing crawl results, and monitoring account usage.
8 actionsAPI key
What your bot can do in Crawlbase
Looks up
- Generate User AgentsGenerate one to ten realistic randomized User-Agent strings for a selected device class.
- Get Crawling UsageReturn current Crawling API usage and optionally previous-month statistics for the connected Crawlbase account.
- Get Storage CountReturn the number of pages currently held in Crawlbase Cloud Storage.
- Get Stored PageRetrieve the latest Crawlbase Cloud Storage page by exactly one request ID or original URL.
- Get Stored PagesRetrieve up to 100 Crawlbase Cloud Storage pages by request ID without deleting them.
- List Stored Page IDsReturn one page of Crawlbase Cloud Storage request IDs and a short-lived continuation cursor.
Takes action
- Crawl URLFetch a public HTTP or HTTPS URL through Crawlbase using the connected Normal token and return its content with separate target and…
- Delete Stored PagesIrreversibly delete explicitly selected pages from Crawlbase Cloud Storage by request ID and return each page's deletion outcome.
