Endpoints

Crawl

A documentation site, a help center or a whole blog, page by page, with depth and path filters. Poll the job or take a webhook.

POST /v1/crawl

Request

Send a JSON body with a URL (or a list) and any options. The response holds data and what the call cost.

import requestsr = requests.post(    'https://api.snoopscan.com/v1/crawl',    headers={'Authorization': 'Bearer sk_YOUR_KEY'},    json={        'url': 'https://example.com',        'limit': 100,        'maxDepth': 3,        'includePaths': [            '/blog/*'        ]    },)print(r.json())

const r = await fetch('https://api.snoopscan.com/v1/crawl', {  method: 'POST',  headers: { Authorization: 'Bearer sk_YOUR_KEY', 'Content-Type': 'application/json' },  body: JSON.stringify({      "url": "https://example.com",      "limit": 100,      "maxDepth": 3,      "includePaths": [          "/blog/*"      ]  }),});console.log(await r.json());

curl -X POST https://api.snoopscan.com/v1/crawl \  -H 'Authorization: Bearer sk_YOUR_KEY' \  -H 'Content-Type: application/json' \  -d '{    "url": "https://example.com",    "limit": 100,    "maxDepth": 3,    "includePaths": [        "/blog/*"    ]}'

Returns job id · poll or webhook · from 1 credit a page.

Parameters

Field	Type	Default	What it does
`url` required	string	—	The page to start from. An absolute http or https URL.
`allowBackwardLinks`	boolean	false	Follow links above the starting path on the same host.
`allowExternalLinks`	boolean	false	Follow links onto other domains. Off by default, since with a high `limit` a crawl can reach many unrelated sites.
`deduplicateSimilarURLs`	boolean	true	Collapse URLs that differ only in tracking parameters or a trailing slash.
`delay`	integer	0	Minimum milliseconds between requests to the same host.
`excludePaths`	array of string	—	Skip URLs whose path and query match any of these regexes. Wins over `includePaths`.
`ignoreQueryParameters`	boolean	false	Treat URLs that differ only in their query string as the same page.
`ignoreSitemap`	boolean	false	Find pages by following links only, without reading the site's sitemap.
`includePaths`	array of string	—	Only crawl URLs whose path and query match one of these regexes, e.g. `^/blog/`. Empty means every path.
`limit`	integer	100	The most pages to crawl, 1 to 100,000.
`maxConcurrency`	integer	5	How many pages to fetch at once, 1 to 50.
`maxDepth`	integer	3	How many links deep to follow from the start page. 0 is the start page alone.
`respectRobots`	boolean	false	Honor the site's robots.txt for this job.
`scrapeOptions`	object	—	How each page is fetched and what comes back: the same options /v1/scrape takes.
`webhook`	object	—	Where to send job events as the crawl progresses.

Response

A success returns {"success": true, "data": {…}}. data holds the formats you asked for, the page's metadata, and a cost object with what the call was charged. A failure returns {"success": false, "error": {…}} with an error code. See Errors.

Failed requests are not charged. Re-reading a page you already fetched, within maxAge, is free. See Credits for every rate.

Scrape

Map