Endpoints

Crawl

A documentation site, a help centre or a whole blog, page by page, with depth and path filters. Poll the job or take a webhook.

POST /v1/crawl

Request

Every tool takes the same shape: a URL or a list, a few options, and back comes data plus what the call cost.

import requestsr = requests.post(    'https://api.snoopscan.com/v1/crawl',    headers={'Authorization': 'Bearer sk_YOUR_KEY'},    json={        'url': 'https://example.com',        'limit': 100,        'maxDepth': 3,        'includePaths': [            '/blog/*'        ]    },)print(r.json())

const r = await fetch('https://api.snoopscan.com/v1/crawl', {  method: 'POST',  headers: { Authorization: 'Bearer sk_YOUR_KEY', 'Content-Type': 'application/json' },  body: JSON.stringify({      "url": "https://example.com",      "limit": 100,      "maxDepth": 3,      "includePaths": [          "/blog/*"      ]  }),});console.log(await r.json());

curl -X POST https://api.snoopscan.com/v1/crawl \  -H 'Authorization: Bearer sk_YOUR_KEY' \  -H 'Content-Type: application/json' \  -d '{    "url": "https://example.com",    "limit": 100,    "maxDepth": 3,    "includePaths": [        "/blog/*"    ]}'

Returns job id · poll or webhook · from 1 credit a page.

Parameters

Read from the engine itself, so this table is the request it actually validates.

Field	Type	Default	What it does
`url` required	string	—	—
`allowBackwardLinks`	boolean	false	—
`allowExternalLinks`	boolean	false	—
`deduplicateSimilarURLs`	boolean	true	—
`delay`	integer	0	—
`excludePaths`	array of string	—	—
`ignoreQueryParameters`	boolean	false	—
`ignoreSitemap`	boolean	false	—
`includePaths`	array of string	—	—
`limit`	integer	100	—
`maxConcurrency`	integer	5	—
`maxDepth`	integer	3	—
`respectRobots`	boolean	false	—
`scrapeOptions`	object	—	—
`webhook`	object	—	—

Response

A success is always {"success": true, "data": {…}}. data carries what you asked for in formats, the page's metadata, and a cost object saying what the call was charged. A failure is {"success": false, "error": {…}} with a code you can branch on — see Errors.

Nothing is charged for a request that failed. A page you already fetched, re-read within maxAge, is free; one served from the shared index costs a single credit. The cost object says which it was.

Scrape

Map