Working with it

Jobs and webhooks

Long crawls and batches, without polling.

Crawl and Batch run as jobs. They return a job id right away and keep working in the background.

import requestsr = requests.post(    'https://api.snoopscan.com/v1/crawl',    headers={'Authorization': 'Bearer sk_YOUR_KEY'},    json={        'url': 'https://example.com',        'limit': 100,        'maxDepth': 3,        'webhook': 'https://hooks.example.com/snoop'    },)print(r.json())

const r = await fetch('https://api.snoopscan.com/v1/crawl', {  method: 'POST',  headers: { Authorization: 'Bearer sk_YOUR_KEY', 'Content-Type': 'application/json' },  body: JSON.stringify({      "url": "https://example.com",      "limit": 100,      "maxDepth": 3,      "webhook": "https://hooks.example.com/snoop"  }),});console.log(await r.json());

curl -X POST https://api.snoopscan.com/v1/crawl \  -H 'Authorization: Bearer sk_YOUR_KEY' \  -H 'Content-Type: application/json' \  -d '{    "url": "https://example.com",    "limit": 100,    "maxDepth": 3,    "webhook": "https://hooks.example.com/snoop"}'

Returns a job id right away. Pages are added as they finish.

Polling

GET /v1/crawl/{id} returns the job's status and progress. GET /v1/crawl/{id}/pages returns the pages finished so far, with a cursor, so you can use results before the job ends. GET /v1/crawl/{id}/errors lists the pages that failed.

Webhooks

Pass a webhook URL when you start a job, and we send a POST request to it when the job finishes. Monitors send one when a watched page changes.

Every webhook is signed with your key's signing secret. Verify the signature before you trust the body.

Failed deliveries

Failed deliveries are retried with exponential backoff. If every delivery fails, the job's results are still available from the status endpoints.

Respond quickly. Return 200 as soon as you receive the webhook, then process it. A slow response counts as a failed delivery.

Mentions

Errors