Jobs and webhooks

Reading one page is immediate. Reading a site is not, so Crawl and Batch answer straight away with a job id and get on with it.

import requestsr = requests.post(    'https://api.snoopscan.com/v1/crawl',    headers={'Authorization': 'Bearer sk_YOUR_KEY'},    json={        'url': 'https://example.com',        'limit': 100,        'maxDepth': 3,        'webhook': 'https://hooks.example.com/snoop'    },)print(r.json())

const r = await fetch('https://api.snoopscan.com/v1/crawl', {  method: 'POST',  headers: { Authorization: 'Bearer sk_YOUR_KEY', 'Content-Type': 'application/json' },  body: JSON.stringify({      "url": "https://example.com",      "limit": 100,      "maxDepth": 3,      "webhook": "https://hooks.example.com/snoop"  }),});console.log(await r.json());

curl -X POST https://api.snoopscan.com/v1/crawl \  -H 'Authorization: Bearer sk_YOUR_KEY' \  -H 'Content-Type: application/json' \  -d '{    "url": "https://example.com",    "limit": 100,    "maxDepth": 3,    "webhook": "https://hooks.example.com/snoop"}'

Polling

GET /v1/crawl/{id} gives the job's status and how far it has got. GET /v1/crawl/{id}/pages pages through what has been read so far with a cursor, so you can start using results before the job finishes. Errors, if there were any, are on /errors.

Webhooks

Better: give a webhook URL when you start the job and nothing has to poll at all. We call you when the job finishes, and monitors call you when a watched page actually changed.

Every webhook is signed with your key's signing secret. Verify the signature before you trust the body — the URL is public, and a signature is the only thing separating our call from anyone else's.

Failed deliveries

Delivery is retried with a widening gap between attempts. A job's results do not depend on the webhook: if every delivery fails, the job is still finished and still readable by polling. A webhook is a convenience, never the only copy.