API Performance & Optimization

1. Use Basic Mode When Possible

Basic mode is 10x faster than deep mode (800ms vs 8s median). Only use deep mode when you need multi-page crawling.

2. Cache Audit Results

Audit results rarely change once completed. Cache them aggressively:

• Cache completed audits for 1-24 hours depending on update frequency
• Use ETags or Last-Modified headers for conditional requests
• Store job_id mappings to avoid duplicate audits for same URL

3. Use Webhooks Instead of Polling

Polling wastes API quota and adds latency. Webhooks deliver results instantly:

4. Batch Operations

For multiple URLs, batch requests with delays to avoid rate limits:

// Bad: All at once (rate limit)
await Promise.all(urls.map(url => createAudit(url)));

// Good: Sequential with delay
for (const url of urls) {
  await createAudit(url);
  await sleep(1000); // 1 second between requests
}

// Better: Concurrent with limit
import pLimit from 'p-limit';
const limit = pLimit(2); // Max 2 concurrent
await Promise.all(
  urls.map(url => limit(() => createAudit(url)))
);

5. Use NDJSON Streaming for Large Results

For audits with 100+ issues, use the streaming endpoint to avoid timeouts:

// Stream large result sets
GET /v1/audit/{job_id}/issues/stream

// Filters available:
?severity=critical,high
?fields=message,selector,impact
?page=1&per_page=50

6. Optimize Page Weight

Audit speed correlates with page load time. Optimize your pages:

• Minimize JavaScript bundle size (<500KB recommended)
• Lazy load images and heavy resources
• Use CDN for static assets
• Reduce external dependencies (fonts, analytics)

Node.js / TypeScript

• Use undici or node-fetch for faster HTTP
• Enable HTTP/2 with keepAlive: true
• Use p-limit for concurrent request control
• Cache with Redis or in-memory LRU cache

Python

• Use httpx with async/await for parallel requests
• Enable connection pooling with limits=httpx.Limits(max_keepalive_connections=10)
• Use asyncio.Semaphore to limit concurrency
• Cache with Redis or cachetools

Ruby

• Use faraday with persistent connections
• Enable HTTP/2 with faraday-http2 adapter
• Use concurrent-ruby for parallel requests
• Cache with Rails.cache or Memcached