Python Examples (and REST API)

The most common scrape workflows, using the Python SDK and raw REST calls.

Read this first: this service is site-intelligence oriented — it works with sitemaps, robots.txt, whole-domain maps, crawl jobs, and PDFs. There is no single-URL "fetch this page and give me HTML" endpoint here (/v1/scrape and /v1/fetch do not exist). To grab one page, start a crawl with max_pages: 1, or use your own HTTP client.

Base URL: https://scrape.toolkitapi.io Auth: X-API-Key header.

Endpoint Method Purpose
/v1/scrape/sitemap GET Parse a sitemap.xml
/v1/scrape/robots GET Parse robots.txt rules
/v1/scrape/map GET Map every known URL on a domain
/v1/scrape/crawl POST Start an async multi-page crawl
/v1/scrape/crawl/{job_id} GET Poll a crawl job
/v1/scrape/pdf POST Extract text from a PDF URL
/v1/scrape/audit GET Audit a single page
/v1/scrape/pagespeed GET Page speed metrics
/v1/scrape/broken-links GET Broken link check
/v1/scrape/keyword-density GET Keyword analysis
/v1/scrape/mobile-friendly GET Mobile-friendliness check
/v1/scrape/bulk-audit POST Audit many URLs at once
/v1/scrape/compare POST Compare two pages

Install the SDK

pip install toolkitapi

Discover a domain's URLs

GET /v1/scrape/map merges the sitemap with links found on the landing page.

curl -G "https://scrape.toolkitapi.io/v1/scrape/map" \
  -H "X-API-Key: YOUR_KEY" \
  --data-urlencode "url=https://toolkitapi.io" \
  --data-urlencode "include_sitemap=true" \
  --data-urlencode "limit=200"
import requests

r = requests.get(
    "https://scrape.toolkitapi.io/v1/scrape/map",
    headers={"X-API-Key": "YOUR_KEY"},
    params={"url": "https://toolkitapi.io", "include_sitemap": True, "limit": 200},
    timeout=60,
)
r.raise_for_status()
data = r.json()
for u in data["urls"]:
    print(u)
const qs = new URLSearchParams({
  url: "https://toolkitapi.io",
  include_sitemap: "true",
  limit: "200",
});
const resp = await fetch(`https://scrape.toolkitapi.io/v1/scrape/map?${qs}`, {
  headers: { "X-API-Key": "YOUR_KEY" },
});
const data = await resp.json();
console.log(data.urls);
Parameter In Type Default Notes
url query string required Any URL on the domain
include_sitemap query boolean true Also merge URLs from /sitemap.xml
limit query integer 500 1–5000

Read a sitemap

GET /v1/scrape/sitemap

Parameter In Type Default Notes
url query string required Sitemap URL
limit query integer 500 1–5000
discover_links query boolean false Also scrape the landing page for links
r = requests.get(
    "https://scrape.toolkitapi.io/v1/scrape/sitemap",
    headers={"X-API-Key": "YOUR_KEY"},
    params={"url": "https://example.com/sitemap.xml", "limit": 100},
    timeout=60,
)
print(r.json()["urls"][:5])

Read robots.txt

GET /v1/scrape/robots

curl -G "https://scrape.toolkitapi.io/v1/scrape/robots" \
  -H "X-API-Key: YOUR_KEY" \
  --data-urlencode "url=https://example.com"
r = requests.get(
    "https://scrape.toolkitapi.io/v1/scrape/robots",
    headers={"X-API-Key": "YOUR_KEY"},
    params={"url": "https://example.com"},
    timeout=30,
)
rules = r.json()
print(rules["allow"], rules["disallow"])

Crawl a site (async job)

POST /v1/scrape/crawl starts a background job and returns a job_id immediately. Poll GET /v1/scrape/crawl/{job_id} until status is completed.

JOB=$(curl -s -X POST "https://scrape.toolkitapi.io/v1/scrape/crawl" \
  -H "X-API-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"start_url": "https://example.com", "max_pages": 20}' | jq -r .job_id)

curl "https://scrape.toolkitapi.io/v1/scrape/crawl/$JOB" -H "X-API-Key: YOUR_KEY"
import time
import requests

headers = {"X-API-Key": "YOUR_KEY"}

start = requests.post(
    "https://scrape.toolkitapi.io/v1/scrape/crawl",
    headers=headers,
    json={"start_url": "https://example.com", "max_pages": 20, "output": "markdown"},
    timeout=30,
)
job_id = start.json()["job_id"]

while True:
    job = requests.get(
        f"https://scrape.toolkitapi.io/v1/scrape/crawl/{job_id}",
        headers=headers,
        timeout=60,
    ).json()
    print(job["status"], job["pages_crawled"], "/", job["pages_total"])
    if job["status"] in ("completed", "failed"):
        break
    time.sleep(2)

for page in job.get("pages", []):
    print(page["url"], len(page.get("content", "")))
const start = await fetch("https://scrape.toolkitapi.io/v1/scrape/crawl", {
  method: "POST",
  headers: { "X-API-Key": "YOUR_KEY", "Content-Type": "application/json" },
  body: JSON.stringify({ start_url: "https://example.com", max_pages: 20 }),
});
const { job_id } = await start.json();

const job = await (
  await fetch(`https://scrape.toolkitapi.io/v1/scrape/crawl/${job_id}`, {
    headers: { "X-API-Key": "YOUR_KEY" },
  })
).json();
console.log(job.status, job.pages_crawled);

Request body (CrawlV1Request):

Field Type Default Notes
start_url string (URL) required Where to start
max_pages integer 10 1–100
max_depth integer 3 1–10
include_patterns string[] null URL globs to include
exclude_patterns string[] null URL globs to exclude
output enum markdown markdown, text, html
clean boolean false Extract main article content only
respect_robots boolean true Honor robots.txt

max_pages: 1 is the closest thing to a single-page fetch.

Extract text from a PDF

POST /v1/scrape/pdf

curl -X POST "https://scrape.toolkitapi.io/v1/scrape/pdf" \
  -H "X-API-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/report.pdf", "pages": "1-3"}'
r = requests.post(
    "https://scrape.toolkitapi.io/v1/scrape/pdf",
    headers={"X-API-Key": "YOUR_KEY"},
    json={"url": "https://example.com/report.pdf", "pages": "1-3"},
    timeout=120,
)
print(r.json()["text"][:500])
const resp = await fetch("https://scrape.toolkitapi.io/v1/scrape/pdf", {
  method: "POST",
  headers: { "X-API-Key": "YOUR_KEY", "Content-Type": "application/json" },
  body: JSON.stringify({ url: "https://example.com/report.pdf", pages: "1-3" }),
});
const data = await resp.json();
console.log(data.text.slice(0, 500));
Field Type Required Notes
url string (URL) yes PDF to download and extract
pages string no 1-3 or 1,3,5. Omit for all pages.

Audit a single page

Add format=markdown to any GET endpoint to get a human-readable report instead of JSON.

curl -G "https://scrape.toolkitapi.io/v1/scrape/audit" \
  -H "X-API-Key: YOUR_KEY" \
  --data-urlencode "url=https://example.com"

curl -G "https://scrape.toolkitapi.io/v1/scrape/audit" \
  -H "X-API-Key: YOUR_KEY" \
  --data-urlencode "url=https://example.com" \
  --data-urlencode "format=markdown"

Error handling

Status When Body
400 Malformed URL, bad pages range, invalid enum {"detail": "..."}
401 Missing/invalid X-API-Key {"detail": "Invalid or missing API key"}
404 Unknown crawl job_id {"detail": "Crawl job '...' not found"}
422 Body failed Pydantic validation {"detail": [ ... ]}
502 Target site unreachable {"detail": "..."}
504 Target site timed out {"detail": "..."}
resp = requests.post(
    "https://scrape.toolkitapi.io/v1/scrape/crawl",
    headers={"X-API-Key": "YOUR_KEY"},
    json={"start_url": "not-a-url", "max_pages": 5},
)
if resp.status_code == 422:
    print(resp.json()["detail"])

Tips

  • Start with map, then crawl. map is cheap and tells you how big the site is — use it to pick a max_pages and include_patterns that won't blow up a crawl.
  • Crawls are async by design. Do the POST once, persist job_id, and poll from a worker. Don't hold an HTTP request open for a big crawl.
  • clean=true is the RAG shortcut. It strips nav/footer/ads so you get article body only — much better for embeddings than raw HTML.
  • Respect robots. respect_robots defaults to true; turning it off is your call and your legal risk.
  • Use format=markdown on GET endpoints while exploring — same data, far easier to read in a terminal.