Python Examples (and REST API)¶
The most common scrape workflows, using the Python SDK and raw REST calls.
Read this first: this service is site-intelligence oriented — it works with sitemaps, robots.txt, whole-domain maps, crawl jobs, and PDFs. There is no single-URL "fetch this page and give me HTML" endpoint here (
/v1/scrapeand/v1/fetchdo not exist). To grab one page, start a crawl withmax_pages: 1, or use your own HTTP client.
Base URL: https://scrape.toolkitapi.io
Auth: X-API-Key header.
| Endpoint | Method | Purpose |
|---|---|---|
/v1/scrape/sitemap |
GET |
Parse a sitemap.xml |
/v1/scrape/robots |
GET |
Parse robots.txt rules |
/v1/scrape/map |
GET |
Map every known URL on a domain |
/v1/scrape/crawl |
POST |
Start an async multi-page crawl |
/v1/scrape/crawl/{job_id} |
GET |
Poll a crawl job |
/v1/scrape/pdf |
POST |
Extract text from a PDF URL |
/v1/scrape/audit |
GET |
Audit a single page |
/v1/scrape/pagespeed |
GET |
Page speed metrics |
/v1/scrape/broken-links |
GET |
Broken link check |
/v1/scrape/keyword-density |
GET |
Keyword analysis |
/v1/scrape/mobile-friendly |
GET |
Mobile-friendliness check |
/v1/scrape/bulk-audit |
POST |
Audit many URLs at once |
/v1/scrape/compare |
POST |
Compare two pages |
Install the SDK¶
pip install toolkitapi
Discover a domain's URLs¶
GET /v1/scrape/map merges the sitemap with links found on the landing page.
curl -G "https://scrape.toolkitapi.io/v1/scrape/map" \
-H "X-API-Key: YOUR_KEY" \
--data-urlencode "url=https://toolkitapi.io" \
--data-urlencode "include_sitemap=true" \
--data-urlencode "limit=200"
import requests
r = requests.get(
"https://scrape.toolkitapi.io/v1/scrape/map",
headers={"X-API-Key": "YOUR_KEY"},
params={"url": "https://toolkitapi.io", "include_sitemap": True, "limit": 200},
timeout=60,
)
r.raise_for_status()
data = r.json()
for u in data["urls"]:
print(u)
const qs = new URLSearchParams({
url: "https://toolkitapi.io",
include_sitemap: "true",
limit: "200",
});
const resp = await fetch(`https://scrape.toolkitapi.io/v1/scrape/map?${qs}`, {
headers: { "X-API-Key": "YOUR_KEY" },
});
const data = await resp.json();
console.log(data.urls);
| Parameter | In | Type | Default | Notes |
|---|---|---|---|---|
url |
query | string | required | Any URL on the domain |
include_sitemap |
query | boolean | true |
Also merge URLs from /sitemap.xml |
limit |
query | integer | 500 |
1–5000 |
Read a sitemap¶
GET /v1/scrape/sitemap
| Parameter | In | Type | Default | Notes |
|---|---|---|---|---|
url |
query | string | required | Sitemap URL |
limit |
query | integer | 500 |
1–5000 |
discover_links |
query | boolean | false |
Also scrape the landing page for links |
r = requests.get(
"https://scrape.toolkitapi.io/v1/scrape/sitemap",
headers={"X-API-Key": "YOUR_KEY"},
params={"url": "https://example.com/sitemap.xml", "limit": 100},
timeout=60,
)
print(r.json()["urls"][:5])
Read robots.txt¶
GET /v1/scrape/robots
curl -G "https://scrape.toolkitapi.io/v1/scrape/robots" \
-H "X-API-Key: YOUR_KEY" \
--data-urlencode "url=https://example.com"
r = requests.get(
"https://scrape.toolkitapi.io/v1/scrape/robots",
headers={"X-API-Key": "YOUR_KEY"},
params={"url": "https://example.com"},
timeout=30,
)
rules = r.json()
print(rules["allow"], rules["disallow"])
Crawl a site (async job)¶
POST /v1/scrape/crawl starts a background job and returns a job_id immediately. Poll
GET /v1/scrape/crawl/{job_id} until status is completed.
JOB=$(curl -s -X POST "https://scrape.toolkitapi.io/v1/scrape/crawl" \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"start_url": "https://example.com", "max_pages": 20}' | jq -r .job_id)
curl "https://scrape.toolkitapi.io/v1/scrape/crawl/$JOB" -H "X-API-Key: YOUR_KEY"
import time
import requests
headers = {"X-API-Key": "YOUR_KEY"}
start = requests.post(
"https://scrape.toolkitapi.io/v1/scrape/crawl",
headers=headers,
json={"start_url": "https://example.com", "max_pages": 20, "output": "markdown"},
timeout=30,
)
job_id = start.json()["job_id"]
while True:
job = requests.get(
f"https://scrape.toolkitapi.io/v1/scrape/crawl/{job_id}",
headers=headers,
timeout=60,
).json()
print(job["status"], job["pages_crawled"], "/", job["pages_total"])
if job["status"] in ("completed", "failed"):
break
time.sleep(2)
for page in job.get("pages", []):
print(page["url"], len(page.get("content", "")))
const start = await fetch("https://scrape.toolkitapi.io/v1/scrape/crawl", {
method: "POST",
headers: { "X-API-Key": "YOUR_KEY", "Content-Type": "application/json" },
body: JSON.stringify({ start_url: "https://example.com", max_pages: 20 }),
});
const { job_id } = await start.json();
const job = await (
await fetch(`https://scrape.toolkitapi.io/v1/scrape/crawl/${job_id}`, {
headers: { "X-API-Key": "YOUR_KEY" },
})
).json();
console.log(job.status, job.pages_crawled);
Request body (CrawlV1Request):
| Field | Type | Default | Notes |
|---|---|---|---|
start_url |
string (URL) | required | Where to start |
max_pages |
integer | 10 |
1–100 |
max_depth |
integer | 3 |
1–10 |
include_patterns |
string[] | null | URL globs to include |
exclude_patterns |
string[] | null | URL globs to exclude |
output |
enum | markdown |
markdown, text, html |
clean |
boolean | false |
Extract main article content only |
respect_robots |
boolean | true |
Honor robots.txt |
max_pages: 1 is the closest thing to a single-page fetch.
Extract text from a PDF¶
POST /v1/scrape/pdf
curl -X POST "https://scrape.toolkitapi.io/v1/scrape/pdf" \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/report.pdf", "pages": "1-3"}'
r = requests.post(
"https://scrape.toolkitapi.io/v1/scrape/pdf",
headers={"X-API-Key": "YOUR_KEY"},
json={"url": "https://example.com/report.pdf", "pages": "1-3"},
timeout=120,
)
print(r.json()["text"][:500])
const resp = await fetch("https://scrape.toolkitapi.io/v1/scrape/pdf", {
method: "POST",
headers: { "X-API-Key": "YOUR_KEY", "Content-Type": "application/json" },
body: JSON.stringify({ url: "https://example.com/report.pdf", pages: "1-3" }),
});
const data = await resp.json();
console.log(data.text.slice(0, 500));
| Field | Type | Required | Notes |
|---|---|---|---|
url |
string (URL) | yes | PDF to download and extract |
pages |
string | no | 1-3 or 1,3,5. Omit for all pages. |
Audit a single page¶
Add format=markdown to any GET endpoint to get a human-readable report instead of JSON.
curl -G "https://scrape.toolkitapi.io/v1/scrape/audit" \
-H "X-API-Key: YOUR_KEY" \
--data-urlencode "url=https://example.com"
curl -G "https://scrape.toolkitapi.io/v1/scrape/audit" \
-H "X-API-Key: YOUR_KEY" \
--data-urlencode "url=https://example.com" \
--data-urlencode "format=markdown"
Error handling¶
| Status | When | Body |
|---|---|---|
400 |
Malformed URL, bad pages range, invalid enum |
{"detail": "..."} |
401 |
Missing/invalid X-API-Key |
{"detail": "Invalid or missing API key"} |
404 |
Unknown crawl job_id |
{"detail": "Crawl job '...' not found"} |
422 |
Body failed Pydantic validation | {"detail": [ ... ]} |
502 |
Target site unreachable | {"detail": "..."} |
504 |
Target site timed out | {"detail": "..."} |
resp = requests.post(
"https://scrape.toolkitapi.io/v1/scrape/crawl",
headers={"X-API-Key": "YOUR_KEY"},
json={"start_url": "not-a-url", "max_pages": 5},
)
if resp.status_code == 422:
print(resp.json()["detail"])
Tips¶
- Start with
map, thencrawl.mapis cheap and tells you how big the site is — use it to pick amax_pagesandinclude_patternsthat won't blow up a crawl. - Crawls are async by design. Do the
POSTonce, persistjob_id, and poll from a worker. Don't hold an HTTP request open for a big crawl. clean=trueis the RAG shortcut. It strips nav/footer/ads so you get article body only — much better for embeddings than raw HTML.- Respect robots.
respect_robotsdefaults totrue; turning it off is your call and your legal risk. - Use
format=markdownon GET endpoints while exploring — same data, far easier to read in a terminal.