Python Examples¶

If you prefer Python examples by default, this page shows the most common scrape workflows using the Toolkit API SDK.

Install the SDK¶

pip install toolkitapi

Basic HTML fetch¶

from toolkitapi import Scrape

with Scrape(api_key="tk_...") as scrape:
    result = scrape.fetch(
        url="https://toolkitapi.io",
        output="html",
    )

    print(result["status_code"])
    print(result["content"][:500])

Return Markdown content¶

from toolkitapi import Scrape

with Scrape(api_key="tk_...") as scrape:
    result = scrape.extract_markdown(
        url="https://toolkitapi.io/blog/post",
        include_links=True,
        include_tables=True,
    )

    print(result["content"])

Return plain text content¶

from toolkitapi import Scrape

with Scrape(api_key="tk_...") as scrape:
    result = scrape.page_text(
        url="https://toolkitapi.io/blog/post",
    )

    print(result["content"])

JavaScript rendering¶

from toolkitapi import Scrape

with Scrape(api_key="tk_...") as scrape:
    result = scrape.render_page(
        url="https://quotes.toscrape.com/js/",
        wait_until="networkidle",
        output="clean",
    )

    print(result["js_rendered"])
    print(result["content"])

Wait for a selector¶

from toolkitapi import Scrape

with Scrape(api_key="tk_...") as scrape:
    result = scrape.render_page(
        url="https://toolkitapi.io/product/123",
        wait_for=".price",
        wait_timeout=15000,
        output="text",
    )

    print(result["content"])

CSS selector extraction¶

from toolkitapi import Scrape

with Scrape(api_key="tk_...") as scrape:
    result = scrape.css_extract(
        url="https://toolkitapi.io/product/123",
        render_js=True,
        selectors={
            "title": "h1",
            "price": ".price",
            "buy_link": {
                "selector": ".buy-now",
                "attr": "href"
            }
        },
    )

    print(result["selectors"])

Structured metadata extraction¶

from toolkitapi import Scrape

with Scrape(api_key="tk_...") as scrape:
    result = scrape.fetch(
        url="https://toolkitapi.io",
        output="markdown",
        extract={
            "meta_tags": True,
            "link_preview": True,
            "links": True,
            "images": True,
        },
    )

    print(result.get("meta_tags"))
    print(result.get("link_preview"))

AI extraction¶

from toolkitapi import Scrape

with Scrape(api_key="tk_...") as scrape:
    result = scrape.ai_extract(
        url="https://toolkitapi.io/product/123",
        render_js=True,
        prompt="Extract the product name, price, and availability.",
        schema={
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "price": {"type": "string"},
                "availability": {"type": "string"}
            }
        },
    )

    print(result.get("ai_extract"))

Next steps¶

For rendering, timing, cookies, sessions, and proxies, see Rendering, Waits, and Proxies
For structured extraction and LLM workflows, see Extraction and AI
For sitemap, robots, crawl, and PDF flows, see Crawl and Site Discovery

Web Scraping Toolkit

Rendering & Proxies