Rendering, Waits, and Proxies¶

This page focuses on the browser-style controls available through the unified scrape endpoint.

Render JavaScript-heavy pages¶

from toolkitapi import Scrape

with Scrape(api_key="tk_...") as scrape:
    result = scrape.fetch(
        url="https://toolkitapi.io/app",
        render_js=True,
        output="html",
    )

    print(result["js_rendered"])

Wait until the page is ready¶

Use one of these browser wait modes:

load
domcontentloaded
networkidle

from toolkitapi import Scrape

with Scrape(api_key="tk_...") as scrape:
    result = scrape.render_page(
        url="https://toolkitapi.io/dashboard",
        wait_until="load",
        output="clean",
    )

Wait for a DOM element¶

from toolkitapi import Scrape

with Scrape(api_key="tk_...") as scrape:
    result = scrape.render_page(
        url="https://toolkitapi.io/search",
        wait_for="#results",
        wait_timeout=20000,
        output="text",
    )

Infinite-scroll style pages¶

from toolkitapi import Scrape

with Scrape(api_key="tk_...") as scrape:
    result = scrape.render_page(
        url="https://toolkitapi.io/feed",
        scroll=True,
        wait_until="networkidle",
        output="markdown",
    )

Block expensive resources¶

from toolkitapi import Scrape

with Scrape(api_key="tk_...") as scrape:
    result = scrape.render_page(
        url="https://toolkitapi.io",
        block_resources=["image", "font", "media"],
        output="text",
    )

Use stealth mode¶

from toolkitapi import Scrape

with Scrape(api_key="tk_...") as scrape:
    result = scrape.render_page(
        url="https://toolkitapi.io/protected",
        stealth=True,
        output="html",
    )

Send custom headers¶

from toolkitapi import Scrape

with Scrape(api_key="tk_...") as scrape:
    result = scrape.fetch(
        url="https://httpbin.org/headers",
        headers={
            "Accept-Language": "en-GB,en;q=0.9",
            "X-Debug": "true",
        },
        output="html",
    )

Pass custom cookies¶

from toolkitapi import Scrape

with Scrape(api_key="tk_...") as scrape:
    result = scrape.fetch(
        url="https://toolkitapi.io/account",
        render_js=True,
        cookies=[
            {
                "name": "sessionid",
                "value": "abc123",
                "domain": ".toolkitapi.io",
            }
        ],
        output="text",
    )

Reuse a browser session¶

from toolkitapi import Scrape

with Scrape(api_key="tk_...") as scrape:
    scrape.render_page(
        url="https://toolkitapi.io/login",
        session_name="saved-session",
    )

    page = scrape.render_page(
        url="https://toolkitapi.io/account",
        session_name="saved-session",
        output="text",
    )

    print(page["content"])

Proxy options¶

from toolkitapi import Scrape

with Scrape(api_key="tk_...") as scrape:
    us_page = scrape.fetch(
        url="https://toolkitapi.io",
        proxy="US",
        output="html",
    )

    dc_page = scrape.fetch(
        url="https://toolkitapi.io",
        proxy="datacenter",
        output="html",
    )

Notes¶

Use proxy values like US or GB for country-targeted requests
Use datacenter when you want a standard shared proxy mode
Prefer wait_for or wait_until over arbitrary delays when possible
If you need HTML-to-image rendering, use the Image toolkit endpoint /v1/image/from-html alongside this one

Python Examples

Extraction & AI