ScrapeGraphAI: swap your $499 scraper stack for a weekend build

Here is the fast path I used to replace a proxy-backed scraper on a public catalog. You point ScrapeGraphAI at a page, give it a plain-English prompt for the fields, and it returns clean JSON.

First install the library in a fresh virtual environment. If you already have Python 3.10+, this takes under a minute.

bash
pip install scrapegraphai

Quick verify on a stable test page. This asks for a book title and price from Books to Scrape and prints JSON. Set an OpenAI key once to get a fast first run. You can switch to a local LLM later in the full guide.

py
from scrapegraphai.graphs import SmartScraperGraph
import os
os.environ.setdefault("OPENAI_API_KEY", "sk-proj-1234567890abcdef")

graph_config = {
    "llm": {"model": "gpt-3.5-turbo", "api_key": os.environ["OPENAI_API_KEY"]},
    "embeddings": {"model": "text-embedding-3-small", "api_key": os.environ["OPENAI_API_KEY"]},
    "verbose": True,
}

g = SmartScraperGraph(
    prompt="Return JSON with keys: title, price.",
    source="https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    config=graph_config,
)
print(g.run())

You should see a small JSON like { "title": "A Light in the Attic", "price": "£51.77" }. That is the basic loop. The full guide below adds local LLMs with Ollama (a local LLM runner), browser rendering for messy pages, and retry logic.

Here is the fast path I used to replace a proxy-backed scraper on a public catalog. You point ScrapeGraphAI at a page, give it a plain-English prompt for the fields, and it returns clean JSON.

First install the library in a fresh virtual environment. If you already have Python 3.10+, this takes under a minute.

bash
pip install scrapegraphai

Quick verify on a stable test page. This asks for a book title and price from Books to Scrape and prints JSON. Set an OpenAI key once to get a fast first run. You can switch to a local LLM later in the full guide.

py
from scrapegraphai.graphs import SmartScraperGraph
import os
os.environ.setdefault("OPENAI_API_KEY", "sk-proj-1234567890abcdef")

graph_config = {
    "llm": {"model": "gpt-3.5-turbo", "api_key": os.environ["OPENAI_API_KEY"]},
    "embeddings": {"model": "text-embedding-3-small", "api_key": os.environ["OPENAI_API_KEY"]},
    "verbose": True,
}

g = SmartScraperGraph(
    prompt="Return JSON with keys: title, price.",
    source="https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    config=graph_config,
)
print(g.run())

You should see a small JSON like { "title": "A Light in the Attic", "price": "£51.77" }. That is the basic loop. The sections below add local LLMs with Ollama (a local LLM runner), browser rendering for messy pages, and retry logic.

Setup

macOS

bash
# 1) Python env
python3 -m venv .venv && source .venv/bin/activate
python -m pip install -U pip
pip install scrapegraphai

# 2) Optional: local LLMs with Ollama
brew install ollama || true
ollama serve & sleep 2
ollama pull llama3:8b

Linux

bash
# 1) Python env
python3 -m venv .venv && source .venv/bin/activate
python -m pip install -U pip
pip install scrapegraphai

# 2) Optional: local LLMs with Ollama
curl -fsSL https://ollama.com/install.sh | sh
ollama serve & sleep 2
ollama pull llama3:8b

Windows

bash
# In PowerShell
py -3 -m venv .venv
. .venv\Scripts\Activate.ps1
python -m pip install -U pip
pip install scrapegraphai

# Optional: Ollama for Windows (desktop installer)
# Download and install from https://ollama.com/download
# Then run in a new terminal:
ollama serve
ollama pull llama3:8b

Verify

Open a file named quick_check.py and paste this:

py
from scrapegraphai.graphs import SmartScraperGraph
import os
os.environ.setdefault("OPENAI_API_KEY", "sk-proj-1234567890abcdef")

cfg = {
    "llm": {"model": "gpt-3.5-turbo", "api_key": os.environ["OPENAI_API_KEY"]},
    "embeddings": {"model": "text-embedding-3-small", "api_key": os.environ["OPENAI_API_KEY"]},
    "verbose": True,
}

g = SmartScraperGraph(
    prompt=(
        "Extract title, price, rating, and availability as JSON with keys: "
        "title, price, rating, availability"
    ),
    source="https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    config=cfg,
)
print(g.run())

Run it:

bash
python quick_check.py

Expected output shape:

bash
{"title": "A Light in the Attic", "price": "£51.77", "rating": "Three", "availability": "In stock"}

Values can vary a bit in spacing or casing. The keys should match.

Configuration tips

• Switch to a local LLM for zero per-token fees. Start Ollama, then set OpenAI-compatible env and a local model name. Many OpenAI clients respect these env vars.

bash
export OPENAI_API_KEY=ollama
export OPENAI_BASE_URL=http://localhost:11434/v1

Then use "model": "llama3:8b" in your llm config. Keep prompts short and temperature low for consistent fields.

• Turn on a headless browser for heavy JavaScript pages. Install Playwright and Chromium once, then pass a browser loader in your graph if your page needs rendering.

bash
pip install playwright
playwright install chromium

• Lock your output format. Ask for exact keys and types in the prompt. Example: "Return JSON with keys: title (string), price (string like £12.34), rating (one of One..Five)."

• Rate limit and retry. Wrap .run() in a small loop with time.sleep(1) on failure. Keep verbose=True during tests, then set it to False in production.

• Batch with a simple for loop and write newline-delimited JSON to disk. That keeps memory flat and lets you resume on crash.

Troubleshooting

• Error: 401 Unauthorized from OpenAI. Fix: set a valid key and model.

bash
export OPENAI_API_KEY=sk-proj-1234567890abcdef
# or point to Ollama's OpenAI endpoint
export OPENAI_BASE_URL=http://localhost:11434/v1

• Error: connection refused at http://localhost:11434. Fix: start Ollama and pull a model.

bash
ollama serve &
ollama pull llama3:8b

• Error: page content is empty or missing fields on JS-heavy sites. Fix: add a headless browser. Install Playwright and Chromium, then rerun. If the site blocks default headers, set a common user agent in your loader or run from a residential IP.

bash
pip install playwright
playwright install chromium

When it beats a proxy-heavy scraper stack

• Product pages with simple HTML. You describe fields like title, price, and availability. ScrapeGraphAI reads the DOM and hands back clean JSON. No rotating proxies or CSS selector churn.

• Public directories and event listings. The layout changes week to week. An LLM-driven extractor keeps working without you rewriting XPath rules every change.

• Market scans and price checks. You can point it at 50 to 200 public pages and finish in an afternoon. That replaces a $499 monthly proxy plus headless browser subscription for the same scope.

• Lead list enrichment. Feed it company pages and ask for name, domain, headquarters city, and contact URL. Save the output to NDJSON and ship it to your CRM later.

If you step into login walls, geo locks, or heavy bot protection, this still needs care. Keep the tool on public data and you skip the proxy bills for a wide set of tasks.

Sources

-