Web Scraper
The web scraper tool uses crawl4ai to extract content from web pages with advanced anti-bot evasion and JavaScript rendering.
Built on crawl4ai, it's optimised for high-volume scraping of sites that block standard scrapers.
The seven tools
| Tool | Does |
|---|---|
scrape_url | Scrape one page, returning cleaned text |
crawl_multiple | Scrape many URLs in parallel, markdown for each |
scrape_structured | Extract structured rows with CSS selectors — a baseSelector for the repeating container, then fields within it |
scrape_with_session | A multi-step workflow on one persistent browser session; cookies, localStorage and auth survive across steps |
search_page | Scroll an infinite-scroll page repeatedly, letting new content load each cycle |
extract_links | All hyperlinks, split internal and external, with anchor text and a relevance score |
screenshot_url | A screenshot as base64 PNG, full-page optional |
Reach for scrape_url first. The others exist for the cases it cannot cover: many pages at once, a table you want as rows rather than prose, a site that needs a login carried between steps, or a feed that only loads as you scroll.
Tool: scrape_url
Scrapes a URL and returns the cleaned text content, including content from dynamically rendered JavaScript.
Arguments
| Argument | Type | Description |
|---|---|---|
url | string | The URL to scrape |
wait_for | string | Optional CSS selector to wait for before extracting |
scroll | boolean | Scroll to bottom to trigger infinite scroll loading |
Use cases
- Scraping news articles, blog posts, documentation
- Extracting content from sites with bot protection (LinkedIn, financial news sites)
- Crawling dynamic React/Vue/Angular apps
- Monitoring competitor pricing or product listings
Anti-bot evasion
The scraper uses crawl4ai's stealth features:
- Headless browser with realistic browser fingerprints
- Randomised request timing
- Cookie and session handling
- User-agent rotation
This makes it significantly more effective against sites that block requests-based scrapers or require JavaScript.
Example
Scrape the latest articles from https://news.ycombinator.com and summarise the top 5 stories.Get the pricing information from https://competitor.com/pricing.Comparison with Browser tool
See Browser → Comparison for a side-by-side comparison.
Web scraping should comply with a site's Terms of Service and robots.txt. Use this tool responsibly.
Built-in browser
A browser inside Synapse that your agents work in while you watch: take it over and hand it back, keep agents signed in to a site, and record a task once for agents and workflows to repeat.
Code Search
The code search tool gives agents semantic and lexical access to indexed repositories. It started as a single search_codebase call; it now ships as a small toolkit of eight related tools, so an agent can move from "find the file" to "open the right lines" without ever leaving the toolkit.