Description

n8n-nodes-webscraping-ai


This is an n8n community node that integrates WebScraping.AI into your n8n workflows.

WebScraping.AI provides AI-powered web scraping with Chromium JavaScript rendering, rotating datacenter/residential/stealth proxies, and built-in HTML parsing — plus LLM-powered question answering and structured field extraction on any page. This node allows you to leverage these capabilities directly in your n8n automations.

Features

Node version 2 groups the operations under resources (version 1 workflows keep working unchanged; see Node versions):

| Resource | Operation | Version 1 name |
| — | — | — |
| Web Page | Get HTML: full HTML of a page | Get HTML |
| Web Page | Get Text: visible text content | Get Text |
| Web Page | Get Selected HTML: HTML of the elements matching a CSS selector | Get Selected HTML |
| Web Page | Get Multiple Selections: HTML for several CSS selectors at once | Get Multiple Selections |
| AI | Ask Question: ask an LLM a question about a page | AI Question |
| AI | Extract Fields: extract structured fields from a page with an LLM | AI Extract Fields |
| Search | Search Google: parsed Google results for a query, from a chosen location, one or several pages at a time | Search (SERP) |
| Search | Get Locations: look up location names for Search Google (free) | Search Locations |
| Structured Data | Get: structured JSON for any public web page from its normal URL | Get Structured Data |
| Account | Get: account quota and usage | Get Account Info |

The sections below use the version 1 names.

Installation

Follow the installation guide in the n8n community nodes documentation.

Install from npm

npm install n8n-nodes-webscraping-ai

Manual Installation

1. Clone this repository
2. Run npm install to install dependencies
3. Run npm run build to build the node
4. Copy the dist folder to your n8n custom nodes directory

Prerequisites

You need to have an account with WebScraping.AI and obtain an API key from your dashboard. Sign up to get started — a free trial, no credit card required.

Credentials

Create a new credential of type “WebScraping.AI API” in n8n and enter your API key.

Operations

AI Question

Ask AI a question about a webpage. The LLM will analyze the page content and provide an answer.

Parameters:

• URL (required): The webpage to analyze
• Question (required): The question to ask about the page
• Response Format: JSON or Text

AI Extract Fields

Extract structured data from a webpage using AI.

Parameters:

• URL (required): The webpage to scrape
• Fields to Extract (required): JSON object defining fields and their descriptions

– Example: {"title":"Main product title","price":"Current product price"}

Get HTML

Get the full HTML content of a webpage.

Parameters:

• URL (required): The webpage to retrieve
• Response Format: JSON or Text
• Return Script Result: Return result of custom JavaScript execution instead of HTML

Get Text

Get the visible text content of a webpage.

Parameters:

• URL (required): The webpage to retrieve
• Text Format: Plain, JSON, or XML
• Return Links: Include links in JSON response (works only with JSON format)

Get Selected HTML

Get HTML content of selected page areas by CSS selector.

Parameters:

• URL (required): The webpage to retrieve
• CSS Selector: The selector for target elements
• Response Format: JSON or Text

Get Multiple Selections

Get HTML content of multiple page areas by CSS selectors.

Parameters:

• URL (required): The webpage to retrieve
• CSS Selectors: Array of CSS selectors in JSON format

– Example: ["h1", ".price", "#description"]

Get Account Info

Get your API account quota and usage information.

Returns:

• Email address
• Remaining API credits
• Next billing cycle reset time
• Remaining concurrent requests

Search (SERP)

Get parsed search engine results for a query. Query-shaped rather than URL-shaped: WebScraping.AI handles proxy routing and parsing, so the scraping Advanced Options below don’t apply. Billed per results page returned (see pricing); failed searches are not charged.

Every search runs from one fixed location: Location, Latitude and Longitude, or UULE, otherwise the capital of the Country. So repeats and later pages are consistent, and Country defaults to the location’s country. Pages fetches up to 5 consecutive pages in one call as one de-duplicated list.

Parameters:

• Query (required): The search query, e.g. coffee machines
• Search Options:

– Country: Two-letter country code for the search (Google gl). Defaults to the country of Location, Latitude/Longitude or UULE, otherwise us
– Engine: Search engine to query (currently google, the default)
– Language: Two-letter language code for the results (Google hl, default en)
– Latitude / Longitude: Search origin in decimal degrees (set both; sent as plain decimals with up to 7 fraction digits). A number or a decimal string from an expression; a blank value is an error, so leave the option out when there is none
– Location: Where the search runs from, e.g. Austin,Texas,United States, Austin, TX or an id from Search Locations. Large cities also match local names, e.g. Москва. An unknown name returns a 400 that is not charged
– Page: Results page number, 10 results per page (default 1)
– Pages: Number of consecutive pages to fetch starting at Page, 1 to 5 (default 1). Duplicates across pages are removed, position counts through the combined list and each result gets a page field
– Radius (Meters): Radius around Location or Latitude/Longitude, 1 to 1000
– UULE: Google-encoded location (w+CAIQICI… or a+…)

Location, Latitude/Longitude and UULE are alternatives: use one of them. Invalid combinations and out-of-range values are rejected by the API with a 400 that is not charged.

Returns JSON with searchparameters (including locationused, the catalog place searched from: the Location you gave, a UULE’s place, or the Country’s capital when none is set, absent for Latitude/Longitude searches; locationrequested, the Location or UULE as sent; lat/lon, the point searched from; and radius/pages when set), searchinformation, organicresults (each with position, title, link, domain, displayedlink, and optional snippet/date; page when Pages is more than 1), optional relatedsearches, and pagination (with pagesreturned when Pages is more than 1).

Search Locations

Look up names for Search (SERP)’s Location option. Free, no credits used.

Parameters:

• Query: Part of a location name in English, e.g. austin. Without it, the most populous locations are returned
• Limit: Maximum number of locations to return (default 50). The API returns at most 10, so a larger Limit is sent as 10

Returns one item per location, most populous first, with id, googleid, googleparentid, name, canonicalname, countrycode, targettype (e.g. City, State, Country), reach (approximate population) and gps ([longitude, latitude]). Pass canonical_name (or id) as Location.

Get Structured Data

Get structured JSON for any public web page from its normal URL. Some popular sites have dedicated parsers with fixed fields; for other sites, an AI model learns the structure of each kind of page once, and pages of that kind then return data straight away for every user. The node never checks the URL itself. Priced per request (see pricing), including pages that parse empty (parsefailed) or no longer exist (notfound). Failed fetches are not charged. The scraping Advanced Options below don’t apply.

• First request for a new kind of page: on a site without a dedicated parser, the first request for a new kind of page also learns the page’s structure, so it can take up to about 45 seconds.
• 503 structureprocessing: learning the structure took longer than the request, or another request is already learning that site. The API returns HTTP 503 with {"message": "...", "errorcode": "structureprocessing", "retryafter": 30}. It is not charged, and a retry after retry_after seconds can return data (if learning failed meanwhile, it may learn again or return the 422 below). Like other API errors, it fails the node, or with “Continue on Fail” outputs { "error": "We're still processing the data structure of this site. Retry in about 30 seconds; …" }. n8n’s “Retry On Fail” waits at most 5 seconds between tries, which is too short; instead, enable “Continue on Fail”, route items that have an error field through a Wait node of about 30 seconds, and run Get Structured Data again (or use an error workflow).
• 422 structure_unavailable: no structured data could be found on the page (e.g. an empty page or a login wall). It is not charged; use AI Extract Fields instead. Like other API errors, it fails the node, or with “Continue on Fail” outputs { "error": "" }.
• Redirects: redirects within the same site (http→https, www, trailing slash) are followed; a redirect to another site returns HTTP 500 with errorcode targetredirect (the target is in the message), not charged.
• 429 rate_limited: too many new kinds of pages started in an hour.

Parameters:

• URL (required): The page’s normal URL, e.g. https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html. Surrounding whitespace is trimmed; otherwise it is sent unchanged.
• Country: Two-letter country code of the proxy used to fetch the page, us by default.
• Transcript: Video pages on sites with a dedicated parser only. Also fetch the video’s transcript into data.transcript. It’s null when no matching captions are available. If the transcript fetch itself fails, the whole request fails with a 500 and is not charged.
• Transcript Language: With Transcript on. Caption language to pick, e.g. en or de. Without it, English is preferred, then the first available track. If the video has no captions in that language, data.transcript is null.
• Extra Parameters: Name/value pairs sent to the API as-is, for parameters added after this node version (e.g. timeout, jstimeout, waitfor or device, which the API accepts for pages without a dedicated parser). There are no js or proxy options: /data renders pages with JavaScript where the site needs it and always uses proxy=auto. url, api_key, a name that repeats one of the fields above, or a name used in two rows is an error.

Returns JSON with requestparameters (url, provider: the site the page belongs to, e.g. toscrape.com; type: the kind of page, e.g. product), parsestatus (ok, parsefailed or notfound) and data, whose snake_case fields depend on provider and type (data can be null). For example:

{
  "request_parameters": {
    "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "provider": "toscrape.com",
    "type": "product"
  },
  "parse_status": "ok",
  "data": { "title": "A Light in the Attic", "price": 51.77, "currency": "GBP" }
}

Advanced Options

All URL-based scraping operations support these advanced options:

Browser & Rendering

• Enable JavaScript: Execute on-page JavaScript (default: true)
• JavaScript Timeout: Max JS rendering time in ms (default: 2000)
• Wait For Selector: CSS selector to wait for before returning content
• Custom JavaScript: Execute custom JavaScript code on the page
• Device: Desktop, Mobile, or Tablet emulation

Proxy Settings

• Proxy Type: Datacenter or Residential
• Country: Select proxy country (US, GB, DE, IT, FR, CA, ES, RU, JP, KR, IN)
• Custom Proxy: Use your own proxy in “http://user:password@host:port” format

Request Configuration

• Headers: Custom HTTP headers as JSON object
• Timeout: Max page retrieval time in ms (default: 10000, max: 30000)
• Error on 404: Return error on 404 status
• Error on Redirect: Return error on redirects

Usage Examples

Extract Product Data from E-commerce Site

1. Add WebScraping.AI node to your workflow
2. Select resource “AI”, operation “Extract Fields”
3. Enter product URL
4. Set Fields to Extract:

{
  "title": "Product title",
  "price": "Current price",
  "description": "Product description",
  "rating": "Product rating",
  "availability": "In stock or out of stock"
}

Monitor Website Content Changes

1. Use resource “Web Page”, operation “Get Text” with JSON format
2. Store the result in a database
3. Schedule the workflow to run periodically
4. Compare new results with previous ones to detect changes

Scrape Google Search Results

1. Use resource “Search”, operation “Search Google”
2. Enter the search query (e.g., coffee machines)
3. Optionally set Country, Language, and Page under Search Options
4. Split out organic_results to process each result (position, title, link, snippet)

Search From a City, Two Pages at Once

1. Optionally use Search → “Get Locations” with Query austin and pick the canonical_name you want (e.g. Austin,Texas,United States)
2. Use Search → “Search Google” with Query coffee machines
3. Under Search Options set Location to Austin, TX (or the canonical_name from step 1) and Pages to 2
4. organic_results holds both pages as one list; each result’s page says where it came from

Get Structured Data for a Product Page

1. Use resource “Structured Data”, operation “Get”
2. Enter the page URL (e.g., https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html)
3. If the node fails with a 503 structure_processing (or, with “Continue on Fail”, outputs an error), wait about 30 seconds and run it again
4. Read data.title, data.price, data.currency and so on from the output

Node versions

• Version 2 (default for new nodes since 1.4.0): a Resource selector (Web Page, AI, Search, Structured Data, Account) with an Operation list per resource. Fields, options and output are the same as in version 1.
• Version 1: one flat Operation list. Workflows created before 1.4.0 stay on version 1 and behave as before. To move one to version 2, add a new WebScraping.AI node and copy the fields over.

Error Handling

The node handles the following error scenarios:

• 400: Parameter validation errors – Check your input parameters
• 422: Get Structured Data found no structured data on the page (structure_unavailable, not charged) – Use AI Extract Fields instead
• 503: Get Structured Data is still learning the page’s structure (structure_processing, not charged) – Wait about 30 seconds and run it again
• 402: Billing/quota exceeded – Check your account balance
• 403: Invalid API key – Verify your credentials
• 429: Rate limiting – Reduce request frequency
• 500: Target page errors – The target website returned an error
• 504: Timeout errors – Increase timeout parameter

Enable “Continue on Fail” in the node settings to handle errors gracefully in your workflow.

Live smoke test (development)

scripts/smoke.ts sends one live API request for each of the 10 operations on https://example.com (the SERP operation searches for “coffee machines”; Search Locations looks up austin; Get Structured Data fetches a page on a site with a dedicated parser), a second SERP request with Location Austin, TX and Pages 2, plus one Get Structured Data request on https://example.com/ (a site without a dedicated parser) that passes if it returns a success envelope with requestparameters, a 503 with errorcode structureprocessing (learning the structure outlasted the request), or a 422 with errorcode structureunavailable. Each request is built by the node’s real buildRequest helper. The script then adds apikey to the query string, as the credential does, and encodes the query string the way n8n’s httpRequest helper does with the arrayFormat: 'repeat' that buildRequest sets: arrays go out as repeated keys (selectors=h1&selectors=p; the API ignores bracketed selectors[0]= arrays), objects as bracketed keys (fields[title]=..., headers[X-Foo]=...). It checks results, not just status codes: SERP must return non-empty organicresults with searchparameters.q matching the query (and, for the Austin request, searchparameters.locationused containing Austin), Search Locations a non-empty list whose first canonicalname contains Austin, the dedicated-parser Get Structured Data request must return parsestatus ok, the expected requestparameters.provider and a non-empty data.title, Selected Multiple must return at least one non-empty inner array (the API answers mis-encoded selectors with [[]]), AI Fields a non-empty object, and the rest a non-empty body. It prints ok/FAIL for each operation (never the API key) and exits non-zero if any operation fails. An optional WEBSCRAPINGAIAPIURL overrides the base URL and must include the scheme (http:// or https://).

WEBSCRAPINGAIAPI_KEY=your-key npm run smoke

It uses real credits (see pricing; page operations run with js: false and the datacenter proxy to keep the cost down, SERP is billed per results page returned, Search Locations is free, and the https://example.com/ structured-data call is not charged when it returns 503 or 422). The script is compiled with tsc -p tsconfig.smoke.json into .smoke/, which is gitignored. It isn’t part of dist/ or the npm package, and it sits outside the nodes/ and credentials/ paths that the n8n community-package scanner checks. npm run lint does check it; a header comment allows the console and process.env a dev script needs.

Development

Built with n8n’s @n8n/node-cli (n8n-node). Its dev dependencies need Node.js 24.

npm install
npm run dev      # start n8n locally with this node, rebuilding on change
npm run lint     # n8n's community-node lint rules (strict mode)
npm test         # unit tests (test/)
npm run build    # compile to dist/

Releases are published to npm only from GitHub Actions (.github/workflows/release.yml, run when a GitHub Release is published), with npm provenance, as n8n requires for verified community nodes.

Resources

• WebScraping.AI — features, pricing, signup
• WebScraping.AI Documentation
• API Reference
• n8n Community Nodes
• Other official clients: Python · JavaScript · Ruby · PHP · Go · Java · .NET · CLI · MCP server

License

MIT

Support

For issues related to this n8n node, please open an issue on GitHub.

For WebScraping.AI API issues, contact [email protected].