Scrape search results on lidl.com
The call that returns search results from lidl.com today, with the parameters, patterns and failure modes read off the request log since 14 Aug 2026. Last tested 6 Sep 2026.
97.9% success · Aug 2026No browser neededStandard price · 1 credit
About lidl.com
lidl.com describes itself as “Lidl US | Grocery Store | More to Value”.
Its robots.txt names 1 sitemap and disallows 12 paths.
Measured on Crawlbase
Success rate over every request from every account in August 2026: 97.9%. Patterns, countries and failures come from the request log since 14 Aug 2026; 1.1% of successful calls used the JavaScript token and the median answer took 3.7s.
Among the 9 grocery sites in the Cookbook, lidl.com ranks 6 of 9 by success rate; the category median is 98.8%. Its median answer of 3.7s is slower than the category median of 2.6s. 5 of the 9 accept the plain token, this one among them.
The call
Search result pages are the shape the accounts fetch most here, with the country and token that succeed most often. Replace the placeholder query with your own.
curl "https://api.crawlbase.com/?token=YOUR_TOKEN&country=US&url=https%3A%2F%2Fwww.lidl.com%2Fs%2Fexample-section%2Fexample-page"from crawlbase import CrawlingAPI api = CrawlingAPI({'token': 'YOUR_TOKEN'}) r = api.get('https://www.lidl.com/s/example-section/example-page', {'country': 'US'}) html = r['body']
const { CrawlingAPI } = require('crawlbase'); const api = new CrawlingAPI({ token: 'YOUR_TOKEN' }); const r = await api.get('https://www.lidl.com/s/example-section/example-page', { country: 'US' });
Parameters that matter
| Parameter | Set it to | Why |
|---|---|---|
country | US | 94.1% of successful calls set it and they succeed at 75.0%; calls without a country succeed at 100%. |
javascript | leave unset | Plain-token calls succeed at -% and make up 0.0% of the traffic. A browser costs more and, on this site, buys nothing. |
URL patterns accounts fetch
| Pattern | Share of successes | JavaScript token | Country | Query parameters |
|---|---|---|---|---|
/s/{slug}/{slug} | 92.3% | 100% | US | none |
/ | 7.7% | 100% | none | none |
Fields you get
Structured data Crawlbase found on lidl.com, by page type. These are the schema.org types in the page, the fields your parser can read straight from the JSON-LD block:
- Home:
Organization
What breaks, and the fix
Of the failures since 14 Aug 2026, the site answered:
999: 50.0% of failures 999 means the site refused the fetch. Treat it like a block: retry once with the JavaScript token and the country the successful calls use, then stop.
591: 50.0% of failures A 5xx is the site failing, not the fetch. Retry after a minute.
Run it on a schedule
Search is the expensive side of a site. Keep the query list short, run it on a cadence through the Crawler with a callback, and fetch afterwards only the result pages you need. The median answer on this site is 3.7s.
Questions
Do I need a browser to scrape lidl.com?
No. -% of plain-token calls succeed and accounts send 0.0% of their traffic without a browser.
How much does a lidl.com page cost on Crawlbase?
1 credit without a browser. lidl.com is in the standard tier.
Which country should I set for lidl.com?
US: that is what 94.1% of successful calls use.
Other pages on lidl.com
Pages lidl.com lists in its own sitemaps, fetched by Crawlbase on 6 Sep 2026 and verified in 2 consecutive runs. Each row is the cheapest call that returned the page with its content.
| Page | Example | Token | Country | Median |
|---|---|---|---|---|
Home/Structured data: Organization | lidl.com/ | Plain | none | 2.1s |
Site names identify the pages a recipe fetches. Crawlbase is not affiliated with any site listed. Use recipes within the acceptable use policy.Measured numbers from August 2026, refreshed monthly