eBay, with its vast and dynamic marketplace, is a treasure trove of data. Scraping eBay can help you gather information on product prices, seller ratings, and customer reviews. Whether you’re an e-commerce enthusiast, a data analyst, or a business owner, having access to eBay data can provide valuable insights. In this guide, we will show you how to scrape eBay using JavaScript, one of the most versatile programming languages for web scraping. Additionally, we will introduce you to the Crawlbase Crawling API, a powerful tool to make the scraping process efficient and reliable.

Table of Contents:

Understanding eBay’s Website Layout

Understanding eBay Website Structure

Understanding eBay’s website layout is a fundamental aspect of successful web scraping. eBay, being a vast and dynamic online marketplace, organizes its data in a hierarchical and complex manner. Here’s a glimpse of what you need to grasp:

  1. Categories and Listings: eBay categorizes its products into various categories and subcategories. These act as primary navigation points for users. Each category contains numerous listings, each representing a specific product.

  2. Search and Filters: eBay’s search functionality enables users to find products efficiently. Filters further refine search results based on criteria like price, location, and seller rating. These filters significantly impact how you extract data.

  3. Product Pages: When you click on a listing, it takes you to a product page. This page contains detailed information about the item, including images, price, seller details, product description, and more. Scraping this information is often the primary goal.

  4. Pagination: eBay divides search results into multiple pages. Understanding how pagination works is crucial to collecting data comprehensively, as you’ll need to navigate through various pages to gather all relevant information.

  5. Dynamic Content: eBay employs JavaScript to load certain page parts dynamically. To scrape such content, you may use techniques like web scraping with headless browsers or API calls.

Structure of an eBay page

eBay’s website comprises both the product and search results pages, playing crucial roles in the platform’s functionality. Understanding their layout is vital for users and sellers, influencing their eBay journey from product discovery to successful transactions.

1. Product Page

A typical eBay webpage is filled with a wide range of valuable data. Using the Apple iPhone 14 Pro Max product page example shown below, we can see numerous categories that can be extracted, including:

  • Product title
  • Product description
  • Product image
  • Product rating
  • Price
  • Availability
  • Customer reviews
  • Shipping cost
  • Seller Information
  • Delivery time
eBay Product Page

2. Search Results Page

Once you enter a specific keyword, such as “Smart Phones,” you will be directed to the search results page. This page will be similar to the one shown in the image below.

eBay Search Results Page

Importantly, it is possible to extract all the products listed under “Smart Phones,” including their links, titles, prices, ratings, and images. Additionally, this search process provides valuable data for various research and analysis purposes, enabling users to make informed decisions or conduct market research effectively.

Now that you have understood the significance of the data we can obtain from eBay let’s kick off our concise, step-by-step guide on how to fetch, retrieve, and parse eBay data using JavaScript and Crawlbase Crawling API.

Setting Up the Environment

Step 1: Register for an account on Crawlbase and obtain your personal token. You can get this token from the account documentation section of your Crawlbase account.

Step 2: Select the particular eBay product page you wish to scrape. In this example, we have picked the eBay product page featuring the Apple iPhone 14 Pro Max. It’s important to choose a product page that contains various elements to demonstrate the versatility of the scraping procedure.

eBay Product Page

Step 3: Now install the Crawlbase Node.js library. Please follow these steps:

  • First, make sure that you have installed Node.js on your system. If it’s not installed, you can download and install it from Node.js official website.
  • After confirming that Node.js is installed, install the Crawlbase Node.js library using npm (Node Package Manager). Open your terminal and run the following command:
1
npm install crawlbase

This will download and install the Crawlbase Node.js library on your system, making it available for your scraping project.

Step 4: To create an "ebay-product-scraper.js" file, you can use a text editor or an integrated development environment (IDE) of your choice. Here’s how you can create the file using a common command-line:

1
touch ebay-product-scraper.js

After running this command, you will have an empty "ebay-product-scraper.js" file created in the specified directory. You can then open this file with your preferred text editor to add your JavaScript code.

Step 5: To set up the Crawlbase Crawling API, you’ll need to define the necessary parameters and endpoints for the API to work correctly. To get started, ensure you have created a file named "ebay-product-scraper.js" in the above step. Then, paste the script provided below into this file. Finally, run the script in your terminal by using the command node ebay-product-scraper.js.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
// Import the Crawling API
const { CrawlingAPI } = require('crawlbase');

// Set your Crawlbase token
const api = new CrawlingAPI({ token: 'YOUR_CRAWLBASE_TOKEN' });

// URL of the eBay page to scrape
const eBayPageURL = 'https://www.ebay.com/itm/354586733872';

// Get request to crawl the URL
api
  .get(eBayPageURL)
  .then((response) => {
    if (response.statusCode === 200) {
      console.log(response.body);
    }
  })
  .catch((error) => console.error);

The above script explains how to use Crawlbase’s Crawling API to access and extract data from an eBay product page. This is achieved by configuring the API token, specifying the target URL, and initiating a GET request. The result of running this code will be the unprocessed HTML content of the designated eBay product page, which will be presented in the console as shown below:

HTML response of eBay Product Page

Scrape meaningful data with Crawlbase Scrapers

In the previous example, we discussed how we could obtain the basic structure of eBay product data, essentially the HTML of the page. However, there are times when we don’t require this raw data; instead, we are interested in extracting specific and important information from the page. No need to worry! Crawlbase’s Crawling API includes built-in eBay scrapers known as “ebay-serp” and “ebay-product” to help us extract valuable content. We’ll discuss both scrapers one by one:

Crawlbase “ebay-product” Scraper

We must include a “scraper” parameter with “ebay-product” in the JavaScript code when using the Crawling API to enable this functionality. This “scraper” parameter allows us to extract the relevant portions of the page in a JSON format. We will be making changes to the same file, "ebay-product-scraper.js". Let’s take a look at the following example to get a clearer understanding:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
// Import the Crawling API
const { CrawlingAPI } = require('crawlbase');

// Set your Crawlbase token
const api = new CrawlingAPI({ token: 'YOUR_CRAWLBASE_TOKEN' });

// URL of the eBay page to scrape
const ebayPageURL = 'https://www.ebay.com/itm/354586733872';

// options for Crawling API
const options = {
  scraper: 'ebay-product',
};

// Get request to crawl the URL
api
  .get(ebayPageURL, options)
  .then((response) => {
    if (response.statusCode === 200) {
      // Parse the JSON response and print it
      console.log(JSON.parse(response.body));
    }
  })
  .catch((error) => {
    console.error('API request error:', error);
  });

The above JavaScript code uses Crawlbase’s Crawling API to scrape data from a specific eBay product page. The target eBay page URL is defined, and scraping options are configured, specifying the ‘ebay-product’ scraper. A GET request is then made to the URL, and upon receiving a successful response with a status code of 200, the code parses and prints the extracted data in JSON format to the console.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
{
  "original_status": 200,
  "pc_status": 200,
  "url": "https://www.ebay.com/itm/354586733872",
  "body": {
    "title": "Apple iPhone 14 Pro Max A16 128GB 256GB 512GB 1TB Unlocked New Sealed",
    "price": { "value": "$1,429.49", "currency": "USD" },
    "url": "https://www.ebay.com/str/beyondtheworld?_trksid=p4429486.m145687.l149086",
    "sellersDescriptionURL": "https://vi.vipr.ebaydesc.com/ws/eBayISAPI.dll?ViewItemDescV4\u0026item=354586733872\u0026t=0\u0026category=9355\u0026seller=beyond_theworld\u0026excSoj=1\u0026excTrk=1\u0026lsite=0\u0026ittenable=false\u0026domain=ebay.com\u0026descgauge=1\u0026cspheader=1\u0026oneClk=2\u0026secureDesc=1",
    "notification": "",
    "condition": "New: A brand-new, unused, unopened, undamaged item in its original packaging (where packaging is ...  Read moreabout the conditionNew: A brand-new, unused, unopened, undamaged item in its original packaging (where packaging is applicable). Packaging should be the same as what is found in a retail store, unless the item is handmade or was packaged by the manufacturer in non-retail packaging, such as an unprinted box or plastic bag. See the seller's listing for full details. See all condition definitionsopens in a new window or tab ",
    "images": [
      "https://i.ebayimg.com/images/g/~C4AAOSwT-Rj7H8k/s-l500.jpg/s-l600.jpg",
      "https://i.ebayimg.com/images/g/uNsAAOSw~VFj7H8f/s-l1600.jpg/s-l600.jpg",
      "https://i.ebayimg.com/images/g/mWsAAOSwtb9j7IBC/s-l1600.jpg/s-l600.jpg",
      "https://i.ebayimg.com/images/g/PjQAAOSw75hj7IBD/s-l1600.jpg/s-l600.jpg",
      "https://i.ebayimg.com/images/g/4BYAAOSwjn9j7IBF/s-l1600.jpg/s-l600.jpg",
      "https://i.ebayimg.com/images/g/bqMAAOSwjLhj7IBI/s-l1600.jpg/s-l600.jpg"
    ],
    "shippingSummary": "US $8.70",
    "shipsTo": "Albania, Algeria, Angola, Antigua and Barbuda, Argentina, Armenia, Australia, Austria, Azerbaijan Republic, Bahrain, Bangladesh, Belarus, Belgium, Benin, Bermuda, Bolivia, Bosnia and Herzegovina, Botswana, Brunei Darussalam, Bulgaria, Burkina Faso, Burundi, Cambodia, Cameroon, Cape Verde Islands, Central African Republic, Chad, Chile, Colombia, Comoros, Costa Rica, Cyprus, Czech Republic, Côte d'Ivoire (Ivory Coast), Democratic Republic of the Congo, Denmark, Djibouti, Dominica, Dominican Republic, Ecuador, Egypt, El Salvador, Equatorial Guinea, Eritrea, Estonia, Ethiopia, Fiji, Finland, Gabon Republic, Gambia, Georgia, Ghana, Greece, Greenland, Guadeloupe, Guatemala, Guinea, Guinea-Bissau, Guyana, Honduras, Hungary, Iceland, India, Indonesia, Ireland, Israel, Italy, Jamaica, Jordan, Kazakhstan, Kenya, Kuwait, Laos, Latvia, Lebanon, Lesotho, Liberia, Libya, Lithuania, Luxembourg, Macedonia, Madagascar, Malawi, Maldives, Mali, Malta, Martinique, Mauritania, Mauritius, Mayotte, Mexico, Moldova, Mongolia, Montenegro, Morocco, Mozambique, Namibia, Nepal, Netherlands, New Caledonia, New Zealand, Niger, Nigeria, Norway, Oman, Pakistan, Papua New Guinea, Peru, Poland, Portugal, Puerto Rico, Qatar, Republic of Croatia, Republic of the Congo, Reunion, Romania, Rwanda, Saint Helena, Saudi Arabia, Senegal, Serbia, Seychelles, Sierra Leone, Slovakia, Slovenia, Solomon Islands, Somalia, South Africa, Sri Lanka, Swaziland, Sweden, Switzerland, Tanzania, Togo, Trinidad and Tobago, Tunisia, Turkey, Uganda, United Arab Emirates, Uzbekistan, Venezuela, Western Sahara, Worldwide, Zambia, Zimbabwe",
    "delivery": "Estimated between Thu, Sep 21 and Fri, Sep 29 to EC2N3AREstimated delivery dates - opens in a new window or tab include seller's handling time, origin ZIP Code, destination ZIP Code and time of acceptance and will depend on shipping service selected and receipt of cleared payment cleared payment - opens in a new window or tab. Delivery times may vary, especially during peak periods.Seller ships within 3 days after receiving cleared payment.Please allow additional time if international delivery is subject to customs processing.",
    "payments": ["PayPal", "Visa", "Master Card", "American Express", "Discover"],
    "returns": "30 days returns .  Buyer pays for return shipping . ",
    "sellerInformation": {
      "id": "p4429486.m3561.l2559",
      "url": "https://www.ebay.com/usr/beyond_theworld?_trksid=p4429486.m3561.l2559",
      "feedbackScore": 0.0,
      "feedbackUrl": "#LISTING_FRAME_MODULE"
    },
    "productInformation": []
  }
}

Crawlbase “ebay-serp” Scraper

In this example, we will focus on scraping the eBay search results page, specifically the URL https://www.ebay.com/sch/i.html?_nkw=Smart+Phones. Crawlbase’s Crawling API includes a built-in scraper tailored for eBay search results pages, making extracting essential data from these pages easy. To do this, you need to modify the “scraper” parameter value in the JavaScript code provided above from “ebay-product” to “ebay-serp”. Below is an example to illustrate this change and help you better understand the process:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
// Import the Crawling API
const { CrawlingAPI } = require('crawlbase');

// Set your Crawlbase token
const api = new CrawlingAPI({ token: 'YOUR_CRAWLBASE_TOKEN' });

// URL of the eBay page to scrape
const ebayPageURL = 'https://www.ebay.com/sch/i.html?_nkw=Smart+Phones';

// options for Crawling API
const options = {
  scraper: 'ebay-serp',
};

// Get request to crawl the URL
api
  .get(ebayPageURL, options)
  .then((response) => {
    if (response.statusCode === 200) {
      // Parse the JSON response and print it
      console.log(JSON.parse(response.body));
    }
  })
  .catch((error) => {
    console.error('API request error:', error);
  });

JSON response:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
{
  "original_status": 200,
  "pc_status": 200,
  "body": {
    "title": "Apple iPhone X (iPhone 10) Smartphone 64GB/256GB Grey/Silver Unlocked Good",
    "price": {
      "value": 249.99,
      "currency": "GBP"
    },
    "url": "https://www.ebay.com/usr/thetechout?_trksid=p2047675.l2559",
    "sellersDescriptionURL": "https://vi.vipr.ebaydesc.com/ws/eBayISAPI.dll?ViewItemDescV4&item=393541114176&t=1633085604000&category=9355&seller=thetechout&excSoj=1&excTrk=1&lsite=3&ittenable=false&domain=ebay.com&descgauge=1&cspheader=1&oneClk=2&secureDesc=1&oversion=74650c1b",
    "notification": null,
    "condition": "Good - Refurbished",
    "images": [
      "https://i.ebayimg.com/images/g/47AAAOSwqGRhVuip/s-l600.jpg",
      "https://i.ebayimg.com/images/g/3o4AAOSwQp9hVuin/s-l600.jpg",
      "https://i.ebayimg.com/images/g/VgkAAOSwHN5hVuin/s-l600.jpg",
      "https://i.ebayimg.com/images/g/W-MAAOSwjthhVuin/s-l600.jpg",
      "https://i.ebayimg.com/images/g/678AAOSwI2phVuin/s-l600.jpg",
      "https://i.ebayimg.com/images/g/LkgAAOSw0JdhVt7z/s-l600.jpg"
    ],
    "options": [
      {
        "Storage Capacity": {
          "values": ["64Gb", "256Gb[out of stock]"],
          "selectedValue": "64Gb"
        }
      },
      {
        "Colour": {
          "values": ["Silver", "Space Grey"],
          "selectedValue": null
        }
      },
      {
        "Network": {
          "values": ["Unlocked"],
          "selectedValue": "Unlocked"
        }
      }
    ],
    "availableQuantity": 1,
    "soldQuantity": 47,
    "soldFeedbackUrl": "https://www.ebay.com/bin/purchaseHistory?item=393541114176&rt=nc&_trksid=p2047675.l2564",
    "soldHistory": "https://www.ebay.com/bin/purchaseHistory?item=393541114176&rt=nc&_trksid=p2047675.l2564",
    "buyerSatisfaction": "100%",
    "watchersCount": 47,
    "shippingSummary": "",
    "location": "Barnstaple, United Kingdom",
    "delivery": "Estimated between Thu. Nov. 4 and Mon. Nov. 8 Estimated delivery date help - opens a layer Estimated delivery dates - opens in a new window or tab include seller's handling time, origin ZIP Code, destination ZIP Code and time of acceptance and will depend on shipping service selected and receipt of cleared payment - opens in a new window or tab. Delivery times may vary, especially during peak periods. Includes international tracking",
    "payments": ["PayPal", "Visa", "Master Card", "American Express"],
    "returns": "60 days, buyer pays return shipping",
    "description": "Good - Fully functional with 85%+ battery health, moderate signs of use with small scratches or dents - 12 months warranty",
    "sellerInformation": {
      "id": "thetechout",
      "url": "https://www.ebay.com/usr/thetechout?_trksid=p2047675.l2559",
      "feedbackScore": 1685,
      "feedbackUrl": "https://www.ebay.com/fdbk/feedback_profile/thetechout?filter=feedback_page:RECEIVED_AS_SELLER&_trksid=p2047675.l2560"
    },
    "productInformation": [
      {
        "name": "Model",
        "value": "Apple iPhone X"
      },
      {
        "name": "Camera Resolution",
        "value": "12.0 MP"
      },
      {
        "name": "Brand",
        "value": "Apple"
      },
      {
        "name": "Chipset Model",
        "value": "Apple A11 Bionic"
      },
      {
        "name": "RAM",
        "value": "3 GB"
      },
      {
        "name": "Connectivity",
        "value": "2G, 3G, 4G, 4G+, Bluetooth, Dual-Band, GPRS, GPS, Lightning, Wi-Fi, WAP, NFC"
      }
    ]
  }
}

Value of scraping eBay

Value of scraping eBay

A typical eBay page has valuable information. It includes elements like images, pricing details, product availability, page numbers, URLs, status codes, and numerous other essential features. When you harness the power of eBay scraping to extract this data, you open up a world of possibilities for various use cases:

  1. Competitive Analysis: Data extraction from eBay pages allows you to gain insights into your competitors’ products, pricing strategies, and sales volumes. Armed with this information, you can formulate strategies to set your products apart in the market and make more informed pricing decisions.
  2. Market Research: eBay scraping empowers you to identify which products are performing well on the platform and the frequency at which they are being sold. This knowledge is invaluable for understanding market trends and consumer preferences.
  3. Product Development: Utilizing eBay scraping, you can extract data that offers insights into the products that are in high demand. This information becomes the foundation for making informed product design and development decisions.

Handling eBay’s Anti-Scraping Measures

eBay, one of the largest e-commerce platforms globally, has implemented robust anti-scraping measures to protect its data and ensure a fair marketplace. These measures are designed to prevent automated bots from scraping and potentially disrupting the platform. As a web scraper seeking valuable data from eBay, it’s essential to understand and effectively handle these anti-scraping measures.

  • One common anti-scraping measure employed by eBay is using CAPTCHAs, those annoying puzzles that require human interaction to solve. To bypass CAPTCHAs, you may use headless browsers or incorporate CAPTCHA-solving services like Crawlbase Crawling API. So, you can seamlessly bypass these obstacles and save time and effort.
  • Rate limiting is another challenge. eBay may restrict the number of requests a user can make in a given time frame. To counter this, implement rate limiting in your scraping code to ensure you don’t flood eBay’s servers with requests.
  • IP blocking is a more severe measure that eBay might resort to if it detects excessive scraping from a particular IP address. Crawlbase provides proxy management, allowing you to rotate IP addresses and avoid being throttled by rate limits. This ensures that your scraping activities remain undetected and uninterrupted.

Conclusion

The convergence of JavaScript and the Crawlbase Crawling API offers a transformative gateway for web scraping on platforms as extensive as eBay. This synergy presents an unmatched opportunity for e-commerce specialists, data analysts, and researchers to unlock insights and trends hidden within eBay’s massive marketplace. Following a structured methodology, we can efficiently extract and leverage this data for diverse applications, enriching our understanding of the e-commerce domain.

However, the essence of responsible web scraping lies in upholding ethical standards. Strict adherence to eBay’s terms of service and scraping guidelines is not just a mandate but a commitment to sustainable and respectful data extraction. In doing so, we blend the might of technology with a sense of duty, ensuring a future where data-driven insights coexist with digital respect and integrity.

Frequently Asked Questions (FAQs)

Is it possible to scrape eBay?

Yes, it is possible to scrape eBay. eBay, like many other websites, can be scraped to extract data such as product listings, prices, seller information, and more. However, when scraping eBay or any other website, it’s important to be aware of and comply with eBay’s terms of service and scraping guidelines to ensure that you are scraping responsibly and legally.

Web scraping can be legal if done for legitimate purposes, such as personal use or research. However, scraping for malicious purposes or violating eBay’s terms of service is illegal.

Does eBay allow scraping?

Scraping eBay is generally acceptable if you’re not accessing data requiring login credentials or personal information without consent. In general, web scraping is considered legal as long as it is conducted in accordance with the relevant rules and laws governing the websites being scraped and the data being collected.

How do I handle eBay’s anti-scraping measures, like CAPTCHAs and rate limiting?

eBay employs anti-scraping measures to protect its data. To overcome these, you should implement strategies such as using proxies, solving CAPTCHAs, and incorporating rate limiting into your scraping code.

Can Crawling API provide scraped content without HTML source code?

Yes, our Crawling API offers optional data scrapers designed specifically for eBay Search Engine Results Pages (SERP) and eBay product pages. You can explore the comprehensive details of using these scrapers and integrating them into your projects by visiting our documentation. Our documentation provides step-by-step guidance, code examples, and valuable tips to make your eBay data scraping experience seamless and efficient.