Agoda has millions of hotel and property listings. Whether you’re doing research, building a travel aggregator, or analyzing hotel pricing, scraping data from Agoda can be really useful for your project.

In this guide, we’ll show you how to scrape hotel data from Agoda using Python and the Crawlbase Crawling API. You’ll learn how to extract hotel names, prices, reviews, and ratings while navigating Agoda’s scroll-based pagination. We’ll also cover how to set up your Python environment and store the scraped data in a structured format like JSON.

You’ll have an Agoda web scraper that can scrape dynamic content and fetch hotel listings. In addition to covering intermediate subjects like pagination and leveraging an API to facilitate scraping, this guide is designed with beginners in mind.

Ready to get started? Here’s what we’ll cover:

Table of Contents

  1. Why Scrape Hotel Data from Agoda?
  2. Key Data Points to Extract from Agoda
  3. Crawlbase Crawling API for Scraping Hotel Listings on Agoda
  • Installing the Crawlbase Python Library
  1. Setting Up Your Python Environment
  • Installing Python and Required Libraries
  • IDE for Web Scraping
  1. Scraping Agoda Property Listings
  • Inspecting the HTML to Identify Selectors
  • Writing the Agoda Search Listings Scraper
  • Handling Scroll-Based Pagination
  • Storing Scraped Data in a JSON File
  • Complete Python Code Example
  1. Final Thoughts
  2. Frequently Asked Questions (FAQs)

Why Scrape Hotel Data from Agoda?

Agoda is a popular online travel booking site trusted by millions of users worldwide for the best hotel deals. With millions of accommodations in its database, Agoda is a treasure trove of information for businesses, researchers, and developers. By scraping Agoda hotel data, you can get insights that are hard to get manually.

Here are some reasons why scraping Agoda hotel data is valuable:

1. Market Research

Scraping Agoda helps you to analyze hotel prices, trends, and availability for travel agencies, hotel managers, and competitors to optimize pricing and find opportunities.

2. Building a Travel Aggregator

Agoda data provides real-time hotel prices, ratings, reviews, and availability for users, for travel comparison sites or apps.

3. Competitor Analysis

Hotels can use Agoda’s data to track competitors’ pricing, promotions and reviews to improve pricing and services.

4. Personalized Recommendations

Scraping Agoda’s hotel data allows developers to create personalized travel recommendations based on amenities, ratings, and location.

5. Academic Research

Researchers can use Agoda’s data for studies on tourism trends, user behavior, and hospitality for academic projects and reports.

Key Data Points to Extract from Agoda

When scraping hotel data from Agoda, focusing on the most valuable data points will help you get the most out of your efforts. Here are the key data points to extract:

Image of key data points to extract from Agoda
  1. Hotel Name – The name of the hotel helps identify properties.
  2. Price Per Night – The cost for a single night’s stay.
  3. Total Price – The full price for the entire stay, including taxes.
  4. Hotel Rating – Customer ratings to assess the quality of the property.
  5. Number of Reviews – Indicates the popularity and trustworthiness of the hotel.
  6. Location – City or neighborhood where the hotel is located.
  7. Amenities – Features like Wi-Fi, pool, gym, etc., that the hotel offers.
  8. Property Type – Type of accommodation, such as hotel, resort, or apartment.
  9. Room Availability – Information about available rooms during specific dates.
  10. Hotel Images – Visuals of the property are helpful for showcasing listings.

Crawlbase Crawling API for Scraping Hotel Listings on Agoda

Crawlbase Crawling API is the perfect tool for scraping Agoda, which has dynamic content that loads as you scroll. It handles JavaScript-rendered pages and bypasses security measures like IP blocks, so scraping is efficient and smooth.

Here’s why Crawlbase is suitable for Agoda scraping:

  • Handles Dynamic Content: Crawlbase simulates human scrolling so all hotel listings are captured as the page loads more data.
  • IP Rotation: Rotates IPs to avoid rate limits and blocks from Agoda.
  • Fast and Reliable: Scrape lots of data quickly.
  • Customizable Requests: Adjust headers, cookies, and request parameters to your needs.

Crawlbase Python Library

The Crawlbase Python library makes using the API easy. To get started, you’ll need your Crawlbase access token, which you can get by signing up for their service.

Here’s an example code to fetch data from Agoda using Crawlbase:

1
2
3
4
5
6
7
8
9
10
11
12
13
from crawlbase import CrawlingAPI

# Initialize Crawlbase API with your access token
crawling_api = CrawlingAPI({ 'token': 'YOUR_CRAWLBASE_TOKEN' })

def make_crawlbase_request(url):
    response = crawling_api.get(url)

    if response['headers']['pc_status'] == '200':
        return response['body'].decode('utf-8')
    else:
        print(f"Failed to fetch the page. Status code: {response['headers']['pc_status']}")
        return None

Note: A JS Token from Crawlbase is required to scrape JavaScript content. Crawlbase offers 1,000 requests free for its Crawling API. See the documentation for more. Next, we’ll set up your Python environment for Agoda scraping!

In the next section, we’ll set up your Python environment for Agoda scraping!

Setting Up Your Python Environment

Installing the libraries and configuring your environment are prerequisites for beginning Agoda scraping. Follow the below steps to complete the setup.

Installing Python and Required Libraries

Make sure you have Python installed on your computer. If not, download and install the latest version from the official Python website.

After installing Python, we need to install a few required libraries:

  1. Crawlbase Python Library: To interact with the Crawlbase Crawling API.
  2. BeautifulSoup: To parse HTML and extract data.

You can install these libraries using pip:

1
pip install crawlbase beautifulsoup4

IDE for Web Scraping

An Integrated Development Environment (IDE) will make coding and running your scraper easier. Popular choices for Python are:

Pick one that suits you, and you’re good to go!

Scraping Agoda Property Listings

In this section, we will scrape Agoda property listings for the city “Kuala Lumpur”. The search URL is:

Agoda Search URL for Kuala Lumpur

We will go through the following steps to scrape the listings efficiently:

Inspecting the HTML to Identify Selectors

Before we start scraping, we need to understand the Agoda search results page’s HTML structure so we can determine the selectors for the hotel data we want to extract.

  1. Open the Agoda URL: Navigate to the Agoda search results page for Kuala Lumpur.
  2. Inspect the Page: Right-click on the page and choose “Inspect” or press Ctrl + Shift + I to open the Developer Tools.
Image inspecting Agoda property HTML structure
  1. Identify Key Elements: In the page’s HTML structure, find the elements that contain hotel details. We will focus on:
  • Hotel Name: Within an <h3> element with data-selenium="hotel-name".
  • Price: In a <div> element with data-element-name="final-price".
  • Rating: In a <p> element with data-element-name="review-score".
  • Hotel Link: In an <a> tag with class PropertyCard__Link that links to the hotel’s page.

Writing the Agoda Search Listings Scraper

To scrape data from Agoda, we will use Crawlbase Crawling API to handle dynamic content and render the page like a real browser. This ensures that we capture all hotel listings, even those loaded by JavaScript.

Here is how we can write the scraper:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
from crawlbase import CrawlingAPI
from bs4 import BeautifulSoup
import json

# Initialize Crawlbase API with your token
crawling_api = CrawlingAPI({'token': 'YOUR_CRAWLBASE_TOKEN'})

# Function to fetch the Agoda search results page
def fetch_agoda_page(url):
    response = crawling_api.get(url)
    if response['headers']['pc_status'] == '200':
        return response['body'].decode('utf-8')
    else:
        print("Error fetching the page.")
        return None

# Function to extract hotel data from the HTML content
def extract_agoda_data(html_content):
    soup = BeautifulSoup(html_content, 'html.parser')
    hotels = []

    # Loop through all hotel listings on the page
    for hotel in soup.select('div#contentContainer ol.hotel-list-container > li.PropertyCard'):
        name = hotel.select_one('h3[data-selenium="hotel-name"]').text.strip() if hotel.select_one('h3[data-selenium="hotel-name"]') else ''
        price = hotel.select_one('div[data-element-name="final-price"]').text.strip() if hotel.select_one('div[data-element-name="final-price"]') else ''
        rating = hotel.select_one('p[data-element-name="review-score"]').text.strip() if hotel.select_one('p[data-element-name="review-score"]') else ''
        link = hotel.select_one('a.PropertyCard__Link')['href'] if hotel.select_one('a.PropertyCard__Link') else ''

        hotels.append({
            'name': name,
            'price': price,
            'rating': rating,
            'link': f"https://www.agoda.com{link}"  # Complete the URL
        })

    return hotels

Handling Scroll-Based Pagination

Agoda uses scroll-based pagination, so more hotel listings will appear as you scroll down the page. We can instruct Crawlbase Crawling API to simulate scrolling and capture more listings.

To manage scrolling, we can use the scroll and scroll_interval options. Here’s how we can set them:

1
2
3
4
5
6
options = {
    'scroll': 'true',
    'scroll_interval': '20'  # Scroll for 20 seconds to load more listings
}

response = crawling_api.get(url, options)

This will make the crawler scroll for 20 seconds, ensuring that all hotel listings are loaded before scraping.

Storing Scraped Data in a JSON File

Once we have the data we need to save it in a structured format like JSON so we can analyze or process the data later. Here’s how we can save the scraped hotel data to a JSON file:

1
2
3
4
def save_to_json(data, filename='hotels_data.json'):
    with open(filename, 'w') as f:
        json.dump(data, f, indent=4)
    print(f"Data saved to {filename}")

Complete Python Code Example

Now that we have all the components in place let’s combine everything into a complete working example. This script will fetch the Agoda search results for Kuala Lumpur, extract hotel information, and store it in a JSON file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
from crawlbase import CrawlingAPI
from bs4 import BeautifulSoup
import json

# Initialize Crawlbase API with your token
crawling_api = CrawlingAPI({'token': 'YOUR_CRAWLBASE_TOKEN'})

# Function to fetch the Agoda search results page
def fetch_agoda_page(url):
    options = {
        'scroll': 'true',
        'scroll_interval': '20'
    }

    response = crawling_api.get(url, options)
    if response['headers']['pc_status'] == '200':
        return response['body'].decode('utf-8')
    else:
        print("Error fetching the page.")
        return None

# Function to extract hotel data from the HTML content
def extract_agoda_data(html_content):
    soup = BeautifulSoup(html_content, 'html.parser')
    hotels = []

    # Loop through all hotel listings on the page
    for hotel in soup.select('div#contentContainer ol.hotel-list-container > li.PropertyCard'):
        name = hotel.select_one('h3[data-selenium="hotel-name"]').text.strip() if hotel.select_one('h3[data-selenium="hotel-name"]') else ''
        price = hotel.select_one('div[data-element-name="final-price"]').text.strip() if hotel.select_one('div[data-element-name="final-price"]') else ''
        rating = hotel.select_one('p[data-element-name="review-score"]').text.strip() if hotel.select_one('p[data-element-name="review-score"]') else ''
        link = hotel.select_one('a.PropertyCard__Link')['href'] if hotel.select_one('a.PropertyCard__Link') else ''

        hotels.append({
            'name': name,
            'price': price,
            'rating': rating,
            'link': f"https://www.agoda.com{link}"  # Complete the URL
        })

    return hotels

# Function to save the data to a JSON file
def save_to_json(data, filename='hotels_data.json'):
    with open(filename, 'w') as f:
        json.dump(data, f, indent=4)
    print(f"Data saved to {filename}")

# Main script
if __name__ == "__main__":
    url = "https://www.agoda.com/search?city=14524&checkIn=2025-01-05&los=2&rooms=1&adults=2&checkOut=2025-01-09"
    html_content = fetch_agoda_page(url)

    if html_content:
        hotels_data = extract_agoda_data(html_content)  # Extract data from HTML content
        save_to_json(hotels_data)

Example Output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
[
    {
        "name": "Diamond Suite KL at Mercu Summer Suite KLCC",
        "price": "USD28",
        "rating": "7.7",
        "link": "https://www.agoda.com/axon-suite-klcc-by-home/hotel/kuala-lumpur-my.html?countryId=198&finalPriceView=1&isShowMobileAppPrice=false&cid=-1&numberOfBedrooms=&familyMode=false&adults=2&children=0&rooms=1&maxRooms=0&checkIn=2025-01-7&isCalendarCallout=false&childAges=&numberOfGuest=0&missingChildAges=false&travellerType=1&showReviewSubmissionEntry=false&currencyCode=USD&isFreeOccSearch=false&tspTypes=17&los=4&searchrequestid=c975ab20-8534-4286-8e45-b8f3b25524f3"
    },
    {
        "name": "Summer Suites KLCC By Castle Classy",
        "price": "USD34",
        "rating": "8.4",
        "link": "https://www.agoda.com/summer-suites-klcc-by-castle-classy/hotel/all/kuala-lumpur-my.html?countryId=198&finalPriceView=1&isShowMobileAppPrice=false&cid=-1&numberOfBedrooms=&familyMode=false&adults=2&children=0&rooms=1&maxRooms=0&checkIn=2025-01-7&isCalendarCallout=false&childAges=&numberOfGuest=0&missingChildAges=false&travellerType=1&showReviewSubmissionEntry=false&currencyCode=USD&isFreeOccSearch=false&tspTypes=3&los=4&searchrequestid=c975ab20-8534-4286-8e45-b8f3b25524f3"
    },
    {
        "name": "Deface Platinum 2 Kuala Lumpur",
        "price": "USD72",
        "rating": "8.5",
        "link": "https://www.agoda.com/deface-victory-suites-kuala-lumpur/hotel/kuala-lumpur-my.html?countryId=198&finalPriceView=1&isShowMobileAppPrice=false&cid=-1&numberOfBedrooms=&familyMode=false&adults=2&children=0&rooms=1&maxRooms=0&checkIn=2025-01-7&isCalendarCallout=false&childAges=&numberOfGuest=0&missingChildAges=false&travellerType=1&showReviewSubmissionEntry=false&currencyCode=USD&isFreeOccSearch=false&tspTypes=8,-1&los=4&searchrequestid=c975ab20-8534-4286-8e45-b8f3b25524f3"
    },
    {
        "name": "Riveria City Kuala Lumpur by Guestonic",
        "price": "USD20",
        "rating": "9.2",
        "link": "https://www.agoda.com/riveria-city-kuala-lumpur-by-guestonic/hotel/kuala-lumpur-my.html?countryId=198&finalPriceView=1&isShowMobileAppPrice=false&cid=-1&numberOfBedrooms=&familyMode=false&adults=2&children=0&rooms=1&maxRooms=0&checkIn=2025-01-7&isCalendarCallout=false&childAges=&numberOfGuest=0&missingChildAges=false&travellerType=1&showReviewSubmissionEntry=false&currencyCode=USD&isFreeOccSearch=false&tspTypes=5&los=4&searchrequestid=c975ab20-8534-4286-8e45-b8f3b25524f3"
    },
    .... more
]

Final Thoughts

Scraping hotel data with Python and Crawlbase enables businesses to gain insights through competitive analysis, price monitoring, and market research. Using the Crawlbase Crawling API, you can scrape data from dynamic, JavaScript-heavy websites like Agoda without running into common issues like pagination or content loading delays.

In this blog, we covered everything from finding the key HTML elements on Agoda’s search results page to writing and running a complete Python scraper. We also showed how to handle scroll-based pagination and store the scraped data in a JSON file for further analysis.

If you’re interested in learning how to scrape data from other real estate websites, check out our helpful guides below.

📜 How to Scrape Realtor.com
📜 How to Scrape Zillow
📜 How to Scrape Airbnb
📜 How to Scrape Booking.com
📜 How to Scrape Expedia

If you have any questions or feedback, our support team is always available to assist you on your web scraping journey. Remember to follow ethical guidelines and respect the website’s terms of service. Happy scraping!

Frequently Asked Questions

Web scraping is a gray area legally. While it’s generally acceptable to scrape publicly available data, always check Agoda’s terms of service to ensure compliance. Scrape responsibly and avoid using the data for unauthorized purposes.

Q. How do I handle CAPTCHA or anti-bot measures on Agoda?

Agoda uses CAPTCHAs and other bot-detecting techniques. You can get around these issues by using the Crawlbase Crawling API, which has features like browser-based rendering and IP rotation.

Q. Can I scrape data for multiple cities at once?

Yes, you can scrape data for multiple cities by modifying the query parameters in the Agoda URL. For example, update the city parameter with the desired city’s ID. Just make sure to follow scraping best practices, like limiting request frequency, to avoid being blocked.