How Companies Used Data in COVID-19lessons from a crisis

When COVID-19 reshaped daily life in 2020, it also reshaped how businesses understood their own markets almost overnight. Demand patterns that had held steady for years shifted in weeks, supply routes were interrupted, and customer behavior changed faster than most planning cycles could track. For a lot of teams, the instinct was the same: look at the data, and look at it more often.

This article is a calm, factual retrospective on how companies leaned on web and public data during that period to make decisions under unusual uncertainty. It is about business and data practice, not health advice. By the end you should have a clear picture of where external data helped, what kinds of signals teams tracked, and which habits formed then that are still worth keeping.

Why data mattered more under sudden uncertainty

Data is simply raw information that has been collected and organized into a form a team can analyze. On its own a row of numbers means little. Turned into a trend, a comparison, or a forecast, it becomes the basis for a decision: what to stock, where to focus support, which assumption to revisit. In normal times, businesses can lean partly on experience and on patterns that hold from one quarter to the next.

The early pandemic period removed a lot of that stability. Historical baselines stopped predicting the present, so the gap had to be filled with fresher, more external signals. Companies that already collected and analyzed data well were better positioned to notice change early and respond, while teams relying on slower, backward-looking reports were often a step behind. The value of data did not change, but the cost of ignoring it rose sharply.

Tracking demand when patterns broke

The most immediate use of data was demand sensing. With purchasing habits shifting week to week, internal sales history alone was no longer a reliable guide to what customers wanted next. Teams supplemented it with external, public signals: search interest trends, marketplace availability and pricing, category-level movement across retail sites, and the visible shift toward online channels as physical options narrowed.

None of this required guessing at private information. A great deal of useful signal lives on public web pages: which products are listed as in or out of stock, how prices move across competitors, which categories suddenly fill with new listings, and how product availability changes by region. Aggregated and watched over time, those public data points gave teams an earlier read on demand than waiting for the next internal sales cycle to close.

The practical lesson was about cadence as much as content. Monthly reporting was too slow for a fast-moving period, so many teams moved to refreshing key external signals far more frequently, and to watching a small set of indicators closely rather than a large dashboard occasionally.

Adapting supply and operations with public signals

Supply chains felt the disruption directly, and data helped teams react to it. When a usual supplier or route was constrained, the question became where else the same goods or materials were available, and at what price. Public web data offered a way to answer that at scale: scanning supplier directories, marketplace listings, and catalog pages to map alternatives, compare availability, and spot price movements that signaled scarcity before it reached the warehouse.

Operations teams used similar inputs to plan. Public information about regional conditions, business hours, service availability, and logistics changes fed into decisions about where to route work and how to set customer expectations. The common thread was substituting fresh external observation for assumptions that no longer held. Where a static supplier list had been good enough before, teams now wanted to see the current state of the market and refresh that view regularly.

Reading changing customer behavior

Consumer behavior is the study of how people decide what to buy, use, and prioritize. During the pandemic those decisions shifted noticeably as routines changed, and the shift showed up in observable, aggregate ways: more activity online, different times of day for certain services, new product categories rising in interest, and changes in how people researched before purchasing. Streaming, delivery, home goods, and remote-work tools are commonly cited examples of categories that saw clear movement.

For businesses, the goal was not to profile individuals but to understand these aggregate trends well enough to respond. Public data such as category-level search interest, review volumes, and content engagement gave teams a way to map how customer priorities were moving and to adjust messaging, support, and product focus accordingly. The companies that adapted fastest tended to be the ones already in the habit of turning behavioral signals into concrete strategy rather than treating them as background noise.

This is where data analysis earned its keep. Raw observations became useful only once a team processed them into patterns and acted on them: shifting a marketing emphasis, reprioritizing a roadmap, or staffing support where new demand appeared. Web and public data widened the lens, but the discipline of analysis is what turned that wider view into decisions.

Monitoring performance with the right indicators

As work moved largely online and many teams operated remotely, leaders needed a clearer view of how the business was actually performing without the informal signals an office provides. Data filled that gap. By identifying a focused set of key performance indicators, the measurable values that show whether core objectives are being met, teams could monitor performance objectively and adjust as conditions changed rather than waiting for a quarter to end.

The same data that tracked external demand also helped internally: measuring how channels performed as customers moved online, where support load was concentrating, and which parts of the operation were keeping pace. Used this way, data became a tool for steady, incremental learning under pressure, letting organizations adapt deliberately instead of reacting blindly. That habit, choosing a few meaningful indicators and watching them honestly, outlasted the crisis that prompted it.

How companies collected web data at scale

Gathering this kind of data by hand does not scale. Finding one nearby supplier is easy, but tracking thousands of listings, prices, and availability changes across many sites, and refreshing that view often, is a different problem. This is where automated collection comes in. Extracting data from the web is usually described as crawling, retrieving pages broadly, and scraping, pulling structured information out of those pages so it can be analyzed.

The practical challenge is that many sites actively limit automated access. Pages render content with JavaScript, rotate layouts, rate-limit requests, and present CAPTCHAs, all of which can stop a naive collection job. Building and maintaining the infrastructure to handle rendering, IP rotation, and blocks is real work, and it is the part teams most often underestimate when they decide to collect public web data themselves. If you want a fuller picture of the obstacles, our guide on how to scrape websites without getting blocked walks through them in detail.

Once collection is handled, the data still has to be shaped before it is useful. Pages are built for people to read, so the extract arrives messy, with varying fields and formats across sites. Giving it a consistent structure is what turns a pile of pages into something a team can query and trend. Our walkthrough of building a scalable web data pipeline covers how that flow holds up as volume grows.

Scraping responsibly

Collecting public web data carries real responsibilities, and they matter more, not less, during a sensitive period. Respect each site's terms of service and its robots.txt directives, focus on public and aggregate data rather than anything tied to identifiable individuals, and keep request rates reasonable so you do not strain the sites you depend on. Where any personal data is involved, follow applicable privacy rules such as GDPR and CCPA, and prefer official APIs when a site provides one. Responsible collection is both an ethical baseline and a practical one, since it keeps your access durable over time.

What stayed after the disruption

Many of the data habits that formed under pressure proved worth keeping. Sensing demand from external signals, refreshing key indicators frequently instead of quarterly, watching the market's current state rather than last year's assumptions, and treating public web data as a standard input rather than a special project: all of these started or accelerated in 2020 and remained useful afterward. The crisis did not invent data-driven decision making, but it made the cost of slow, inward-looking analysis impossible to ignore.

The lasting takeaway is modest and practical. Businesses that could see their environment clearly, through fresh, public, well-organized data, adapted more calmly than those that could not. That advantage was never about predicting the future perfectly. It was about noticing change a little earlier and responding a little faster, which is exactly what good data practice has always offered.

Frequently Asked Questions (FAQs)

How did businesses use web data during COVID-19?

Many teams used public web data to sense demand and adapt operations when normal patterns broke. They tracked signals like search interest, product listings, pricing, and availability across sites to get an earlier read on shifting demand, to find alternative suppliers when usual routes were constrained, and to understand aggregate customer behavior. The common goal was substituting fresh external observation for assumptions that no longer held.

What kinds of data were most useful in that period?

Public, aggregate, and commercial signals were the most useful: product availability and stock status, competitor pricing, category-level trends, search interest, and review or engagement volumes. These point to how a market is moving without relying on private information. Combined with internal data and refreshed frequently, they gave teams a current view of conditions rather than a backward-looking one.

Is collecting public web data the same as tracking individuals?

No. The practices described here focus on public, aggregate market signals such as listings, prices, and category trends, not on identifying or profiling people. Responsible collection keeps that distinction firm, treats any personal data it encounters with care, and follows privacy rules such as GDPR and CCPA when personal information is involved.

What is the difference between crawling and scraping?

Crawling is the broad retrieval of web pages, following links to gather content across a site or many sites. Scraping is the extraction of specific, structured information out of those pages so it can be analyzed. In practice they work together: a crawler fetches the pages, and a scraper pulls the fields you actually need into a usable form.

Why is collecting web data at scale difficult?

Gathering data from a few pages is simple, but doing it across thousands of listings and refreshing often runs into obstacles. Many sites render content with JavaScript, rate-limit requests, rotate layouts, and present CAPTCHAs to limit automated access. Handling rendering, IP rotation, and blocks reliably takes real infrastructure, which is why teams often use a managed service rather than building and maintaining it themselves.

Which data habits from the pandemic are worth keeping?

The durable ones are practical: watch a focused set of indicators frequently rather than relying on slow quarterly reports, treat public web data as a standard input instead of a special project, and base decisions on the market's current state rather than last year's assumptions. These habits help a business notice change earlier and respond more calmly, which is valuable in any conditions, not just a crisis.