What is price intelligence?
Price intelligence is the discipline of knowing. in close to real time. what your competitors charge, who they charge it to, and how those numbers move. It's run by retail pricing teams, marketplace sellers, fintech analysts working consumer credit, OEMs negotiating with distributors, and increasingly by anyone whose margin depends on staying inside a couple of percent of the market median.
The output looks deceptively simple: a table of SKUs, prices, and timestamps. The work behind it is anything but. To produce a defensible feed you need to scrape thousands to millions of product pages every day, often segmented by region, currency, customer cohort, and time-of-day. You need the data to be representative. not just whatever price the retailer wanted to show you from your office IP. And you need the pipeline to run for months without bans, drift or silent corruption.
Most teams discover the proxy layer is where their pipeline lives or dies. Build it on weak infrastructure and your dashboards become opinionated fiction.
Why proxies matter here
Price pages are the single most aggressively defended surface in modern retail. The retailer's logic is straightforward. if their competitors have their live prices, they lose pricing power, so they invest accordingly. That investment shows up as fingerprinting (TLS, header order, JS execution), per-IP and per-session rate limits, dynamic price serving (different price for different visitor segments), and outright cloaking when bot probability crosses a threshold.
Without a proxy layer, you're stuck on a handful of egress IPs. Every retailer you scrape sees the same fingerprint, and your data gets corrupted in three ways at once: the prices you do get are the bot-discouragement prices (often inflated or stale), the prices you don't get show up as nulls in your dataset, and the geo-priced markets are completely invisible because you can only present as one country at a time.
A residential proxy network solves all three. Each request looks like a different consumer in a different city. The retailer can't easily distinguish your scraper from real shoppers, the rate-limit logic resets per IP, and you can see. separately. what a Madrid customer pays versus a Lisbon one for the same SKU.
What VaultProxies brings to this
Price intelligence is the textbook case for our Residential Unlimited plan. Pricing teams are bandwidth-hungry by nature. every product page is a few hundred kilobytes, and a serious operation pulls millions of pages a day. Per-GB pricing is fine until you scale, then it stops being fine. Unlimited removes the cost ceiling and replaces it with a thread ceiling, which for most teams is the more comfortable constraint.
On the unlimited plan you get up to 50K concurrent threads through the same residential pool. same 32M+ IPs, same country/state/city/ASN targeting, same sticky-session controls. The gateway sits behind an anycast edge, so the routing overhead per request is negligible against the time the retailer's app server spends rendering the page anyway.
- 50K concurrent threadsRun a continuous crawl across millions of SKUs without throttling yourself.
- Predictable costHourly billing, no bandwidth surprises. Plan a quarterly crawl budget without spreadsheet acrobatics.
- Geo-segmented pricingPin one worker pool to UK exits, another to Italy, another to the US. Compare like-for-like SKUs across markets.
- 99.9% uptime SLAPricing dashboards that go dark on Monday morning kill credibility. The gateway is built to not.
Recommended setup
Use the Residential Unlimited plan with rotating sessions for the bulk of the crawl. Most price endpoints are stateless. you don't need to hold a session across requests. Rotate per-request, randomize User-Agent across a small whitelist of recent Chrome and Safari builds, and parse the price out of structured data (JSON-LD, microdata, or a known JSON endpoint) rather than scraping rendered HTML where you can.
Segment workers by target geography. Don't run a single thread pool against all retailers. each market should have its own pool with the matching country flag, because that's also how you discover dynamic-pricing experiments.
import requests, random
from concurrent.futures import ThreadPoolExecutor
# Rotating residential proxy, country-pinned to GB
PROXY = "http://USER-country-gb:[email protected]:8080"
proxies = {"http": PROXY, "https": PROXY}
UAS = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 13_5) AppleWebKit/605.1.15 ...",
]
def fetch(url):
r = requests.get(
url, proxies=proxies, timeout=20,
headers={"User-Agent": random.choice(UAS), "Accept-Language": "en-GB,en;q=0.9"},
)
r.raise_for_status()
return r.text
with ThreadPoolExecutor(max_workers=400) as pool:
pages = list(pool.map(fetch, urls))Common pitfalls
- Trusting one geography for all pricesIf you only scrape from US exits, you'll miss every market where the retailer prices regionally. That's most of them.
- Treating null as dataWhen a request gets cloaked, the retailer often returns a 200 with no price. Don't pipe nulls into the dashboard. flag them.
- Crawling too fast for the catalogueUnlimited threads doesn't mean unlimited speed. If you re-scrape the same SKU every minute, you're over-paying for noise. Hourly is usually enough.
- Skipping User-Agent rotationEven on residential, a static UA across 50K threads is a tell. Rotate within a small, realistic whitelist.
- Forgetting to verify price extraction monthlyRetailers redesign. Your selector breaks. Run a regression suite weekly or you'll discover the breakage in a board deck.
Getting started
Create an account, pick the Unlimited plan at the thread tier that matches your crawl size, and point your existing pipeline at the gateway. Most pricing teams move from a per-GB plan to unlimited within two months. once volume is predictable, the math always favors unlimited. If your operation also covers SERP monitoring or travel and streaming intel, the same plan handles those workloads without changes.