Typical web data workflows
- Gathering publicly available data for research and analytics.
- Accessing region-specific public content for comparison.
- Distributing lawful collection across many independent requests.
- Refreshing the same public pages on a schedule to track change.
The common thread is volume and location. A single IP gets rate-limited, and a single location only shows one version of a page.
How residential proxies help
Rotating residential proxies send each request through a different real home IP, so no single address carries the whole job. Targeting lets you pick the exit country, state, city or ISP, from 227 countries and territories, 22,417 cities and 14,252 ISPs/ASNs.
Rotation does not replace good manners. Keep request rates reasonable, retry politely and cache what you already fetched.
A collection workflow
- Define the public pages, fields and regions you need, and check each site's terms and robots directives.
- Create credentials and connect your scraper to the gateway
gw.nefo.net:9000over HTTP, HTTPS or SOCKS5. - Run a small sample, record success rates and bytes per page, and fix the parser before scaling.
- Scale up with rotating sessions, adding region tokens where content differs by location.
- Store raw responses with timestamps so you can re-parse without fetching again.
Code samples for Python Requests, Scrapy, Golang and curl are in the integrations section.
Proxy type and session mode
| Job | Proxy type | Session mode |
|---|---|---|
| Independent requests at volume | Rotating residential | Rotating |
| Multi-page sequence or paginated listing | Rotating residential | Sticky up to 24 hours |
| Tolerant targets where cost matters most | Rotating datacenter | Rotating |
| Mobile-first sites | Rotating mobile | Rotating or sticky up to 2 hours |
Targeting tips
- Collect from the region your audience is in so currency, language and offers match.
- Use sticky sessions for pagination so pages stay in order, and rotate between listings.
- Spread concurrency as you need. Concurrency is unlimited, so your own rate limits are the control.
Collect responsibly
Responsible collection means respecting robots directives where applicable, honoring site terms, avoiding personal data you are not permitted to process and keeping request rates reasonable. See the acceptable use policy and the guide to safe and acceptable proxy use cases.
Rotating residential bandwidth starts at $4.50 for 5 GB and never expires; see pricing.
Handling errors and blocks politely
Not every request succeeds, and the way you react matters. A calm, predictable client is better for you and for the sites you read.
- Treat a failed or empty response as a signal to slow down, not to retry in a tight loop.
- Retry with a new rotating IP a small number of times, and then log the URL for later.
- Cache what you already fetched, so you do not request the same public page twice.
- Check a page in a real browser when a parser suddenly returns nothing. The layout may have changed, not the proxy.
- Respect robots directives and site terms, and stop if a site asks you to.
Measure two numbers on every sample run: the share of requests that return the expected page, and the average bytes per page. They tell you how much bandwidth a full run will need.
Common mistakes to avoid
- Scaling before sampling. A short test shows page size, success rate and parser errors cheaply.
- Using one session mode for everything. Rotate for independent requests and use sticky sessions only for sequences.
- Paying residential rates where they are not needed. Try a datacenter proxy on tolerant sites first.
- Collecting more than you need. Personal data you are not permitted to process should never enter your pipeline.
Plan the data you keep, too. Store raw responses with the market and time so that you can re-parse later, and delete what you have no lawful reason to hold.
Frequently asked questions
Collecting publicly available data is permitted when it complies with the target site's terms and applicable law. Prohibited and abusive activity is not allowed. See the acceptable use policy.
Rotating sessions suit high-volume, stateless collection where each request is independent. Use sticky sessions, from 1 minute up to 24 hours, when a page sequence must come from one IP.
Use datacenter proxies, from $0.66 per GB, on sites that accept server traffic, and residential on sites that block servers or vary content by location. The comparison lists the trade-offs.
Any tool that accepts a proxy. There are setup pages for Scrapy, Python Requests, Puppeteer, Playwright and Node.js Axios.
It depends on page size and volume. Begin with 5 GB for $4.50, measure a sample run, then buy more. The price per GB falls with package size, and bandwidth never expires.