Before you start
Nefo offers residential proxies across 227 countries over HTTP, HTTPS, and SOCKS5 with username/password auth. Generate credentials for free in the dashboard proxy generator; only routed traffic uses your prepaid bandwidth (from 5 GB for $4.50, non-expiring on flexible plans).
Scrapy ships with HttpProxyMiddleware enabled by default, so you only need to supply the proxy URL per request.
- Python 3.9+ with Scrapy installed
- A Nefo account with prepaid bandwidth
- A username/password from /dashboard/generator
- HttpProxyMiddleware enabled (default)
- AUTOTHROTTLE and a sane DOWNLOAD_DELAY configured
Scrapy proxy in request meta
Attach the proxy in your spider's requests:
The username token USERNAME-country-us requests a rotating US residential IP. Change the country code, or add state, city, or ASN tokens, to steer exits. For SOCKS5, install scrapy-rotating-proxies or a SOCKS-capable middleware and point at port 11000.
import scrapy
class ExampleSpider(scrapy.Spider):
name = 'example'
urls = ['https://api.ipify.org']
def start_requests(self):
proxy = 'http://USERNAME-country-us:[email protected]:9000'
for url in self.urls:
yield scrapy.Request(url, meta={'proxy': proxy}, callback=self.parse)
def parse(self, response):
self.logger.info(response.text)Scrapy rotating vs sticky proxy sessions
Rotating sessions issue a new residential IP per request on HTTP port 9000, which pairs naturally with Scrapy's concurrent scheduler for wide crawls.
Sticky sessions hold one IP from 1 minute up to 24 hours on HTTP port 10000. Use a token like USERNAME-country-us-session-cat12-lifetime-20 to keep the same IP while paginating a section, then rotate the session id for the next block.
Geo-targeting and polite crawling in Scrapy
Targeting is set through username tokens: country across 227 countries, US state and city, or ASN/ISP within a country. Only the username changes, so a single spider can crawl multiple regions.
Crawl lawfully, honor robots directives and site terms, and keep AUTOTHROTTLE on to stay within reasonable rates.
Frequently asked questions
No. Scrapy's built-in HttpProxyMiddleware reads meta['proxy'], so setting it per request is enough for HTTP proxies. Make sure the middleware has not been disabled in your settings.
Use port 10000 and vary the session id in the username, for example change session-cat12 to session-cat13 when the spider moves to the next section. Each id holds its own IP.
Yes, with a SOCKS-capable proxy middleware pointing at port 11000 (rotating) or 12000 (sticky) and your username token and password. For plain HTTP the built-in middleware is enough.
Set the country token in the proxy username, such as USERNAME-country-fr. US state, city and ASN options use further tokens, so one spider can crawl several regions.
Keep AUTOTHROTTLE enabled, set a DOWNLOAD_DELAY, honor robots directives and spread requests with rotating port 9000. See rotating proxies for how per-request rotation works.
No. Generating credentials is free; only the bandwidth your crawl consumes is billed against your prepaid balance.