# Rotating and sticky proxies in Scrapy with a downloader middleware

Scrapy already knows how to speak to an authenticated HTTP proxy. A twenty-line middleware turns the ProxShift username syntax into per-request rotation and per-item sticky sessions.


## The simplest version

Scrapy's built-in `HttpProxyMiddleware` reads `request.meta[\"proxy\"]` and handles credentials embedded in the URL. Set it in the spider and every request rotates on the gateway.

```python
import scrapy

PROXY = "http://USER-cc-us:PASS@res.proxshift.com:9000"

class PricesSpider(scrapy.Spider):
    name = "prices"
    start_urls = ["https://example.com/catalog"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, meta={"proxy": PROXY})

    def parse(self, response):
        for href in response.css("a.product::attr(href)").getall():
            yield response.follow(href, callback=self.parse_product, meta={"proxy": PROXY})

    def parse_product(self, response):
        yield {"url": response.url, "price": response.css(".price::text").get()}
```


## A middleware for sticky sessions per item

```python
import hashlib

class ProxShiftMiddleware:
    GATEWAY = "res.proxshift.com:9000"

    def __init__(self, user, password, country):
        self.user, self.password, self.country = user, password, country

    @classmethod
    def from_crawler(cls, crawler):
        s = crawler.settings
        return cls(s.get("PROXSHIFT_USER"), s.get("PROXSHIFT_PASS"), s.get("PROXSHIFT_COUNTRY", "us"))

    def process_request(self, request, spider):
        # Sticky when the request carries a session key (e.g. one per product flow), rotating otherwise
        key = request.meta.get("session_key")
        user = f"{self.user}-cc-{self.country}"
        if key:
            sid = hashlib.sha1(key.encode()).hexdigest()[:12]
            user += f"-sid-{sid}-ttl-10m"
        request.meta["proxy"] = f"http://{user}:{self.password}@{self.GATEWAY}"
```

```python
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ProxShiftMiddleware": 350,   # before HttpProxyMiddleware (750)
}
PROXSHIFT_USER = "USER"
PROXSHIFT_PASS = "PASS"
PROXSHIFT_COUNTRY = "de"

CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 8
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0
DOWNLOAD_TIMEOUT = 45
RETRY_ENABLED = True
RETRY_TIMES = 2
RETRY_HTTP_CODES = [403, 429, 500, 502, 503, 504, 522, 524]
COMPRESSION_ENABLED = True
```

A retried request goes through the middleware again: without a session key it leaves from a new exit, which is exactly what you want after a 403 or 429. With a session key it stays on the same exit, which is what a multi-step flow needs.


## Keeping the bill down

- Scrapy does not load images or scripts; HTML and JSON only, so a page costs its HTML size.
- Enable compression (default) and avoid `Splash` or browser rendering unless the data needs it.
- Cache with `HTTPCACHE_ENABLED = True` during development so you do not pay to re-fetch the same pages while debugging selectors.


## Per-domain politeness

A pool is a way to be many polite visitors, not one impolite one. `CONCURRENT_REQUESTS_PER_DOMAIN` and AutoThrottle keep each target at a human-plausible rate per exit; the [acceptable use policy](https://proxshift.com/legal/acceptable-use) forbids anything that looks like an attack.


## Questions

**Do I need scrapy-rotating-proxies?**

Not with a gateway: rotation happens on the ProxShift side for every new connection. That package manages lists of static proxies; with res.proxshift.com there is one proxy to configure.

**How do I use a different country per request?**

Set request.meta["country"] in the spider and read it in the middleware to build the -cc- parameter; the example uses a single setting for brevity.

Source: https://proxshift.com/guides/scrapy-rotating-proxies