Description

Scrapy is a high-level web crawling and scraping framework for Python. Prior to 2.17.0, in scrapy/core/downloader/handlers/s3.py, Scrapy's S3DownloadHandler converts an S3-scheme bucket and key request into a plaintext HTTP request to the corresponding S3 endpoint unless request.meta["is_secure"] is explicitly enabled, then signs and sends the plaintext request with configured AWS credentials. A network attacker who can observe traffic between Scrapy and S3 can read the bucket and key path, AWS Authorization header, X-Amz-Security-Token when temporary credentials are used, S3 object contents, and S3 response headers. An active man-in-the-middle attacker can also modify the plaintext S3 response body, status code, and headers before Scrapy processes them, causing scraped-data poisoning, poisoned exports, HTTP cache poisoning when caching is enabled, or influence over later crawl targets through forged redirects or attacker-controlled links. Users making S3-scheme requests with AWS credentials are affected. This issue is fixed in version 2.17.0.

Executive Summary

Technical Details

Mitigations

References

Additional Metadata

{
  "score": "7.4",
  "pubDate": "2026-09-01T21:18:47.593Z",
  "pubdate": "2026-09-01T21:18:47.593Z",
  "executiveSummary": "Scrapy versions prior to 2.17.0 contain a critical vulnerability in the S3DownloadHandler that results in the transmission of S3 requests over plaintext HTTP instead of TLS-encrypted HTTPS.\nThe vulnerability occurs because the framework defaults to plaintext requests unless the 'is_secure' metadata flag is explicitly enabled, leading to the transmission of sensitive AWS credentials, authorization headers, and security tokens in cleartext.\nA network-based attacker capable of observing traffic can perform credential theft, intercept S3 bucket contents, and execute man-in-the-middle (MitM) attacks.\nActive attackers can manipulate response payloads, status codes, and headers, resulting in scraped-data poisoning, HTTP cache poisoning, or malicious redirection of crawl targets.\nThe flaw affects users interacting with the S3-scheme who rely on default configurations, significantly compromising the confidentiality and integrity of AWS-integrated scraping workflows.\nRemediation requires upgrading Scrapy to version 2.17.0 or later to ensure secure protocol enforcement.",
  "technicalDetails": "The vulnerability resides within the scrapy/core/downloader/handlers/s3.py module of the Scrapy framework. The S3DownloadHandler component fails to enforce secure transport layer security by default when processing S3-scheme requests.\nRoot Cause: The handler logic converts requests to an S3 endpoint into plaintext HTTP requests unless the request.meta['is_secure'] parameter is set to True. Consequently, the framework inadvertently transmits sensitive data—including the AWS Authorization header, X-Amz-Security-Token, and the bucket/key path—over an unencrypted channel.\nExploitation Flow: A network attacker positioned between the Scrapy client and the AWS S3 endpoint can intercept the traffic. Because the transport layer lacks TLS/SSL encryption, the attacker can sniff the packets to extract temporary or long-term AWS credentials and sensitive S3 object data.\nMitM Capabilities: Beyond passive credential harvesting, the lack of transport encryption allows an active MitM attacker to intercept and modify the HTTP response body, status codes, and headers before they reach the Scrapy crawler.\nImpact Analysis: 1) Data Exfiltration: Attackers can capture AWS security tokens and credentials, potentially gaining unauthorized access to the victim's S3 resources. 2) Scraped-data Poisoning: By modifying the response body, an attacker can inject malicious content into the scraped dataset. 3) HTTP Cache Poisoning: If the Scrapy project utilizes HTTP caching, the malicious response can be cached and served repeatedly. 4) Crawl Redirection: An attacker can modify HTTP headers (e.g., Location) or response content to force the crawler to traverse malicious links, potentially leading to a persistent compromise or redirection to attacker-controlled infrastructure.\nAffected Versions: All Scrapy versions prior to 2.17.0 are vulnerable. The exploit requires no specific privilege level, only the ability to intercept network traffic between the crawler and the S3 service. Post-exploitation, the impact ranges from unauthorized data access to the compromise of the scraping pipeline's integrity and subsequent crawl targets."
}