What is rs-trafilatura and how does it work with Scrapy?

rs-trafilatura is a Rust-based content extractor that plugs into Scrapy's item pipeline. Add it to settings.py, yield items with 'body' or 'html', get structured extraction auto-added.

Is rs-trafilatura fast enough for large Scrapy crawls?

Yes—44ms/page, negligible vs. network. For ultra-high throughput, offload to processes.

What data does rs-trafilatura extract from HTML?

Title, author, date, main content, markdown, page_type, quality score, language, site name, description.

📦 Open Source

Scrapy's New Best Friend: rs-trafilatura Pipeline Tears Through HTML Junk

Scrapy spiders spew raw HTML like a firehose of garbage. rs-trafilatura cleans it up, Rust-fast, right in your pipeline—no more manual parsing hell.

DevTools Feed Apr 03, 2026 3 min read

Scrapy pipeline diagram with rs-trafilatura extracting clean text from HTML

⚡ Key Takeaways

rs-trafilatura integrates smoothly as a Scrapy pipeline for instant content extraction. 𝕏
Rust speed (44ms/page) adds zero real overhead to crawls. 𝕏
Page-type routing and quality filters make pipelines production-ready. 𝕏

Published by

DevTools Feed

Ship faster. Build smarter.

#content extraction #rs-trafilatura #scrapy #web scraping

Worth sharing?

Get the best Developer Tools stories of the week in your inbox — no noise, no spam.

Originally reported by dev.to

⚡ Key Takeaways

The 60-Second TL;DR

DevTools Feed

Share this article

Worth sharing?

Related Stories

rs-trafilatura Fixes Web Scraping's Dirty Secret: Non-Article Pages Finally Extract Right

rs-trafilatura + Firecrawl: The Web Scraping Duo That Thinks Like a Journalist

rs-trafilatura Meets spider-rs: Finally, Crawling That Doesn't Suck

Rust-Powered rs-trafilatura Supercharges Crawl4AI: 0.910 F1 on Benchmarks

Stay in the loop