- Built a high-performance clean content extraction service in Rust, matching or beating industry-standard (Trafilatura) quality of extraction while running orders of magnitude faster
- Architected a sharded, lock-free crawler that dramatically improved crawling throughput while enforcing strict politeness policies
- Worked across the crawler’s core: URL redirection, URL filtering, robots.txt enforcement, specialized scrapers and end-to-end testing
Qwant is one of Europe’s leading search engines. The crawler team owns the web-scale infrastructure that discovers and processes the web.