Data Acquisition
ML-Enhanced Web Scraping Pipeline
An automated pipeline that extracts and classifies data across more than 15,000 CrossFit websites — every one of them built differently.
The problem
Fifteen thousand websites, each one built by a different person with a different idea of where information goes. Scraping one is trivial. Scraping all of them with a rule per site does not scale, and the rules break constantly.
What I built
An automated system for extracting and classifying data across 15,000+ CrossFit websites — using machine learning to identify and categorise the relevant content rather than hand-writing an extraction rule per site. That is what makes the pipeline hold up across a corpus this heterogeneous: the classifier generalises where selectors do not.