Jobs Classification & Search
Collect and classify job descriptions at scale, then make them searchable.
- Role
- Backend / data engineer
- Year
- 2020
- Timeline
- Research + delivery
This is an NDA-signed client project, so it isn’t linked publicly. I’ll walk you through it live on a call and answer anything about how it was built.
Job descriptions were scattered across many sites in inconsistent formats, with no unified way to categorise or search them.
Collection had to be fast and parallel across many sources, and the messy text had to be cleaned before it could be classified or searched usefully.
Built parallel scrapers feeding a clean NLP + Elasticsearch pipeline.
Instead of Sequential scraping into a plain database.
Why: Multithreaded/multiprocessing collection plus a proper search index turned a slow scrape into a searchable, categorised dataset.
Normalised every source into one clean schema before indexing.
Instead of Indexing each site's raw, inconsistent format as-is.
Why: Consistent fields for location, skills and experience were the whole reason cross-site search and categorisation worked at all.
Kept collection and processing decoupled so a slow source couldn't stall the rest.
Instead of One sequential scrape-then-process job.
Why: Decoupling let scrapers run in parallel and continuously while NLP and indexing consumed at their own pace, so one blocked site never held up the pipeline.
- Multithreaded / multiprocessing scrapers
- NLP categorisation by location, skills & experience
- Cleaning & preprocessing pipeline
- Elasticsearch search over the corpus
Hemant Manglani
Ahmedabad, India
Have a product that has to work? Let’s talk.
A 20-minute call, no pitch. We work out whether this is a real problem worth solving, and if I’m not the right person, I’ll tell you.
Every message answered within one business day