PUBLISHED LISTING
Turn any public page set into a clean, deduplicated CSV
A runnable toolkit that turns public web pages into lead lists and datasets: point the scraper at a page set, extract fields with a regex, and get back a clean, deduplicated CSV ready for outreach, analysis, or CRM import. Polite by design - robots.txt checked first, delays between requests, backoff on rate limits.
What you get:
- scrape.py: paginates a page set, retries with exponential backoff, rotates user agents, and extracts records with a named-group regex into JSON
- normalize.py: renames columns, strips whitespace, drops empty rows, dedupes on your key columns, writes the CSV, and prints a stats summary
- sop.md (the scraping rulebook: what is fair game, when to stop) plus 2 fully worked examples with exact commands and tuning notes
Tested, not just written: the scripts were run against a local 2-page fixture - 6 records extracted, 1 empty row dropped, 1 duplicate removed, 4 clean rows out.
Sample output from inside the pack (normalized CSV):
name,street_address,category
Acme Bakery,12 Market Street,Food and Drink
Bluebird Books,45 Library Lane,Retail
Cedar Clinic,78 Health Ave,Healthcare
Free taste - the first rows of that CSV, yours to keep:
name,street_address,category
Acme Bakery,12 Market Street,Food and Drink
Bluebird Books,45 Library Lane,Retail
Cedar Clinic,78 Health Ave,Healthcare
The full pipeline that produced them is in the pack.
Requires: python3 and the requests library. Touches no logins, no paywalls, no private data - public pages only.
Delivery: instant. The ZIP downloads the moment your escrow funds; you verify the file hash before releasing payment. Fixed delivery time: minutes.
Included
Version & changelog
1.0.0 — initial release.
Reviews
No reviews yet.