sitemap-product-matcher

The following is AI-generated (aka vibe-coded) software. The software is specific to my needs and my system and may not work for you or your system. Support is limited, but if you report a bug I'll certainly look into it. All code is provided in a zip file for you to review should you choose.

Crawl the sitemaps of one or more sites, store every discovered page URL in SQLite along with a product name guessed from the URL, then fuzzy-match a list of known product names against those URL-derived names.

Useful for answering “which of these 800 products does this retailer actually list, and at what URL?” without scraping an entire site.

How it works

1. Discover — for each root URL, read robots.txt and follow any Sitemap: lines. If there are none, probe the usual suspects (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, and a few others).
2. Crawl — walk the sitemap graph breadth-first with a thread pool, handling sitemap indexes, nested sitemaps, and plain-text sitemaps. Rows are streamed into SQLite in batches.
3. Derive names — for each page URL, pick the path segment most likely to hold the product name (skipping generic segments like product, shop, collections, locale codes) and normalize it: strip accents, lowercase, split on non-alphanumerics, drop filler tokens and long numeric IDs.
4. Match — for every product name, shortlist candidates using an inverted token index, then score the shortlist with a blend of character-level ratio (difflib.SequenceMatcher), Dice coefficient, and token containment. The best hit above --min-score wins; a close runner-up flags the match as ambiguous.
5. Verify (optional) — fetch each matched page and read the product name back out of JSON-LD, OpenGraph tags, or <title>, then score that against the product name.

Matching is per-site: each root URL gets its own candidate pool and its own block of output.

Requirements

• Python 3.10 or newer
• curl_cffi — the only third-party dependency, used to fetch with a Chrome TLS fingerprint so that WAF-protected sites don’t block the crawl

pip install curl_cffi

Input files

Two plain-text files, one entry per line. Blank lines and lines starting with # are ignored, and a UTF-8 BOM is tolerated.

urls.txt — one root URL per line. A missing scheme is filled in as https://, and trailing slashes are stripped.

https://example-shop.com
another-shop.co.uk

products.txt — one product name per line.

Acme Widget Pro 3000
Blue Titanium Kettle 1.7L

Usage

python sitemap-product-matcher.py --urls-file-path urls.txt --products-file-path products.txt

Re-match against URLs already in the database, fetching nothing:

python sitemap-product-matcher.py --urls-file-path urls.txt --products-file-path products.txt --use-existing

Keep what’s stored and only crawl sites that have no URLs yet:

python sitemap-product-matcher.py --urls-file-path urls.txt --products-file-path products.txt --resume

Restrict the crawl to product pages and confirm each hit against the live page:

python sitemap-product-matcher.py --urls-file-path urls.txt --products-file-path products.txt --include-regex "/product/" --verify

Show the top three candidates per product, plus the products that missed:

python sitemap-product-matcher.py --urls-file-path urls.txt --products-file-path products.txt --top 3 --show-misses --verbose

Write everything to a specific folder:

python sitemap-product-matcher.py --urls-file-path urls.txt --products-file-path products.txt --output-path D:\scans026-09

Output

Everything the script writes goes to --output-path (default: the current folder).

Matches are also printed to the console:

  [0.94 @tier 0.93] Acme Widget Pro 3000
        URL : https://example-shop.com/product/acme-widget-pro-3000
        NAME: acme widget pro 3000
        PAGE: Acme Widget Pro 3000  [sku AW3000, Acme, 249.00 USD, InStock]
              verified 1.00 via json-ld

The score is the fuzzy score; the tier is the strictest confidence band it cleared (0.99, 0.93, 0.87, 0.80, 0.73, 0.66, 0.58).
AMBIGUOUS appears when the runner-up is within --ambiguity-margin. With --verify, a <-- page disagrees with the URL note appears when the page’s own name scores below --verify-warn.

A run ends with a summary line: sites, URLs stored, products, matches, ambiguous matches, HTTP requests, cache hits, and elapsed time.

Database schema

• sitemap_urls — root_url, sitemap_url, url, url_slug, product_name (the normalized slug). Unique on (root_url, url).
• sitemap_docs — one row per sitemap document fetched: source (network, cache-fresh, cache-304), URL count, kept count, fetch time.
• products — raw_name and normalized_name for each input product.
• matches — the winning URL and score per product per site, plus tier, ambiguity flag, runner-up, and any verification fields (verified_name, verified_sku, verified_brand, verified_price, verified_availability, verified_score).

So a run can be queried afterwards rather than re-read from the console:

SELECT product_raw, url, score FROM matches
WHERE url IS NOT NULL AND ambiguous = 0
ORDER BY score DESC;

Options

Files and mode

URL filtering

Applied both while crawling and while matching, so the same filters can be re-applied to an existing database with --use-existing.

Exclusions are evaluated first; regexes are case-insensitive.

Network

HTTP cache

Bodies are gzipped into SQLite alongside their ETag and Last-Modified, so re-crawls send conditional requests and mostly get 304s back.

Crawl limits

Matching

Raising --min-score above 0.80 cuts false positives sharply on large catalogues; lowering it below 0.58 mostly produces noise that --show-misses would have surfaced anyway.

Page verification

Verification fetches bypass the HTTP cache, and one request is made per matched URL, so --verify on a few hundred products is a few hundred extra requests.

Other

--verbose prints per-sitemap crawl progress and any fetch or parse problems to stderr.

Performance notes

Shortlisting is what keeps large sites tractable. Instead of scoring every URL against every product, the script intersects posting lists for the product’s rarest tokens, cheapest list first, backing off whenever a token would empty the set. Intersecting rather than unioning matters on catalogues built from a small vocabulary, where every individual token is common but the combination is not. When a product shares no exact token with anything, a 4-character prefix index catches typos, plurals, and suffixes; if that also fails and the site is under --full-scan-limit URLs, everything gets scored.

Scoring itself short-circuits aggressively: a length-based ceiling, then real_quick_ratio and quick_ratio, before any full SequenceMatcher.ratio() call.

Exit codes

Limitations

• Only sitemap-listed pages are considered. Products reachable only through on-site search or JavaScript navigation will never appear.
• Gzipped sitemap files (sitemap.xml.gz) are only handled when the server sets Content-Encoding: gzip; a body served as raw application/gzip will fail to parse and be reported as “no entries”.
• URL slugs are a proxy for product names. On sites that use opaque IDs (/p/8827461) matching will mostly miss; --verify on a filtered crawl is the workaround, at the cost of a request per page.
• The database is deleted and rebuilt on every crawl run. Use --resume or --use-existing to keep what’s there.
• No robots.txt Disallow enforcement — only Sitemap: lines are read. Be considerate with --delay and --workers.