Link dedup report merges distinct URLs: the query cleaner collapses repeated params and drops blank ones via…
Link dedup report merges distinct URLs: the query cleaner collapses repeated params and drops blank ones via…: a task in agent-harbor-tasks (Harbor dataset). The weekly duplicate-link report ( python -m linkdedup.report samples/links.txt ) claims there are far fewer unique pages than there really…
The task
The weekly duplicate-link report (`python -m linkdedup.report samples/links.txt`) claims there are far fewer unique pages than there really are: for example the shop search with two `tag` filters is reported as the same page as the search with a single tag, and the item page with the `debug=` flag is merged with the…
Part of efederici/agent-harbor-tasks.