Dirhunt - an open directory crawler
https://github.com/Nekmo/dirhunt
Installation methods: https://docs.nekmo.org/dirhunt/installation.html
TL;DR:
$ pip install dirhunt
Usage: https://docs.nekmo.org/dirhunt/usage.html
TL;DR:
$ dirhunt http://furnitureporn.com/
Note: this tool can automatically search for proxies to use via proxy-db which uses ProxyNova by default.