Archive.org is an invaluable tool for SEO tasks, funded by donations. If you search for a domain and select the “URLs” option, you can access up to 10,000 listed URLs.
However, there are a few limitations:
- URL limit: You can only retrieve up to 10,000 URLs, which is insufficient for larger sites.
- Quality: Many URLs may be malformed or reference resource files (e.g., images or scripts).
- No export option: There isn’t a built-in way to export the list.
To bypass the lack of an export button, use a browser scraping plugin like Dataminer.io. However, these limitations mean Archive.org may not provide a complete solution for larger sites. Also, Archive.org doesn’t indicate whether Google indexed a URL—but if Archive.org found it, there’s a good chance Google did, too.