The Wayback Machine takes historical screenshots of web pages and stores them in its public database. Anyone can use the Wayback Machine to view previous versions of pages or entire sites.

Here are five smart ways you can use the Wayback Machine for SEO.

1. Find legacy URLs from old versions of the site

One of the most useful ways to use the Wayback Machine is to find historical URLs that have never been redirected.

The Wayback Machine collects information about your site throughout time. So it might have access to URL data from 10+ years ago.

This is especially important for sites that have been around for a long time. It’s likely that the stakeholder who managed the site years ago has changed the company or left roles and may not have used SEO best practices during site migrations.

The Wayback Machine can be a lifesaver here. You can quickly find old URLs that were never redirected to live versions.

For example, the “Headphones” page from Bose (https://www.bose.com/products/headphones/) from 2003 was never redirected

Using the Wayback Machine, it’s easy to discover legacy versions of key content from previous site versions. You can then find URLs to redirect that you would have likely never discovered otherwise.

Want to take this to the next level? Read Patrick Stox’s article on using the Wayback Machine API to find historical redirects. By querying the API, you can bulk export legacy URLs. This can be much more efficient for larger sites.

2. Find previous page content

Website content changes over time. This happens for a variety of reasons (e.g., SEO, CRO, site migration or highlighting different aspects of a product). There is always an inherent risk of any content changes, especially if they’re significant.

This is where the Wayback Machine comes into play.

If you’re seeing large losses in rankings after changing content, you can check the Wayback Machine to surface previous versions of old pages. Restoring the content to its original version could help your content regain lost visibility.

For example, NYMag’s “Best Pillows For Neck Pain” article has lost organic visibility since mid-2020 for terms such as “neck support pillow.” This has resulted in organic traffic loss over time.

Comparing the page to early 2020, we see that they have changed the content since then. The 2020 version included a quote from a chiropractor from the American Chiropractic Association in the introduction and kept the products above the fold.

However, in the current version, they have added more content to the introduction, pushed the products below the fold and moved the American Chiropractic Association quote further down the page.

While this might not be the sole cause of the ranking drops, looking back at the previous content during peak rankings could help them test restoring some of the content to older versions to see if this helps improve visibility.

3. Finding old robots.txt file

Another great use of the Wayback Machine is checking how your robots.txt has changed from previous versions. This can be particularly helpful during a site migration if your robots.txt file has changed and you don’t have a version of the original file.

Fortunately, the Wayback Machine crawls robots.txt files a lot.

Using this, you can analyze how the robots.txt has changed over time. For instance, IBM’s robots.txt file looks quite different from what it used to.

Using The Wayback Machine can be an extremely effective way to lookup old versions of your robots.txt file. This is especially useful if the information has been lost during a site migration.

4. What sections competitors are adding to their pages

Sites in competitive spaces routinely add or update content. For your highest priority keywords, your competitors are likely making frequent updates to their pages to try to improve their visibility. It can be difficult to track these changes.

Fortunately, the Wayback Machine allows you to understand what types of updates competitors are making to their content.

For example, we can use The Wayback Machine to look at the best cast iron skillet page from Serious Eats on June 27, 2021.

Of we took a look on their page today we can see that they have:

  • Added an “Editor’s Note” to the top of the article
  • Moved the “The Winners” section to the top of the page
  • Implemented internal links at the top of the first paragraph
  • Made “The Winners” section more visual
  • Added an FAQs section

This is extremely valuable information to have when performing a competitive analysis. These changes can now inform the editorial strategy we apply to our own page.

Determining the content differences can be difficult. It requires a manual review. However, you can use tools like Diffchecker to easily spot the content changes.

How frequently competitors are updating content

Use the Wayback Machine to determine how frequently competitors update content. This is especially useful if you’re in a SERP landscape where content freshness matters for visibility.