Accidental 404 Errors... Ugh.

You wouldn't think it would be possible to accidentally issue a 404 error code for every page request to a site. But I'm a particularly talented individual and that's just what I did. You see, apparently I had seen it all so why not go ahead and create brand new problems!?

I make it a point to log into my master analytics account and check out traffic levels for the sites I regularly work on. I typically just look at traffic changes over the last few weeks to identify problems. It takes deeper analysis to actually affect the performance of a site, but traffic is a great diagnostic metric. At the end of January, after a week long hiatus, I saw this:

Figure 1: Traffic Drops Precipitously

Some more digging revealed this wonderful graph showing a spike in 404 errors.

Figure 2: 404 Errors Spike

To confirm the issue I logged into Google's Webmaster Tools. Sure enough, about 75% of the URLs in my XML sitemap were flagged as having generated a 404. The remaining 25% probably weren't crawled during the week before I found the problem.

What's an SEO to do when a site gets de-indexed due because of accidentally issued 404 errors? Obviously the first order of business is to fix the problem. Turns out there's a funky bug in WordPress which took me about 4 hours to uncover. The much harder task still lay ahead of me. How do I get Google to re-index my site? I had 5 ideas immediately pop into my head:

1. Submit an updated XML sitemap with every date stamp set to the current date. Done.
2. Create some new content with links to the de-indexed content knowing that the FeedBurner pinging service will bring Google in quickly. Done.
3. Take advantage of the $1 SEOmoz Pro subscription I was evaluating and ask the pros there for their opinion. Hey, my head is not so big that I won't seek ideas from other SEOs. The response I got was that I need to be patient and let things play out. Probably the right answer, but I was hoping there was something I could actually do.
4. Crawl my site with a browser from a few locations and hope the AdSense ads trigger a visit from Google. A few servers and the iMacros plugin took care of this a few times a day for a few days. Note: I wasn't clicking ads, just crawling my own site. Seemed like an OK thing to do.
5. Redirect some indexed URLs to the de-indexed URLs and hope the content is re-indexed. Seemed a little shady so I passed on this one.

Google was crazy slow with re-indexing my site. Every week I saw single digit percentage increases in organic traffic. Only 4 weeks later did I know my site was back in business -- a more than doubling in traffic from one day to the next along with a return to near 100% inclusion in Google's index.

Figure 3: Traffic Climbs

So while I'm pleased to have recovered, this little excursion has cost me about 5 weeks of revenue. Ugh. If only Google Analytics had alert functionality I likely would've caught this much earlier.