This weekend the Computing & Information Technology (opens new window) (C&IT) department upgraded the power in the data center. An explanation of the reason for the upgrade can be found at the ProfTech blog (opens new window). In addition, C&IT has an announcement of what services were effected (opens new window) on their website.

How this effected the Wayne State Web

In short, this meant every website hosted in the main Web server environment would be down. Basically shutting our visitors out for 10 hours. This included the homepage, admissions website, application, events calendar, api, and content management system, just to name a few.

The down time user experience

It's never a good experience when you click on a dead link. C&IT brought us in the loop early and we tried to come up with a plan to keep the servers online during the maintenance period. Unfortunately the entire datastore would be down, and moving a read only version to our off campus would take longer than the maintenance period itself. We decided a single maintenance page made the most sense for the time of day and number of visitors that it would affect.

We designed the page based on the promotional images we used around campus to warn students/staff about the maintenance. We kept it simple and gave the user some calls to action directing them to additional information. Below is a screenshot of the page.

The maintenance screen

(opens new window)

"Have a little fun"

If you notice on the page there is a second link to "have a little fun (opens new window)". We wanted to give anyone who was unfortunate enough to land on the dead end page something to do to pass the time and show a little personality.

One of our former developers, Nick West (opens new window), was playing around with javascript and gaming a while back and came up with this exploding W page. We passed it around internally for a while but never had a chance to use it publicly. We thought this would be a perfect opportunity. With his permission, we added it to our maintenance page to see how many people found it.

(opens new window)

During the maintenance period only ~9% of visitors clicked through to the "have a little fun" and spent an average of two minutes on the page. We expected about this percentage and amount of time on the page, but I explain why the actual quantity of visitors was quite a bit lower than we expected below.

Lessons learned

The goal was to have this maintenance page come up when any page was accessed with a "503" (the server is temporarily unavailable) response and a "retry-after" so Google and other search engines didn't index the temporary page. Coincidentally an article was posted on SEOMoz (opens new window) just days before our downtime which outlines the best practices for handling maintenance situations.

Everything seemed to be planned well for the maintenance but we encountered two issues which prevented us from analyzing the downtime completely.

1. The .htaccess file didn't get included in the files that were sent to C&IT. (completely my fault for not checking)
2. The Google Analytics account did not have the "full domain" filter enabled. (again, an oversight on my part)

Because of these two issues the maintenance screen was only displayed on the homepage of each domain, not on every single file accessed. That cuts out a major chunk (like 95%) of the traffic to our server and potentially hurt some page ranks. Lastly, the oversight of the "full domain" filter in the Google Analytics prevented us from seeing exactly where the traffic was from. Seeing just "/" and "/w/" give us absolutely no insight.

Test, test, then retest

In the end we were glad to have at least some explanation of the down time up, but because of these two issues I don't have much insight to share here. I am taking this situation as another opportunity for the importance of testing and retesting. Having a dry run of any IT or Web related activity may take a little extra time but in the end will produce the best results.

View the maintenance page: http://wayne.edu/maintenance/ (opens new window)

Have a little fun: http://wayne.edu/maintenance/w/ (opens new window)