Digital Foundry vs. ApocalyPS3

The panic is over. Just as quickly as they were crippled, "fat" PlayStation 3 consoles worldwide started working again as their faulty internal clocks moved away from the fateful date of 1st March 2010. Sony can breathe a deep sigh of release: no firmware update patch will be needed (for a while at least), no hyper-expensive recalls will be required. Everything is as it was. Balance has been restored.

So, what actually happened? And will it happen again? Throughout the debacle, Sony referred to the issue erroneously as a "network connectivity" problem, with the blame pinned squarely on poor old PSN.

However, as gamers rallied and experiences were shared, it quickly became obvious that the issue was far more serious: consoles that were disconnected from the internet displayed the same problems as those that accessed PSN. This wasn't anything to do with the online service, but everything to do with a specific piece of hardware within the PlayStation 3.

The PS3's internal clock is completely invisible to the end user and is used to sync with PSN, as well as time-stamping trophies and downloadable content activation certificates. Yesterday, with this invisible master clock now set to a date that simply didn't exist due to a misunderstanding of leap years, most trophy-supported games wouldn't launch either online or offline, PSN couldn't be accessed, and the activation certificates for downloaded content became invalid. Those affected were left with a games console that wouldn't play games.

Early yesterday afternoon, PS3 users on the #Efnet IRC network claimed to have pinpointed the problem. A small ARM Syscon CPU dedicated to menial tasks within the PlayStation 3 was known to have an issue dealing with leap years. The same piece of silicon had a history of causing similar aggravation for mobile devices with the likes of Zune and some Blackberry devices afflicted. In those cases, MSFT and RIMM were on top of the issue. Sony, unfortunately, was not.

As it happened, the fix for this issue was remarkably straightforward. Similar to a PC motherboard exhibiting CMOS memory issues, the ApocalyPS3 could be resolved by opening up the PlayStation 3, removing the button-shaped battery and letting the power dissipate internally. If you re-inserted the battery and reconstructed your PS3 10 minutes later you were good to go. Great if you're confident dealing with electronics and happen to have a set of the special Torx security screwdrivers needed to disassemble the PS3. Not so great if you're just a regular gamer, nor if you fancied retaining your warranty (if you still have one for a "fat" PS3 anyway).

In the here and now, the problem has righted itself and a patch for the ARM controller can be incorporated at Sony's own pace into a future firmware update. Internally the company must surely be relieved about the bullet it has dodged. Unofficial community-compiled lists of "fat" PS3 hardware affected by the problem suggest that of 11 different SKUs, eight of them use the bugged controller chip, resulting in a colossal amount of affected users.

This isn't just thousands or tens of thousands of units, it's surely millions of them worldwide. While an internet-delivered mandatory firmware update could have solved the problem, it's difficult to imagine how Sony would have handled the multitudes of these units in homes with limited or non-existent internet connectivity.

Thankfully such desperate measures not required. Power up your PS3 and you're back in action. The raging, NSFW fat bloke on YouTube can return to "owning noobs" and that comment-writer on the EU PlayStation blog can probably revise down his estimate that ApocalyPS3 is equivalent to "9/11 x 1,000". However, one of the most disturbing things about the whole episode was the lack of accurate or worthwhile information from Sony itself.

While pointing out the slim owners were safe in its initial blog posting, there was never any acknowledgement that offline consoles didn't work either. Promised updates on Sony's Twitter page never happened in a timely manner, nor offered much in the way of useful information.

As the real cause of the issue became common knowledge within the community by mid-morning, Sony urged gamers not to "trust info regarding this issue unless from an official Sony source" but failed to produce any meaningful information until the late afternoon when the internal clock issue was finally confirmed to be the culprit. The lack of clear, unambiguous information on the root cause and how it would be fixed was disappointing to say the least.

Clearly there are lessons to be learned here and it will be interesting to see how Sony chooses to handle the aftermath.