California’s extreme heat leaves Twitter’s main data center down

Extreme heat in California has left Twitter without one of its main data centers, and a company executive warned in an internal memo obtained by the CNN that another outage elsewhere could result in service shutdown for some of its users.

Twitter, like all major social media platforms, relies on data centers, which are essentially large warehouses filled with computers, including servers and storage systems.

Controlling the temperature in these centers is critical to ensuring that computers do not overheat and malfunction.

To save on cooling costs, some technology companies are increasingly looking to place their data centers in cooler climates. Google, for example, opened a data center in Finland in 2011, and Meta has had a center in northern Sweden since 2013.

“On September 5, Twitter suffered the loss of its Sacramento datacenter region due to extreme weather. The unprecedented event resulted in the complete shutdown of physical equipment at SMF,” Carrie Fernandez, the company’s vice president of engineering, said in a statement. internal message to Twitter engineers on Friday (9).

Leading tech companies often have multiple data centers, in part to ensure their services stay online if one fails; this is known as redundancy.

As a result of the Sacramento outage, Twitter is in a “non-redundant state,” according to Fernandez’s Friday memo.

She explained that Twitter’s Atlanta and Portland datacenters are still operational, but cautioned, “If we lose one of these remaining datacenters, we may not be able to deliver traffic to all Twitter users.”

The memo continues to prohibit non-critical product updates from Twitter until the company can fully restore its Sacramento data center services.

“All production changes, including deployments and releases to mobile platforms, are blocked, with the exception of changes needed to address continuity of service or other urgent operational needs,” Fernandez wrote.

The restrictions highlight the apparent fragility of some of Twitter’s most fundamental systems, a problem that Peiter “Mudge” Zatko, a former Twitter security chief turned whistleblower, raised in a disclosure sent to lawmakers and government agencies in July.

In its disclosure of whistleblowers, reported by CNN and by The Washington Post, Zatko warned that Twitter had “insufficient data center redundancy” that increased the risk of a brief service outage or even the prospect of Twitter going offline forever.

“Even a temporary but overlapping outage of a small number of data centers would likely result in service [Twitter] going offline for weeks, months or permanently,” according to Zatko’s disclosure.

Twitter has criticized Zatko and has largely defended itself against the allegations, saying the disclosure paints a “false narrative” by the company.

News of the data center outage comes a day before Zatko is due to testify before the Senate Judiciary Committee.

Twitter has not disclosed the number or location of its data centers, but the Zatko whistleblower’s disclosure cites a public report identifying Twitter’s data center in Sacramento and another in Atlanta.

In 2020, Amazon announced that Twitter had selected its cloud computing platform, Amazon Web Services, to serve some tweets from Amazon’s data centers.

In a statement about the Sacramento outage, a Twitter spokesperson told CNN : “There have been no disruptions affecting people’s ability to access and use Twitter at this time. Our teams remain equipped with the tools and resources they need to send updates and continue working to provide a seamless Twitter experience.”

Data centers need “reliable water, power, humidity controls and cooling to live,” said retired Brigadier General Greg Touhill, who served as the U.S. government’s director of information security in 2016 and 2017.

“You want redundancy, not duplication, of your data locations to improve your cyber resiliency so you can weather a natural disaster [ou outro evento] that can bring down a single piece of equipment or data center,” Touhill, who now heads the CERT Division of the Software Engineering Institute at Carnegie Mellon University, told CNN .