Data center

An operation engineer overseeing a network operations control room of a data center

A data center (American English) or data centre (British English) is a building or dedicated space within a building used to house computer systems and associated components, such as telecommunications and storage systems. It generally includes redundant or backup components and infrastructure for power supply, data communications connections, environmental controls (e.g. air conditioning, fire suppression) and various security devices. A large data center is an industrial-scale operation using as much electricity as a small town.

History

NASA mission control computer room circa 1962

Data centers have their roots in the huge computer rooms of the 1940s, typified by ENIAC, one of the earliest examples of a data center. Early computer systems, complex to operate and maintain, required a special environment in which to operate. Many cables were necessary to connect all the components, and methods to accommodate and organize these were devised such as standard racks to mount equipment, raised floors, and cable trays (installed overhead or under the elevated floor). A single mainframe required a great deal of power, and had to be cooled to avoid overheating. Security became important – computers were expensive, and were often used for military purposes. Basic design-guidelines for controlling access to the computer room were therefore devised.

During the boom of the microcomputer industry, and especially during the 1980s, users started to deploy computers everywhere, in many cases with little or no care about operating requirements. However, as information technology (IT) operations started to grow in complexity, organizations grew aware of the need to control IT resources. The advent of Unix from the early 1970s led to the subsequent proliferation of freely available Linux-compatible PC operating-systems during the 1990s. These were called "servers", as timesharing operating systems like Unix rely heavily on the client-server model to facilitate sharing unique resources between multiple users. The availability of inexpensive networking equipment, coupled with new standards for network structured cabling, made it possible to use a hierarchical design that put the servers in a specific room inside the company. The use of the term "data center", as applied to specially designed computer rooms, started to gain popular recognition about this time.

The boom of data centers came during the dot-com bubble of 1997–2000. Companies needed fast Internet connectivity and non-stop operation to deploy systems and to establish a presence on the Internet. Installing such equipment was not viable for many smaller companies. Many companies started building very large facilities, called Internet data centers (IDCs), which provide commercial clients with a range of solutions for systems deployment and operation. New technologies and practices were designed to handle the scale and the operational requirements of such large-scale operations. These practices eventually migrated toward the private data centers, and were adopted largely because of their practical results. Data centers for cloud computing are called cloud data centers (CDCs). But nowadays, the division of these terms has almost disappeared and they are being integrated into the term "data center".

With an increase in the uptake of cloud computing, business and government organizations scrutinize data centers to a higher degree in areas such as security, availability, environmental impact and adherence to standards. Standards documents from accredited professional groups, such as the Telecommunications Industry Association, specify the requirements for data-center design. Well-known operational metrics for data-center availability can serve to evaluate the commercial impact of a disruption. Development continues in operational practice, and also in environmentally-friendly data-center design. Data centers typically cost a lot to build and to maintain.

Requirements for modern data centers

Racks of telecommunications equipment in part of a data center

Modernization and data center transformation enhances performance and energy efficiency.

IT operations are a crucial aspect of most organizational operations around the world. One of the main concerns is business continuity; companies rely on their information systems to run their operations. If a system becomes unavailable, company operations may be impaired or stopped completely. It is necessary to provide a reliable infrastructure for IT operations, in order to minimize any chance of disruption. Information security is also a concern, and for this reason a data center has to offer a secure environment which minimizes the chances of a security breach. A data center must therefore keep high standards for assuring the integrity and functionality of its hosted computer environment. This is accomplished through redundancy of mechanical cooling and power systems (including emergency backup power generators) serving the data center along with fiber optic cables.

The Telecommunications Industry Association's Telecommunications Infrastructure Standard for Data Centers specifies the minimum requirements for telecommunications infrastructure of data centers and computer rooms including single tenant enterprise data centers and multi-tenant Internet hosting data centers. The topology proposed in this document is intended to be applicable to any size data center.

Telcordia GR-3160, NEBS Requirements for Telecommunications Data Center Equipment and Spaces, provides guidelines for data center spaces within telecommunications networks, and environmental requirements for the equipment intended for installation in those spaces. These criteria were developed jointly by Telcordia and industry representatives. They may be applied to data center spaces housing data processing or Information Technology (IT) equipment. The equipment may be used to:

• Operate and manage a carrier's telecommunication network
• Provide data center based applications directly to the carrier's customers
• Provide hosted applications for a third party to provide services to their customers
• Provide a combination of these and similar data center applications

Effective data center operation requires a balanced investment in both the facility and the housed equipment. The first step is to establish a baseline facility environment suitable for equipment installation. Standardization and modularity can yield savings and efficiencies in the design and construction of telecommunications data centers, both for now and for later.

Organizations are experiencing rapid IT growth but their data centers are aging. Industry research company International Data Corporation (IDC) puts the average age of a data center at nine years old. Gartner, another research company, says data centers older than seven years are obsolete. The growth in data (163 zettabytes by 2025) is one factor driving the need for data centers to modernize.

In May 2011, data center research organization Uptime Institute reported that 36 percent of the large companies it surveyed expect to exhaust IT capacity within the next 18 months.

Data center transformation takes a step-by-step approach through integrated projects carried out over time. This differs from a traditional method of data center upgrades that takes a serial and siloed approach. The typical projects within a data center transformation initiative include standardization/consolidation, virtualization, automation and security.

• Standardization/consolidation: Reducing the number of data centers and avoiding server sprawl (both physical and virtual) often includes replacing aging data center equipment, and is aided by standardization.
• Virtualize: IT virtualization technologies help to lower capital and operational expenses, and reduce energy consumption. Virtualization technologies are also used to create virtual desktops, which can then be hosted in data centers and rented out on a subscription basis. Investment bank Lazard Capital Markets estimated in 2008 that 48 percent of enterprise operations will be virtualized by 2012. Gartner views virtualization as a catalyst for modernization.
• Automating: Automating tasks such as provisioning, configuration, patching, release management and compliance is needed, not just when facing fewer skilled IT workers.
• Securing: Protection of virtual systems is integrated with existing security of physical infrastructures.

Machine room

The term "Machine Room" is at times used to refer to the large room within a Data Center where the actual Central Processing Unit is located; this may be separate from where high-speed printers are located. Air conditioning is most important in the machine room.

Aside from air-conditioning, there must be monitoring equipment, one type of which is to detect water prior to flood-level situations. One company, for several decades, has had share-of-mind: Water Alert. The company, as of 2018, has 2 competing manufacturers (Invetex, Hydro-Temp) and 3 competing distributors (Longden,Northeast Flooring,, Slayton).

Raised floor

Main article: Raised floor

A raised floor standards guide named GR-2930 was developed by Telcordia Technologies, a subsidiary of Ericsson.

Although the first raised floor computer room was made by IBM in 1956, and they've "been around since the 1960s," it was the 1970s that made it more common for computer centers to thereby allow cool air to circulate more efficienctly.

The first purpose of the raised floor was to allow access for wiring.

Lights out

The "lights-out" data center, also known as a darkened or a dark data center, is a data center that, ideally, has all but eliminated the need for direct access by personnel, except under extraordinary circumstances. Because of the lack of need for staff to enter the data center, it can be operated without lighting. All of the devices are accessed and managed by remote systems, with automation programs used to perform unattended operations. In addition to the energy savings, reduction in staffing costs and the ability to locate the site further from population centers, implementing a lights-out data center reduces the threat of malicious attacks upon the infrastructure.

Data center levels and tiers

The two organizations in the United States that publish data center standards are the Telecommunications Industry Association (TIA) and the Uptime Institute.

Telecommunications Industry Association

Main article: TIA-942

The Telecommunications Industry Association is a trade association accredited by ANSI (American National Standards Institute). In 2005 it published ANSI/TIA-942, Telecommunications Infrastructure Standard for Data Centers, which defined four levels of data centers in a thorough, quantifiable manner. TIA-942 was amended in 2008, 2010, 2014 and 2017.

The simplest requirements for the data center infrastructure is a Level 1 data center, which is basically a server room, following basic guidelines for the installation of computer systems. The most stringent level is a Level 4 data center, which is designed to host the most mission critical computer systems, with fully redundant subsystems, the ability to continuously operate for an indefinite period of time during primary power outages.

Data availability

Data availability is a term used by computer storage manufacturers and storage service providers (SSPs) to describe products and services that ensure that data continues to be available at a required level of performance in situations ranging from normal through "disastrous."

Any time a server loses power, for example, it has to reboot, recover data and repair corrupted data. The time it takes to recover, known as the mean time to recover (MTR), could be minutes, hours or days.

Uptime Institute - Data Center Tier Standards

Four tiers are define by the Uptime Institute:

• Tier I: lacks redundant IT equipment, with 99.671% availability, maximum of 1729 minutes annual downtime
• Tier II: adds redundant infrastructure - 99.741% availability (1361 minutes)
• Tier III: adds more data paths, duplicate equipment, and that all IT equipment must be dual-powered (99.982%, 95 minutes)
• Tier IV: all cooling equipment is independently dual-powered; adds Fault-tolerance (99.995%, 26 minutes)

Data center design

The field of data center design has been growing for decades in various directions:

• Major data centers did not appear in India until the late 1990s.
• a 65 story data center has already been proposed
• the number of data centers as of 2016 had grown beyond 3 million USA-wide, and more than triple

that number worldwide

Local building codes may govern the minimum ceiling heights and other parameters. Some of the considerations in the design of data centers are:

A typical server rack, commonly seen in colocation

• size - one room of a building, one or more floors, or an entire building, and can hold 1,000 or more servers
• space, power, cooling, and costs in the data center.

CRAC Air Handler

• Mechanical engineering infrastructure - heating, ventilation and air conditioning (HVAC); humidification and dehumidification equipment; pressurization.
• Electrical engineering infrastructure design - utility service planning; distribution, switching and bypass from power sources; uninterruptible power source (UPS) systems; and more.

Design criteria and tradeoffs

• Availability expectations: Cost of avoiding downtime should not exceed the cost of downtime itself
• Site selection: Location factors include proximity to power grids, telecommunications infrastructure, networking services, transportation lines and emergency services. Others are flight paths, neighbouring uses, geological risks and climate (associated with cooling costs).
  • Often available power is hardest to change.

Modularity and flexibility

Cabinet aisle in a data center

Main article: Modular data center

Modularity and flexibility are key elements in allowing for a data center to grow and change over time. Data center modules are pre-engineered, standardized building blocks that can be easily configured and moved as needed.

A modular data center may consist of data center equipment contained within shipping containers or similar portable containers. Components of the data center can be prefabricated and standardized which facilitates moving if needed.

Environmental control

Main article: Data center environmental control

The physical environment of a data center is rigorously controlled. Air conditioning is used to control the temperature and humidity in the data center; indirect cooling, such as using outside air, is also increasingly being implemented.

Electrical power

A bank of batteries in a large data center, used to provide power until diesel generators can start

Backup power consists of one or more uninterruptible power supplies, battery banks, and/or diesel / gas turbine generators.

To prevent single points of failure, all elements of the electrical systems, including backup systems, are typically fully duplicated, and critical servers are connected to both the "A-side" and "B-side" power feeds. This arrangement is often made to achieve N+1 redundancy in the systems. Static transfer switches are sometimes used to ensure instantaneous switchover from one supply to the other in the event of a power failure.

Low-voltage cable routing

Data cabling is typically routed through overhead cable trays in modern data centers. But some are still recommending under raised floor cabling for security reasons and to consider the addition of cooling systems above the racks in case this enhancement is necessary. Smaller/less expensive data centers without raised flooring may use anti-static tiles for a flooring surface. Computer cabinets are often organized into a hot aisle arrangement to maximize airflow efficiency.

Fire protection

FM200 Fire Suppression Tanks

Data centers feature fire protection systems, including passive and Active Design elements, as well as implementation of fire prevention programs in operations. Smoke detectors are usually installed to provide early warning of a fire at its incipient stage.

Two water-based options are

• sprinkler
• mist
• No water - some of the benefits of using chemical suppression (clean agent fire suppression gaseous system).

Security

Main article: Data center security

Physical security also plays a large role with data centers. Physical access to the site is usually restricted to selected personnel, with controls including a layered security system often starting with fencing, bollards and mantraps. Video camera surveillance and permanent security guards are almost always present if the data center is large or contains sensitive information on any of the systems within. The use of finger print recognition mantraps is starting to be commonplace.

Documenting access is required by some data protection regulations. To do so, some organizations use access control systems that provide a logging report of accesses. Logging can occur at the main entrance, at the entrances to mechanical rooms and white spaces, as well as in at the equipment cabinets. Modern access control at the cabinet allows for integration with intelligent power distribution units so that the locks can be powered and networked through the same appliance.

Energy use

Google Data Center, The Dalles, Oregon

Main article: IT energy management

Energy use is a central issue for data centers. Power draw for data centers ranges from a few kW for a rack of servers in a closet to several tens of MW for large facilities. Some facilities have power densities more than 100 times that of a typical office building. For higher power density facilities, electricity costs are a dominant operating expense and account for over 10% of the total cost of ownership (TCO) of a data center. By 2012 the cost of power for the data center is expected to exceed the cost of the original capital investment.

According to a Greenpeace study, in 2012, data centers represented 21% of the electricity consumed by the IT sector, which was about 382 billion kWh a year. U.S. data centers use more than 90 billion kWh of electricity a year. Global data centers used roughly 416 TWh in 2016, nearly 40% more than the entire United Kingdom.

Greenhouse gas emissions

In 2007 the entire information and communication technologies or ICT sector was estimated to be responsible for roughly 2% of global carbon emissions with data centers accounting for 14% of the ICT footprint. The US EPA estimates that servers and data centers are responsible for up to 1.5% of the total US electricity consumption, or roughly .5% of US GHG emissions, for 2007. Given a business as usual scenario greenhouse gas emissions from data centers is projected to more than double from 2007 levels by 2020.

Siting is one of the factors that affect the energy consumption and environmental effects of a datacenter. In areas where climate favors cooling and lots of renewable electricity is available the environmental effects will be more moderate. Thus countries with favorable conditions, such as: Canada, Finland, Sweden, Norway and Switzerland, are trying to attract cloud computing data centers.

In an 18-month investigation by scholars at Rice University's Baker Institute for Public Policy in Houston and the Institute for Sustainable and Applied Infodynamics in Singapore, data center-related emissions will more than triple by 2020.

Energy efficiency

The most commonly used metric to determine the energy efficiency of a data center is power usage effectiveness, or PUE. This simple ratio is the total power entering the data center divided by the power used by the IT equipment.

$\mathrm {PUE} ={{\mbox{Total Facility Power}} \over {\mbox{IT Equipment Power}}}$

Total facility power consists of power used by IT equipment plus any overhead power consumed by anything that is not considered a computing or data communication device (i.e. cooling, lighting, etc.). An ideal PUE is 1.0 for the hypothetical situation of zero overhead power. The average data center in the US has a PUE of 2.0, meaning that the facility uses two watts of total power (overhead + IT equipment) for every watt delivered to IT equipment. State-of-the-art data center energy efficiency is estimated to be roughly 1.2. Some large data center operators like Microsoft and Yahoo! have published projections of PUE for facilities in development; Google publishes quarterly actual efficiency performance from data centers in operation.

The U.S. Environmental Protection Agency has an Energy Star rating for standalone or large data centers. To qualify for the ecolabel, a data center must be within the top quartile of energy efficiency of all reported facilities. The United States passed the Energy Efficiency Improvement Act of 2015, which requires federal facilities — including data centers — to operate more efficiently. In 2014, California enacted title 24 of the California Code of Regulations, which mandates that every newly constructed data center must have some form of airflow containment in place, as a measure to optimize energy efficiency.

European Union also has a similar initiative: EU Code of Conduct for Data Centres

Energy use analysis

Often, the first step toward curbing energy use in a data center is to understand how energy is being used in the data center. Multiple types of analysis exist to measure data center energy use. Aspects measured include not just energy used by IT equipment itself, but also by the data center facility equipment, such as chillers and fans. Recent research has shown the substantial amount of energy that could be conserved by optimizing IT refresh rates and increasing server utilization.

Power and cooling analysis

Power is the largest recurring cost to the user of a data center. A power and cooling analysis, also referred to as a thermal assessment, measures the relative temperatures in specific areas as well as the capacity of the cooling systems to handle specific ambient temperatures. A power and cooling analysis can help to identify hot spots, over-cooled areas that can handle greater power use density, the breakpoint of equipment loading, the effectiveness of a raised-floor strategy, and optimal equipment positioning (such as AC units) to balance temperatures across the data center. Power cooling density is a measure of how much square footage the center can cool at maximum capacity. The cooling of data centers is the second largest power consumer after servers. The cooling energy varies from 10% of the total energy consumption in the most efficient data centers and goes up to 45% in standard air-cooled data centers.

Energy efficiency analysis

An energy efficiency analysis measures the energy use of data center IT and facilities equipment. A typical energy efficiency analysis measures factors such as a data center's power use effectiveness (PUE) against industry standards, identifies mechanical and electrical sources of inefficiency, and identifies air-management metrics. However, the limitation of most current metrics and approaches is that they do not include IT in the analysis. Case studies have shown that by addressing energy efficiency holistically in a data center, major efficiencies can be achieved that are not possible otherwise.

Computational fluid dynamics (CFD) analysis

Main article: Computational fluid dynamics

This type of analysis uses sophisticated tools and techniques to understand the unique thermal conditions present in each data center—predicting the temperature, airflow, and pressure behavior of a data center to assess performance and energy consumption, using numerical modeling. By predicting the effects of these environmental conditions, CFD analysis in the data center can be used to predict the impact of high-density racks mixed with low-density racks and the onward impact on cooling resources, poor infrastructure management practices and AC failure or AC shutdown for scheduled maintenance.

Thermal zone mapping

Thermal zone mapping uses sensors and computer modeling to create a three-dimensional image of the hot and cool zones in a data center.

This information can help to identify optimal positioning of data center equipment. For example, critical servers might be placed in a cool zone that is serviced by redundant AC units.

Green data centers

Main article: Green Data Center

This water-cooled data center in the Port of Strasbourg, France claims the attribute green.

Data centers use a lot of power, consumed by two main usages: the power required to run the actual equipment and then the power required to cool the equipment. The first category is addressed by designing computers and storage systems that are increasingly power-efficient. To bring down cooling costs data center designers try to use natural ways to cool the equipment. Many data centers are located near good fiber connectivity, power grid connections and also people-concentrations to manage the equipment, but there are also circumstances where the data center can be miles away from the users and don't need a lot of local management. Examples of this are the 'mass' data centers like Google or Facebook: these DC's are built around many standardized servers and storage-arrays and the actual users of the systems are located all around the world. After the initial build of a data center staff numbers required to keep it running are often relatively low: especially data centers that provide mass-storage or computing power which don't need to be near population centers.Data centers in arctic locations where outside air provides all cooling are getting more popular as cooling and electricity are the two main variable cost components.

Energy reuse

The practice of cooling data centers is a topic of discussion. It is very difficult to reuse the heat which comes from air cooled data centers. For this reason, data center infrastructures are more often equipped with heat pumps. An alternative to heat pumps is the adoption of liquid cooling throughout a data center. Different liquid cooling techniques are mixed and matched to allow for a fully liquid cooled infrastructure which captures all heat in water. Different liquid technologies are categorised in 3 main groups, Indirect liquid cooling (water cooled racks), Direct liquid cooling (direct-to-chip cooling) and Total liquid cooling (complete immersion in liquid). This combination of technologies allows the creation of a thermal cascade as part of temperature chaining scenarios to create high temperature water outputs from the data center.

Network infrastructure

An example of "rack mounted" servers

Communications in data centers today are most often based on networks running the IP protocol suite. Data centers contain a set of routers and switches that transport traffic between the servers and to the outside world which are connected according to the data center network architecture. Redundancy of the Internet connection is often provided by using two or more upstream service providers (see Multihoming).

Some of the servers at the data center are used for running the basic Internet and intranet services needed by internal users in the organization, e.g., e-mail servers, proxy servers, and DNS servers.

Network security elements are also usually deployed: firewalls, VPN gateways, intrusion detection systems, etc. Also common are monitoring systems for the network and some of the applications. Additional off site monitoring systems are also typical, in case of a failure of communications inside the data center.

Offsite backup storage

Main article: Disaster recovery § offsite backup storage

Data centers are also used for off site backups. Companies may subscribe to backup services provided by a data center. This is often used in conjunction with backup tapes. Backups can be taken off servers locally on to tapes. However, tapes stored on site pose a security threat and are also susceptible to fire and flooding. Larger companies may also send their backups off site for added security. This can be done by backing up to a data center. Encrypted backups can be sent over the Internet to another data center where they can be stored securely.

Data backup techniques includes having a copy of the data offsite. Methods used for transporting data are:

• having the customer write the data to a physical medium, such as magnetic tape, and then transporting the tape elsewhere
• directly transferring the data to another site during the backup, using appropriate links
• uploading the data "into the cloud"

Applications

A 40-foot Portable Modular Data Center

For quick deployment or disaster recovery, several large hardware vendors have developed mobile/modular solutions that can be installed and made operational in very short time.

See also

• Colocation center
• Data center management
• Disaster recovery
• Dynamic Infrastructure
• Electrical network
• Internet exchange point
• Internet hosting service
• Neher–McGrath
• Network operations center
• Open Compute Project, by Facebook
• Peering
• Server farm
• Server room
• Telecommunications network
• Utah Data Center
• Web hosting service

External links

• Lawrence Berkeley Lab - Research, development, demonstration, and deployment of energy-efficient technologies and practices for data centers
• DC Power For Data Centers Of The Future - FAQ: 380VDC testing and demonstration at a Sun data center.
• White Paper - Property Taxes: The New Challenge for Data Centers
• The European Commission H2020 EURECA Data Centre Project - Data centre energy efficiency guidelines, extensive online training material, case studies/lectures (under events page), and tools.