chapter 5 A step-by-step

guide to greening
the data center
3 Persuading the C suite
9 Measuring consumption
12 Managing assets
19 Optimizing infrastructure
26 Summary steps

SPONSORED BY
Energy-efficient computing in the 21st century BY MATT STANSBERRY
CHAPTER 5:A Step-by-Step Guide
to Greening the Data Center
Data
IN THE PREVIOUS four chapters of this e-book, we have laid center
out the case for going green in the data center and the
tactics to achieve energy efficiency across server, storage
energy
and infrastructure environments. This chapter summarizes efficiency
strategies for going green in the data center and is designed begins
to help IT managers create a holistic strategy for a data
center efficiency project. with
operations
STEP 1: Persuading the C suite to pursue energy efficiency teams.
STEP 2: Measuring data center energy consumption
STEP 3: IT asset management and purchasing decisions
STEP 4: Infrastructure optimization techniques

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 2
chapter 5 A step-by-step guide to greening the data center

STEP 1: Persuading the C Suite

to Pursue Energy Efficiency
AS OUTLINED IN previous chapters of this sioned server be connected to an
e-book, data center energy efficiency essential application.
begins with operations teams. Data
center managers and IT and facilities BIG-PICTURE SCENARIOS. So how do you C-level executives
staffers contend with data center power get a mandate to go green and unplug understand the
constraints and are often the only unnecessary servers (and other equip- concept of going
employees in a position to do some- ment)? For data center managers, the green. These
thing about them. process begins with speaking an execu-
But without executive buy-in, data tive’s language. Most executives do not days, in fact,
center efficiency projects are doomed relate to bits and BTUs. They look at the executives throw
to fail. IT departments are not motivated big picture. So you need to present your the term around
to reduce their power footprint. Data case from the big-picture perspective. more frequently
center systems are so fragile that than Al Gore.
IT staffers are often unwilling to re- CORPORATE SUSTAINABILITY. C-level
engineer systems to run more efficiently. executives understand the concept of
Experts estimate that 10% to 30% going green. These days, in fact, execu-
of data center servers don’t actually do tives throw the term around more fre-
anything, but managers aren’t motivated quently than Al Gore. So, for example,
to audit and decommission unused someone at the top of the chain writes in
servers. Often they fear they will jeop- a press release, “We’re going to be car-
ardize their careers should a decommis- bon neutral by 2010” but doesn’t convey

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 3
chapter 5 A step-by-step guide to greening the data center

the idea to the IT department. Only trees in Brazil or building wind turbines
when it becomes clear that a company’s in India. But carbon-offset companies
data center is responsible for a huge are unregulated and often unaffiliated
chunk of the company’s carbon footprint with official environmental agencies or
does IT get invited into the conversation. standards. And even when companies
When a company pledges carbon do due diligence on their carbon repara-
Many experts
neutrality, it often means it has various tions, many experts say the scheme say the whole
plans, including sponsoring public trans- of carbon offsetting is misleading and, scheme of
portation and telecommuting options for worse, ineffective because it doesn’t carbon offsetting
employees to reduce gasoline consump- fix the problem of consumption. is misleading
tion; pledging to reduce operational So rather than throw hundreds of
energy consumption; and buying carbon thousands of dollars into the effort to
and, worse,
offsets. Silicon Valley giants Dell Inc., offset the CO2 your company pumps ineffective.
Hewlett-Packard Co., IBM Corp., Yahoo into the atmosphere (as Salesforce.com
Inc., Google Inc. and Salesforce.com Inc. did in 2007), put that money toward a
have pursued one or more of these tac- server refresh and purchase more effi-
tics to garner corporate goodwill. cient hardware or perform an analysis
Unfortunately, many companies don’t of computational fluid dynamics (CFD)
plan to change much about their energy to audit your data center’s layout and
consumption and instead throw money improve cooling efficiency.
at the problem rather than truly fix it, Learn which of your company’s
as is the case with carbon offsets. departments have gone green and
With carbon offsets, a corporation persuade them to sponsor your project.
pledges to pay a third-party company to Now more than ever, as executive teams
invest in renewable energy or plant trees spout green rhetoric, this is the ideal
to sequester carbon dioxide from the time to present a data center project
atmosphere. Examples include planting to management.

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 4
chapter 5 A step-by-step guide to greening the data center

SHOW ME THE MONEY. As demand for San Francisco-based data center real
raised-floor real estate continues to heat estate firm Digital Realty Trust Inc. found
up, the cost of building data centers has similar figures: Over the next 24 months,
skyrocketed. We covered this issue in more than 80% of U.S. companies plan
Chapter 1 (“Building a business case for to expand their data centers.
data center efficiency”), but here is a Many companies will build new data
statistic to bring to the table: The Uptime center facilities because they have run
Institute, a Santa Fe, N.M.-based consor- out of power and cooling capacity and
tium of companies dedicated to energy literally can’t supply more power to the
efficiency, estimates that a 30,000- building. To deal with this issue, some
square-foot data center can cost as companies have cannibalized their
much as $300 million today. Compare power and cooling redundancy, sacrific- The potential
cost of building a
that figure with only $20 million a few ing N+1 cooling and power for greater
30,000-square-
years ago. capacity. But over the long term, this foot data center
According to our 2007 data center isn’t a workable solution. today, compared
construction survey, 82% of respon- If your company can scale back data with only
dents said that between 2007 and 2008 center energy use, it can potentially $20 million a few
they would be involved in a renovation or forestall a nine-figure capital expendi- years ago.
construction project. A 2007 study from ture. Experts estimate that, with best

$300million PERSUADING
THE C SUITE
MEASURING
CONSUMPTION
MANAGING
ASSETS
OPTIMIZING
INFRASTRUCTURE 5
chapter 5 A step-by-step guide to greening the data center

practices and modern equipment, data sound bite? You don’t have two hours
center operators could realistically cut to present metrics.
energy consumption by 50%. Bringing “You really have to give up the whole
these CapEx and OpEx scenarios to a technical aspect of it and have to put
CFO is a surefire way to get attention your business hat on,” said Robert
for your project. Rosen, the former president of Share
If you want
Other potential scenarios include (an IBM mainframe user group) and the to present
leveraging the competitive landscape: CIO of the National Institute of Arthritis executives with
If you’re in the hosting business, for and Musculoskeletal and Skin Diseases a grand plan to
example, and your data centers are per- in Bethesda, Md. “At that level, they save energy in
ceived as “greener” and less costly to don’t care about nuts and bolts. You
operate than those of your competitors, have to come at this as a business
the data center,
your company has a major advantage. problem,” Rosen said. don’t talk about
In 2007, scores of companies jumped If, for example, you want to present saving kilowatts.
on the green bandwagon to beat the executives with a grand plan to save
competition on public image. But more energy in the data center, don’t
important, if delivering IT services approach the issue by talking about
is your business, operating efficiently saving kilowatts. You have to say, “We
is a business requirement. are spending X dollars to provide this
service today, and we can reduce that
TALKING IN EXECUTIVES’ LANGUAGE. Once amount while providing the same level
you solidify the big-picture business of service.”
case for a data center energy-efficiency “You’ve got to put it into terms they’re
project, you need to hone your presenta- interested in or their eyes are going to
tion skills and learn to speak in the lan- glaze over,” Rosen said. “It’s Salesman-
guage of your audience. How do you ship 101. You’re a salesman, trying to
package this project for a 30-second sell your project. Why is it in the best

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 6
chapter 5 A step-by-step guide to greening the data center

interest of the person [to whom] you’re is, you won’t get the green light on a
selling to buy your project?” Usually the project without steps like these to back
answer comes down to money. you up:
Another tip is to research executives’
pet projects. What would a C-level exec- 1. Determine who pays the data
utive want to spend money on after you center electric bill.
“Learn how to get
create major savings from a power bill? • Identify the cost per month to keep your point across
Executives want to be able to answer the servers running. without falling
question “What’s in it for me?” • Determine over, say, the past 24 into the techno
Rosen also recommends that IT staff months, the trend for your data center trap. You have to
get training on making a presentation. power costs.
“Learn how to get your point across
understand the
without falling into the techno trap,” he 2. Work with your application teams to audience and
counseled. “You have to understand your project your IT infrastructure growth. have a pitch
audience and have a pitch appropriate • Determine how much additional appropriate to
to that audience.” capacity you need in the coming period. that audience.”
While it might sound cliché, Rosen • Identify whether your large projects
—ROBERT ROSEN,
also suggests Dale Carnegie’s classic could affect future data center demand. CIO OF THE NATIONAL
self-improvement book How to Win INSTITUTE OF ARTHRITIS
AND MUSCULOSKELETAL
Friends and Influence People. User groups 3. Determine a metric that you can AND SKIN DISEASES
like AFCOM, Share and the Uptime use consistently.
Institute teach the soft skills necessary • Identify how efficient your data centers
to sell a project to management as well. are.
• Determine how efficient they can be.
THE THREE-STEP PROCESS FOR SELLING
DATA CENTER EFFICIENCY. But no matter Next, we discuss how to measure data
how slick your PowerPoint presentation center energy consumption. 

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 7
 CONSERVE POWER CLOSE-COUPLED COOLING CAPACITY MANAGEMENT
Oversized legacy systems Gain thermal efficiency with InRow™ cooling architecture. Optimize management efficiency with
waste power. Benefit from Closely coupling the AC with the heat source reduces the intelligent, integrated capacity management
power efficiency by right-sizing distance cold air must travel (from 50 feet down to 5 feet), software that gives you real-time data on
your infrastructure so that you prevents hot exhaust air from mixing with cool air in the exactly where to cool and what to power.
pay only for what you need. room, and allows more targeted precision cooling.

Introducing the Efficient Enterprise™: $

150,000 THERMAL GUARANTEE AGAINST HARDWARE DAMAGE TO YOUR SYSTEM

more power, more control, more profits WORLD’S ONLY THERMAL GUARANTEE
The Efficient Enterprise™ cooling is so predictable,
we guarantee it. Implement an InfraStruXure®
Can your legacy system say the same? solution with hot air containment and close-
coupled cooling and be eligible for our $150,000
Thermal Guarantee™ - the industry’s only heat
Legacy systems work fine for brute-force cooling the entire room, but
defense policy.
skyrocketing energy costs make them fiscally irresponsible and their
fundamentally oversized design makes them incapable of meeting today’s
high-density challenges. Even worse, power and cooling waste may actually CONTAIN THE HEAT
prevent you from purchasing much-needed new IT equipment. Simple problem, Ensure cooling efficiency by containing the heat
simple solution. Cut your power and cooling costs and use the savings to buy and eliminating expensive temperature cross-
the IT equipment you need. contamination. Our Hot Aisle Containment System
reduces operational expenses by as much as 50%
Gartner Research predicts that by 2008, 50% of today’s data centers will have
over legacy approaches.
insufficient power and cooling capacity to meet the demands of high-density
equipment. Power and/or cooling issues are now the single largest problem
facing data center managers.

There’s only so much power and money to go around

Your service panel limits the amount of power available. Your budget limits the
amount of money. You have to stretch every bit of both as far as you can.
What you need is the APC Efficient Enterprise.™
The APC solution offers modular scalability so that you pay only for what you
use; capacity management so that you know where to put your next server;
and dedicated in-row and heat-containment systems that improve cooling and
thermal predictability. An Efficient Enterprise earns you money through the pre-
planned elimination of waste. For example, simply by switching from room- to
row-oriented cooling, you will save, on average, 31% of your electrical costs.

Our system reimburses you

Whether you’re building a new data center or analyzing the efficiency of existing
systems, your first step is knowing where you stand. Take the online Enterprise
Efficiency Audit to see how you can reap the benefits of a smart, integrated,
efficient system: more power, more control, more profits.

How efficient is your enterprise system?

See exactly where you stand—take our online Enterprise Efficiency Audit today!
Visit www.apc.com/promo Key Code b265w • Call 888.289.APCC x9451• Fax 401.788.2797
©2008 American Power Conversion Corporation and MGE UPS Systems, Inc. All rights reserved. All APC trademarks are property of APC-MGE.
e-mail: esupport@apc.com • 132 Fairgrounds Road, West Kingston, RI 02892 USA EE2D7EF-US

TechTarget_ebook_0308_b265w.indd 1 2/18/2008 11:48:25 AM

chapter 5 A step-by-step guide to greening the data center

STEP 2: Measuring Data

Center Energy Consumption
WHAT KINDS OF metrics do you need to your data center actually goes to IT
manage data center energy consump- equipment (i.e., servers, storage and
tion? You don’t need a formula with an network gear) and how much is sucked
endless number of variables. Experts up by air conditioners and lost in AC/DC
advise data center managers to pick a (alternating current/direct current) “Pick something
simple measurement, make a ratio and conversions for power equipment.
improve on it. This measurement is expressed as a —any metric—
“Pick something—any metric—and get ratio and is most commonly known as and get started.”
started,” advised Pitt Turner, the princi- power usage effectiveness, or PUE. —PITT TURNER,
PRESIDENT, COMPUTERSITE
pal, senior project leader and president PUE is determined by measuring ENGINEERING INC.
of Santa Fe, N.M.-based ComputerSite the power “taken in” to the data center
Engineering Inc. “Even if you pick the (measured at the utility electric meter)
wrong metric, as long as you improve it, and dividing by the power “going out,”
you can drive tremendous change.” which is used to run IT equipment for
There are a lot of potential variables to computing. The equation looks like this:
measure, which is where many data cen-
ter managers get bogged down. But you PUE = Total facility power
don’t need to swallow the ocean in your
first attempt to gauge data center effi- IT equipment power
ciency. Instead, take a simple approach
and measure how much of the power in The measure has been accepted by

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 9
chapter 5 A step-by-step guide to greening the data center

vendor consortiums like the Beaverton, use facility or office building, measure
Ore.-based Green Grid as well as the only at the meter that powers your data
Environmental Protection Agency (EPA) center facility. If power to your data
and ASHRAE in Atlanta to measure data center is not measured on a separate
center efficiency. utility meter, estimate the amount of
The first step in determining PUE is power being consumed by the non–data If your data
to measure at or near the facility’s utility center portion of the building and center is in a
meter. If your data center is in a mixed- remove it from your calculation. mixed-use
facility, measure
only at the meter
POWER USAGE EFFECTIVENESS that powers your
data center
TOTAL FACILITY IT EQUIPMENT
facility.
POWER POWER
• Servers
Power
UTILITY • Storage
POWER IN • Switchgear POWER IN
COMPANY • Telco equipment
• UPS
• Battery backup
Cooling
• Chillers
• CRACs

PUE = Total facility power ÷ IT equipment power

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 10
chapter 5 A step-by-step guide to greening the data center

The next step is to calculate the IT simple and straightforward.” Whichever

equipment load, which should be meas- means users select, Turner said, it’s
ured after power conversion, switching important to measure the same way
and conditioning is completed. consistently.
According to the Green Grid, the most According to the Uptime Institute, a
likely place to measure would be at the typical data center has an average PUE
PUE is an impor-
output of the computer room power dis- of 2.5. This means that for every 2.5 tant first step to
tribution units (PDUs). This measure- watts that goes “in” at the utility meter, managing data
ment should represent the total power only 1 watt is delivered out to the IT load. center energy
delivered to the compute equipment Uptime estimates that most facilities consumption
racks in a data center. could achieve 1.6 PUE using the most
While PDU is an accurate measure- efficient equipment and best practices
holistically, and
ment of IT load, Turner said uninterrupt- and that a large data center (of 30,000 experts agree
ible power supply (UPS) output may be square feet or more) could improve that it is the key
a better option. “It’s really important to power usage effectiveness from 2.5 to building block
have a consistent and predictable way to 2.0 without building a new data center for any energy-
collect data to approximate the IT ener- and could actually increase cooling relia-
gy consumption,” he said. “If you do it at bility. This project would save nearly a
efficiency project.
the UPS output, there’s only one place to million dollars annually in high-utility-
measure. If you do it at the pure location, rate regions.
which is the PDU output, you would like- While more granular measurement
ly have to measure 600 to 700 locations points will be necessary in the future,
to get the same information,” he said. understanding PUE is an important first
The tradeoff is essentially one between step to managing data center energy
accuracy and ease of use. “One [mea- consumption holistically, and experts
surement] is more pure, but much more agree that it is the key building block
difficult and risk oriented. The other is for any energy-efficiency project. 

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 11
chapter 5 A step-by-step guide to greening the data center

STEP 3: Asset Management

and Purchasing Decisions
NOW THAT YOU have your CEO’s blessing APPLICATION CONSOLIDATION. While
for a data center efficiency project and these tag lines suggest starting with
you’ve measured PUE, it’s time to get server components, I recommend going
down to the business of reducing data deeper and beginning with applications
center energy consumption. There are themselves, which is an idea from HP. In Savings at the
hundreds of potential starting points, 2006 the computing giant took some of most basic IT
but in order to deal with the problem its own medicine and conducted a giant level adds up
holistically, you need to have a strategy. data center consolidation project and to exponential
consolidated applications as well as
ROOT COMPONENTS. The logical start- infrastructure.
savings on
ing point is with root components An enterprise data center often sup- big-ticket facility
(i.e., processors), then you can progress ports and delivers thousands of applica- requirements.
through the chain to supporting facility tions, some of which are multiple ver-
infrastructure. HP uses the tagline sions of the same software, some of
“Chips to chillers” to describe this which are different products essentially
process; Emerson Network Power uses doing the same thing, and an even
the phrase “Creating savings that cas- greater number of which are forgotten
cades across systems.” The idea is that or legacy programs just taking up space.
savings at the most basic IT level adds But application consolidation has
up to exponential savings on big-ticket received little attention in data center
facility requirements. consolidation projects. Russ Daniels,

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 12
chapter 5 A step-by-step guide to greening the data center

the vice president and chief technology leave them running where they are.
officer of HP Software and HP Adaptive Decommissioned servers get lost in
Enterprise, stressed the importance of the shuffle.
this step. So how do you get rid of them?
Daniels said that HP asked itself a According to Jack Pouchet, the director
series of questions to reduce the data of energy initiatives at Emerson Network
Application
center application portfolio in its data Power, eliminating such hardware consolidation
centers: “Do we have a business case for requires brute force. He recommends saves costs
this application? Can we consolidate it that IT managers take these steps: in licensing,
with others? Do we have 10 applications management
all doing the same thing? Which one will 1. Round up all legacy servers in your
we standardize on?” data center and determine what they
resources and
Application consolidation saves costs are supposed to do. hardware
in licensing, management resources and support.
hardware support, according to Daniels. 2. Take servers with unknown pur-
But it also lessens the load on servers, poses through all the lines of business
enabling data center managers to scale and ask, “Whose servers are these?”
back computing resources.
3. If you end up with 30 orphans,
ELIMINATING HARDWARE. The next step allow 90 days for them to be claimed,
is to turn off unnecessary servers. Audit then pull the plug. If no one screams,
your hardware, decommission nonfunc- consider them unused.
tional machines and get them off the
grid. Too often, companies pull servers 4. If someone screams, he needed
out of production only to put them an application, not the server. You’ve
somewhere else (in test-and-develop- shut off that five-year-old boat anchor.
ment sandboxes, for example) or, worse, Port the application to a more energy-

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 13
chapter 5 A step-by-step guide to greening the data center

efficient server and speed the process.

And indeed, said Pouchet, “the Active Power Myths
process isn’t pretty, but people do it FOR NUMEROUS REASONS, users have been wary of active power
every day.”
tools, but Cassatt Corp. has set out to disprove some of the
SERVER VIRTUALIZATION. Another way to
myths surrounding active power management.
reduce the number of servers in an envi-
ronment is to use server virtualization MYTH 1:
from companies like VMware Inc., Some believe that turning servers on and off compromises
XenSource (now Citrix Systems Inc.) hardware reliability. But Cassatt cites the Web site of the
and SWsoft (to be renamed Parallels), Energy Star Program at the Environmental Protection
Microsoft (eventually) and others. We Agency (EPA) as saying, “Modern computers are designed
covered this topic in Chapter 2 (“Energy- to handle 40,000 on-off cycles before failure, and you’re not
efficient server technologies”). And likely to approach that number during the average comput-
according to survey data, most of you er’s five- to seven-year life span. In fact, IBM and Hewlett-
have experimented with or run virtual- Packard encourage their employees to turn off idle comput-
ization in production already, but the fol- ers, and some studies indicate it would require on-off cycling
lowing information can provide a sense every five minutes to harm a hard drive.”
of what’s achievable.
Five years ago, 80% of your servers
MYTH 2:
ran at 5% to 15% utilization. Now, with
VMware, you can collapse 5 to 10 oper- It takes more energy to turn off a server for a few hours
ating systems onto a single machine, and then back on rather than to just leave it running.
cluster them for failover protection and Cassatt again cites the EPA Energy Star program: The small
use fewer servers. surge of power created when devices such as computers are
California utilities now pay customers turned on is vastly smaller than the energy used by running
50% of the cost of a virtualization proj- a device when it is not needed.

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 14
chapter 5 A step-by-step guide to greening the data center

ect (with a maximum contribution of at Cassatt, cites the recommendations

$4 million). With VMware virtualization of such organizations as Gartner Inc.,
technology, one user in the pilot pro- the EPA and the Uptime Institute on
gram reduced the number of servers active power management. “A lot of ven-
from 1,000 to 270. Another user virtual- dors are pursing passive power manage-
ized 260 physical servers onto 11 ment: more efficient processors and
“While [more
VMware hosts. VMware and utility com- power supplies,” Fry said. “While those efficient proces-
panies have estimated that the direct efficiency gains can have a huge impact, sors and power
annual energy savings for each server active power management is the other supplies] can
removed is between $300 and $600. half. Users can find idle servers and use have a huge
(For more information on implement- policies to gracefully shut them down.”
ing server virtualization, see Search In addition to tools from Cassatt,
impact, active
DataCenter.com’s sister sites VMware and Citrix Systems, server ven- power manage-
SearchServerVirtualization.com dors themselves offer homegrown tools ment is the
and SearchVMware.com.) to monitor, reduce and cap server power other half.”
use. While these tools and settings work —JAY FRY,
ACTIVE POWER MANAGEMENT. After you only among each manufacturer’s equip- VP OF MARKETING,
CASSATT CORP.
remove unused servers, how do you deal ment, they are often included in the
with the necessary servers that often sit price of the hardware, such as with
idle? HP’s Brad Kirby points users to the IBM’s PowerExecutive feature. HP’s
PowerSmart feature of Citrix Presenta- management tool is called HP Power
tion Server, a program whose application Regulator, and it ships with its ProLiant
delivery system enables users to identify machines. These tools often work in
and power down underutilized servers. concert with the processor-scaling
Another tool in this vein is Cassatt capabilities built into Intel- and
Corp’s Active Response. Advanced Micro Devices Inc. (AMD)-
Jay Fry, the vice president of marketing based machines as well as with the

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 15
chapter 5 A step-by-step guide to greening the data center

operating system. processor. “A lot of applications may not

Many server vendors give users the be multithreaded enough to run on all
option to turn on power-saving features. the processor cores at once,” Noer said.
In the early rollouts of such capabilities, “If you have a dual-socket server with
many users left the power-saving fea- two quad-core processors, you may only
tures disabled, but now most vendors have six cores running at a particular
IT executives
recommend using the features. “On the time. The processor would automatically have operated
Intel Xeon, the Enhanced SpeedStep shut down the cores that are not being under the
Technology [which increases or decreas- used.” misconception
es clock speed based on usage] doesn’t that it’s OK to buy
need a lot of application support,” said SPECIFYING EFFICIENT SERVERS. Over
Geoffrey Noer, the senior director of the past several years, IT executives
cheap hardware.
product management at server vendor have operated under the misconception
Rackable Systems Inc. “It’s very safe and that it’s OK to buy cheap hardware.
easy for customers to turn on.” Meanwhile, facilities executives face
AMD’s new quad-core Dual Dynamic double-digit growth rates of energy-
Power Management technology can guzzling, poorly utilized servers showing
power down individual cores on a up on raised floors. But as data center

25% PERSUADING
THE C SUITE
The percentage of the most

MEASURING
CONSUMPTION
energy-efficient servers
that will receive the
Energy Star label.

MANAGING
ASSETS
OPTIMIZING
INFRASTRUCTURE 16
chapter 5 A step-by-step guide to greening the data center

capacity planners throughout the U.S. high-efficiency power supply.

have hit a wall, the paradigm has begun
to shift. 2. Servers must implement power-
When it comes to buying servers, per- saving features at the OS-firmware level.
formance is still king, followed by price.
But thanks to new specifications that 3. Most important, the EPA will use
The upcoming
allow users to quantitatively measure the new Standard Performance Evalua- Energy Star label
performance per watt, energy efficiency tion Corp. (SPEC) benchmark to calcu- is scheduled to
is gaining ground in the conversation. late the amount of energy that various appear on servers
The EPA will soon release an Energy servers require to perform a set unit of in 2008.
Star label for servers just like the ones work.
it has created for desktop computers,
clothes dryers and ceiling fans. Server The first workload designated for the
energy consumption and performance new SPEC power metric, SPECpower_
would be measured and tested by third ssj2008, will be server-side Java, and it
parties, and the top 25% of energy- is available on the SPEC Web site. In
efficient models would garner the Ener- December 2007, HP, Fujitsu and Dell
gy Star label. The determination would were some of the first companies tested
enable data center managers to weigh with the new benchmark.
energy efficiency alongside other factors These Energy Star servers may com-
in purchasing decisions. mand a slight premium over standard
The upcoming Energy Star label is servers, but with the three-year cost
scheduled to appear on servers in 2008, of electricity expected to exceed the
and the EPA uses three criteria to deter- purchase cost of most servers by 2009
mine energy efficiency for 1U servers: (at a conservative estimate of 7 cents
per kilowatt-hour), up-front costs have
1. Energy Star servers must use a become less of an issue. 

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 17
 Don’t waste another watt.
Add years to the life of your data center by running more efficiently today.
When you manage a whole building, there’s a lot to worry about: building security,
lighting, cooling, HVAC. And all these moving parts need power - especially the
data center. So it’s no surprise that the data center is consuming more and more
of your building’s utility power, which limits your company’s ability not only to grow
IT, but also to supply power to other parts of the building. Is there a way to combat
today’s power-hungry data center? Get more efficient with the APC Efficient
Enterprise™ - yours in two steps for immediate, measurable results.

Step 1: It’s time for a cool change.

Why cool an entire room when only certain racks are running hot? The oversized
system’s “brute-force” whole-room cooling is inefficient by design and a waste,
not only of money, but of valuable power and cooling capacity as well. And as you The Efficient Enterprise™ makes cooling predictable and reduces
well know, this wasted power and cooling could go to good use in other parts of operational expenses by...

your building. Simple problem, simple solution: The APC Efficient Enterprise. In the Employing close-coupled cooling. Our innovative InRow™
architecture allows a more efficient, targeted cooling by shorten-
Efficient Enterprise model, your cooling moves from general “room level” cooling ing the distance between heat generation and heat removal.
to specific, targeted cooling that’s right in the row. Simply by switching from room- Containing the heat. Our Hot Aisle Containment System
to row-oriented cooling, you will, on average, reduce your electrical consumption reduces hot spots by preventing hot exhaust air from mixing with
cool air in the room.
by up to 31%. Plus, modular in-row cooling gets cool air where it’s needed much
more quickly. Managing capacity. Intelligent, integrated capacity
management software gives you real-time data on your power
and cooling demands.
Step 2: Harness that stranded capacity
We know how hard it is to determine where to place that next server. Let the Right-sized and efficient components.
Benefit from power efficiency by right-sizing
tools of the Efficient Enterprise find your “stranded capacity” -- the power and your infrastructure so that you pay only for
cooling you know you have, but just can’t get to. APC change and capacity what you need. Reap efficiency benefits at
the component level or aggregate benefits
management software gives you visibility across your system, taking the at the system level. The MGE Galaxy 5000
guesswork out of server placement and controlling IT growth. You’ll find and use offers ultra-high availability for lower power
ranges, has a compact footprint and is easy
your available power instead of wondering where it is. And by running closer to and quick to connect. Its clean input current
100% capacity, your higher efficiency will automatically save power. means that the mains supply does not
need to be oversized, which can result in
Use your power wisely considerable savings.

With the Efficient Enterprise, you’ll effectively eliminate thermal and power-driven
incidents, and be more agile in meeting IT demands. Better yet, you can avoid
(or at least postpone) building a new data center because your facility will take up
minimal space. In the end, it boils down to this: you need to be as smart as
possible with your power. What you need is the APC Efficient Enterprise.

How efficient is your enterprise system? See exactly where you stand
— take our online Enterprise Efficiency Audit today!
Visit www.apc.com/promo Key Code b266w • Call 888.289.APCC x9452 • Fax 401.788.2797
©2008 American Power Conversion Corporation and MGE UPS Systems, Inc. All rights reserved. All APC trademarks are property of APC-MGE.
e-mail: esupport@apc.com • 132 Fairgrounds Road, West Kingston, RI 02892 USA 998-0753

TechTarget_ebook_0308_b266w.indd 1 2/18/2008 11:28:59 AM

chapter 5 A step-by-step guide to greening the data center

STEP 4: Optimizing
Data Denter Infrastructure
IN CHAPTER 3 (“Data center infrastructure the southeastern U.S.
efficiency”), we explored how to priori-
tize data center facility improvements. BROWN’S DATA CENTER GOES GREEN. UPS’
In this chapter, we use a case study to Windward data center bucks the con-
illustrate how the process works. ventional wisdom. Old data center facili-
Let’s face it: A lot of green data center ties are supposed to be inefficient, and UPS’ Windward
case studies are pretty worthless. Ven- outdated mechanical systems are prima-
dors and customers pat one another on rily to blame. Even worse, considering
data center bucks
the back for buying green products and the amount of redundancy designed into the conventional
offer vague promises to save energy in the facility to prevent downtime, an wisdom.
data centers over a period of time. Uptime Institute Tier 4-rated data center
But the folks at the United Parcel would have to be a real energy hog.
Service of America Inc.’s Alpharetta, Ga., But somehow the 12-year-old, Tier 4
site are about to save you a lot of money facility in Alpharetta scores a PUE (or, in
on your data center air-conditioning the Uptime Institute’s parlance, SI-EER)
bill today. Joe Parrino, data center as low as 1.9. This ratio represents the
manager at UPS’ Windward data center measure of the power going into the
also explains his organization’s load- facility at the utility meter divided by
shedding process and proves that using the distribution unit or uninterruptible
outside air to cool a data center can power supply.
work—even in the hot temperatures of In the case of the Windward data cen-

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 19
chapter 5 A step-by-step guide to greening the data center

ter, PUE was measured at the output in the floor, but you’ve probably still
of the uninterruptible power supply; wasted cold air in a place you wouldn’t
measuring the output of the PDU was expect: the perforated top of power
too difficult. (For a more detailed discus- distribution units.
sion of the differences in measuring at Parrino’s staff learned this by chance.
the power distribution unit versus at The team noticed the perforated roof on
the uninterruptible power supply, listen
to SearchDataCenter.com’s podcast
“Where to measure IT vs. infrastructure
power use: PDU or UPS?” with Pitt
Turner.)
Windward Data Center Specs
According to the Uptime Institute, the • United Parcel Service of America Inc. has two data centers:
average ratio is 2.5. This means that for the Windward facility in Alpharetta, Ga., and the other in
every 2.5 watts going “in” at the utility Mahwah, N.J.
meter, only 1 watt is delivered out to the
IT load. In this regard, United Parcel • Together, the two facilities house 15 mainframes,
Service’s Windward data center is way 2.9 petabytes of storage and nearly 3,000 servers.
ahead of the curve. But how did the
• These data centers handle the logistics for the eighth largest
company do it?
airline in the world, plus nearly 100,000 ground vehicles.
CUTTING OUT THE CRAH UNITS. Forced-air • The Windward facility was completed in 1995 and has
cooling is one of the least efficient sys- 50,400 square feet of raised floor.
tems in data center infrastructure, and
wasting cold air is the most common • Windward achieved the Uptime Institute’s Tier 4 rating for
mistake in data center management. availability: It has multiple active power and cooling distribu-
You can set up hot aisle/cold aisle, tion paths, concurrent maintainability and System + System,
install blanking panels and seal gaps N+1 redundant components.

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 20
chapter 5 A step-by-step guide to greening the data center

a PDU as it sat in a hallway waiting for Fahrenheit operating range. “We didn’t
installation. They took airflow measure- even come close to the shutdown tem-
ments on several installed units using perature,” Parrino said.
a velometer and calculated the cubic- The next step was to seal the top of
feet-per-minute (CFM) loss (i.e., the PDUs with Lexan covers. Parrino hired
velocity of the air multiplied by the a contractor to install covers on all the
“After we
square footage of the opening). United units. The covers have a 3-inch opening installed the
Parcel Service determined that the units to ensure that the transformers get air- covers, we looked
lost 2,000 CFM per PDU. flow but also block 90% of undesirable at the under-floor
“What heats up inside a PDU that bypass airflow. Following the installation static pressure,
would require 2,000 CFM of cooling?” of the Lexan covers, average transformer
Parrino wondered. The only component temperature increase was about 1
and we were
possibility was transformers, which have degree to 2 degrees Fahrenheit. amazed.”
a high temperature tolerance. “After we installed the covers, we —JOE PARRINO,
DATA CENTER MANAGER,
So Parrino conducted an experiment. looked at the under-floor static pressure, UNITED PARCEL SERVICE
He ran a PDU with a solid Lexan cover at and we were amazed at what we got OF AMERICA INC.
full load (i.e., 180 kW) using a load bank back,” Parrino said. The data center had
for one hour in an outside location on 62 PDUs that were wasting 124,000
an 85-degree Fahrenheit day. Measure- CFM of cold air. With the covers
ments of the transformer temperatures installed, Parrino estimated that he
were taken with an infrared camera. The could shut off six computer room air
transformer temperatures increased 20 handlers (CRAH) units based on meas-
degrees from the nominal 115 degrees ured airflow of 19,000 CFM per CRAH
on the air-conditioned raised-floor space unit. In reality, he shut off 10.
to about 135 degrees in a non-air-condi- The cost of covering PDUs was about
tioned location. This was well within the $6,000, and United Parcel Service esti-
manufacturer’s stated 300-plus degrees mated that payback would take about

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 21
chapter 5 A step-by-step guide to greening the data center

4.3 months. Instead the project paid for air temperatures to cool its chilled water.
itself in a month and a half. Also known as water-side economizing,
Parrino plans to implement variable the practice saves energy by allowing
frequency drives on some of Windward’s data centers to turn off chillers. Green
CRAH units, and his team is experiment- data center experts have given this
ing with variable-air-volume floor grates approach a lot of attention lately.
Water-side
controlled by intake temperatures of the Unfortunately, most people don’t take economizing
racks. “This will slow the consumption of advantage of free cooling because they saves energy
CRAH fan energy even further by deliv- aren’t in a region that stays cold long by allowing data
ering the CFM that’s needed for each enough for the system to pay for itself centers to turn
rack instead of delivering based on the or because they lack the automation to
worst-case IT load,” Parrino said. manage the process of going on and off
off chillers.
the plate-and-frame heat exchanger. But
FREE DATA CENTER COOLING. The Wind- UPS has solved both of those problems.
ward data center has two 1,000-ton For starters, Parrino’s staff raised the
centrifugal chillers and two 800-ton temperature of the chilled-water loops
absorption chillers. The data center also from the designed temperature of
has a 650,000-gallon thermal storage 45 degrees Fahrenheit. It now modulates
tank with redundant water sources between 52 degrees and 58 degrees
(a well and city water for backup). The Fahrenheit. The lower temperatures
thermal storage tank was designed to are needed during high-humidity days
provide about 20 hours of emergency (i.e., 100% humidity when it rains) to
cooling, but Parrino’s team also uses it maintain the interior relative humidity
for energy-cost management. between the nominal 40% to 55%.
In 2000, United Parcel Service During the winter months when the
installed a plate-and-frame heat outside air is drier, Windward can use
exchanger to take advantage of outside higher temperatures.

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 22
chapter 5 A step-by-step guide to greening the data center

Further, getting on and off the plate- on the plate-and-frame heat exchanger
and-frame heat exchanger is easier with even further during the seasonal transi-
a thermal storage tank. As the chiller tion periods, the thermal storage tank
shuts down and a condenser water loop provides ride-through during the warm
is lowered, the thermal storage tank pro- afternoons and is then re-charged during
vides uninterruptible cooling to the data cool evenings. The entire process is
United Parcel
processing equipment. automated; no human intervention is Service’s thermal
The Windward data center can do required. storage tank
this because it is a standalone data As of late November 2007, Windward provides ride-
center rather than a mixed-use facility is on plate about 90% of the time and through during
with office space that needs to run cool- will remain that way through the better
er chilled-water temps for latent cooling. part of April. That’s five months of free
the warm after-
(For more on the difference between cooling—in Atlanta, no less. Northern noons and is then
latent and sensible cooling, listen to latitudes should enjoy even longer free re-charged during
SearchDataCenter.com’s podcast cooling periods. cool evenings.
“Servers don’t sweat: Raise your Based on operating 73 days per year,
chilled-water temp.”) the plate-and-frame heat exchanger
Higher chilled-water temperatures project was projected to pay for itself
enable United Parcel Service to extend in two and a half years, saving 4 cents
its use of free cooling dramatically. In per kilowatt-hour. It saved $88,000
2007 the data center used the plate- annually. And the winter of 2000 was
and-frame heat exchanger for the last especially cold, so in the first year of
time on May 18 and switched free cool- operation the project paid for itself.
ing back on for the first time on Oct. 11. According to Parrino, switching on and
During this seasonal transition period, off a plate-and-frame heat exchanger
nighttime temperatures get low enough would be a messy job without a thermal
but the days are warm. To extend time storage tank and solid automation soft-

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 23
chapter 5 A step-by-step guide to greening the data center

ware. He says the Windward building real-time pricing plan with its utility
automation system from Kennesaw, Georgia Power.
Ga.-based Automated Logic Corp. is a The price can range from 4 cents per
story in itself. kilowatt-hour in the morning to 8 cents
In 1995 the system was installed with in the afternoon when there are moder-
the building, and the plan was to bring ate summertime temperatures. Costs for
Running the
all building systems under a single afternoon peak-load times in the month chillers at night
interface. “It was a pretty advanced sys- of August exceeded 30 cents per kilo- is an effective
tem in 1995, [and] even more so today,” watt-hour on days when the outside energy-reduction
Parrino said. “Manufacturers want to temperature surpassed 100 degrees strategy as well,
give you a PC for your UPS system, one Fahrenheit.
for the generator or a chiller. Our system In order to minimize costs, Windward
since outside
interfaces with all of these third-party switches to “plant economy” mode wet-bulb temper-
devices.” during the summer peak-load periods. atures are typi-
The system provides chiller, pumps, Plant-economy mode effectively shuts cally lower than
and cooling tower rotations and man- down the 630 kW chiller plant (includ- they are during
ages the thermal storage tank. It also ing chillers, cooling tower fans, primary
gives Windward visibility into outside air pumps, tower pumps) and cools the
daytime hours.
temperature and humidity conditions to data center using the stored 45-degree-
determine when a data center can use Fahrenheit thermal energy in its
outside cooling. 650,000-gallon thermal storage tank.
At night, when the cost per kilowatt-
HOW PEAK-LOAD SHEDDING GETS DONE. hour is about 4 cents, the facility runs
Automated Logic’s system also helps chillers and re-charges the tank during
United Parcel Service shed its power off-peak hours.
load during high-demand peaks in the Running the chillers at night is an
summer. United Parcel Service is on a effective energy-reduction strategy as

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 24
chapter 5 A step-by-step guide to greening the data center

well, since outside wet-bulb tempera- Center.com reported on a broad survey

tures are typically lower than they are of TechTarget Inc. members. Results
during daytime hours. A lower wet-bulb indicated that in 2008 green computing
temperature allows for more efficient will remain a minor initiative. For the
removal of heat via cooling towers. This moment, many companies have simply
reduces the condenser water tempera- fed the energy-consumption beast by
United Parcel
ture, also reducing “lift” in the chiller and building new data centers to provide Service equates
enabling it to run more efficiently. additional raw power for an increasing energy efficiency
Previously the thermal storage tank number of servers. with increasing
provided six to eight hours of cooling United Parcel Service is a notable the useful life of
capacity on a 95-degree day, with 40% exception. According to Parrino, the
thermal capacity remaining. Raising the company’s founder, James Casey,
the data center.
chilled-water set point has increased embedded in UPS two principles: to
the ton-hours capacity to approximately always be “constructively dissatisfied”
16 hours of cooling, with 50% thermal and to constantly seek opportunities to
capacity remaining. improve efficiency. United Parcel Service
And the tank is never completely has actively gone green in its data center
discharged, so it can provide thermal and views its efforts as having a broader
backup in the event of a chiller problem. impact on the environment.
Finally, it removes stress on Georgia “When you look at electrical costs
Power’s peak power capacity infra- of $100,000 a month in our $47 billion
structure. company, it’s not a lot of money for
one building,” Parrino said. “But going
ECO-AWARENESS AT UNITED PARCEL beyond that, as a good corporate citizen
SERVICE. It’s not clear that companies UPS has learned to manage the con-
are yet in a green mood. In “IT priorities sumption of energy in all aspects of their
in 2008: A truly new year,” SearchData- business. The data center is no excep-

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 25
chapter 5 A step-by-step guide to greening the data center

tion. Energy sources in the Southeast are

plentiful but not necessarily renewable.
As a renewable generating source in
Chapter Summary Steps
Georgia, wind is not good, solar is mar-
ginal, and geothermal and hydro simply
aren’t available. Typical electrical gener-
ating sources are 50% coal and 50%
1
Build the business case for data center energy efficiency.
How much do you spend on energy now? How soon will
you need to build a new data center to keep pace with
natural gas.” company growth? Use this information to get a mandate
United Parcel Service also equates to pursue efficiency from C-level executives.
energy efficiency with increasing the
useful service life of the data center. “All
these data centers around the country
are running out of power and cooing
and are having to expand,” Parrino said.
2
Determine a simple metric such as power usage effective-
ness (PUE) to create a baseline for how much power goes
to servers and how much is lost on cooling and infrastruc-
“Becoming energy efficient is a great ture. Set goals to improve the ratio.
payback when you don’t have to expand
into additional infrastructure.” 
Matt Stansberry is the senior site editor of SearchData
Center.com. Since 2003, Stansberry has reported on the
convergence of IT, facility management and energy issues.
3
Tackle IT inefficiencies by consolidating applications,
eliminating dead-weight servers, implementing active
power-saving features and specifying energy-efficient
Previously, Stansberry was the managing editor of Today’s hardware.
Facility Manager magazine and a staff writer for the U.S.
Green Building Council. He can be reached at

4
mstansberry@techtarget.com.
Address facility fundamentals by implementing hot
aisle/cold aisle, sealing your data center’s floor, raising the
voltage on power distribution units (PDUs) from 120 volts
to 208 volts and, when possible, exploiting free cooling.

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 26
 Resources from our sponsor

ABOUT APC: Computer Rooms are the heart of many small/medium

businesses. Often not originally or specifically designed to provide
the ideal environment for IT equipment, special attention to Net-
work Critical Physical Infrastructure (NCPI) when deploying racks
of servers, switches, and other IT equipment is required to ensure
q White Paper: Implementing availability. APC offers solutions for the Computer Room including
Energy Efficient Data Centers UPS, racks, rack PDUs, air-conditioning/ airflow, environmental
management, and other related accessories to achieve the level of
q White Paper: Making Large availability required.
UPS Systems More Efficient Many challenges have evolved in the computer room environ-
ment over the past few years due to trends such as server consoli-
q White Paper: Electrical Efficiency dation, technology compaction, and the convergence of voice,
Measurement for Data Centers video, and data. Challenges include: getting enough power and
power receptacles to each of the racks to support today’s increasing
q White Paper: Increasing Data Center power densities, effectively cooling all of the IT equipment
Efficiency by Using Improved deployed, organizing the large amount of data and power cables,
High Density Power Distribution providing security, and managing the complete system. APC offers
complete Network Critical Physical Infrastructure solutions for the
q White Paper: A Quantitative Computer Room including UPS, racks, rack PDUs, air-condition-
Comparison of High Efficiency AC vs ing/airflow, environmental management, and other related acces-
DC Power Distribution for Data Centers sories to achieve the level of availability required. 

PERSUADING MEASURING MANAGING OPTIMIZING

THE C SUITE CONSUMPTION ASSETS INFRASTRUCTURE 27