Analysis of performance metrics for data center efficiency – should the Power Utilization Effectiveness PUE still be used as the main indicator? (Part 2)

Part1 of this article has been published in REHVA Journal 2017-01

Previous research posed that PUE (Power Usage Effectiveness) does not always reflect the real energy performance of data centers. This is because PUE does not show performance regarding IT efficiency, water usage, heat recovery, on-site energy generation or carbon impact. Broadening the scope of performance assessment beyond PUE has therefore been proposed by including these subjects. Using a simulation study, this paper shows the potential of finding energy efficiency measures beyond the scope of PUE by using complementary metrics. In this way, a heat reuse potential of 11-15% of the total energy use is found for a 1MW data center in Killarney. It also shows a 4% energy impact reduction for a roof sized PV-system in Sevilla as well as the potential and challenges accompanying the implementation of larger PV-systems. To better evaluate the efficiency of on-site generation the GUE (Grid Usage Effectiveness) metric is introduced. By broadening the scope of data center energy performance assessment, the next step energy efficiency improvement can be taken and the industry can take environmental responsibility by reducing its energy footprint.

Keywords: PUE, Performance metrics, Data Center, Energy Efficiency, Indicators, Simulation, GUE

The data center industry was responsible for between 1.1% and 1.5% of global energy consumption in 2010 (Koomey, 2011) and this value is expected to double by 2020 (Whitney et al., 2014) as the growth of the data center industry is expected to continue following the increasing number of connected devices requiring this infrastructure (Modoff et al., 2014).

Awareness of this trend has led to an effort to improve the sustainability of the data center industry by improving its energy efficiency. Currently, the energy flows within a data center are monitored at different levels to be able to assess both overall and subsystem energy efficiency. Performance metrics are calculated and used as indicators for the efficiency of the systems (Wang et al., 2011).

The main indicator that is being used to assess overall data center energy efficiency is PUE, which shows the ratio between total facility power use and IT equipment power use (Avelar et al., 2012):

Therefore, the optimal value for PUE is 1.0, the maximum value is infinity. PUE was developed give data collection standards ‘to determine the effectiveness of any changes made within a given data center’ (idem, 2012), but is widely being used to compare energy efficiency between data centers. The scope of PUE however is insufficient to accurately reflect the overall energy performance of a data center as it does not cover, among others, IT equipment efficiency, water usage, energy recovery or on-site renewable energy generation (Van de Voort et al., 2017). This paper aims to show the added benefit of using metrics complementary to PUE in data center performance assessment to further decrease the data center industry energy impact.

Research methodology

To find a solution to the problem described above, the following research question has been formulated:

‘How can performance metrics complementary to PUE help to better reflect the real energy performance of a data center?’

High resolution data of the energy flows in a data center is required to evaluate to which extent these additional metrics improve the assessment of the actual energy performance of data centers. Because this information obtained from measurements in data centers is very confidential, a virtual environment has been created. Another benefit is the ability to define different boundary conditions making it possible to simulate various scenarios under controlled conditions. This building energy simulation model is used to analyze in detail the benefit of using complementary metrics beside PUE for data center energy performance assessment. The simulation provides hourly values of the energy flows within the data center for one year. The different energy flows calculated by the simulation can be found in Table 1. From this high-resolution data, all required values for the relevant performance metrics can be calculated.

Table 1. Simulation output parameters.

The analysis of the simulation results is focused on three scenarios which implement different energy efficiency measures. Namely, on-site sustainable energy generation; energy recovery; and geothermal energy harvesting. The calculated values for PUE and other relevant metrics: ERF (Energy Reuse Fraction, Patterson, 2010):, OEM (Onsite Energy Matching, Cao et al., 2013), OEF (Onsite Energy Fraction, Cao et al., 2013) and GUE (Grid Usage Effectiveness, Van de Voort et al., 2017) will be used to assess the benefit of using metrics complementary to PUE.

Simulation setup

For these simulations, an adaptation of the data center simulation model by Van Schie et al. (2015) has been used to represent a 1 MW data center. TRNSYS was used as a modeling tool to create a white-box model which represents this data center. An overview of the model can be found in appendix A[1].

The model has been used to simulate the effect of the different variables described in Table 2 on the energy flows in the data center. The locations were chosen to represent three different climate conditions in Europe. Four different HVAC systems have been modelled, representing a wide spectrum of cooling system efficiency. Two different IT workload profiles are used as input to evaluate the influence of IT load on energy flows. Also, two inlet temperature set points have been used as control strategies. Lastly, three differently sized PV-systems are introduced to evaluate the benefit of on-site renewable generation.

The first PV system is sized to the dimensions of the roof area of a typical 1MW data center (2 000 m² PV). This simulation shows to what extent PV systems can reduce the energy impact of a data center within this realistic boundary. There are no issues with energy matching as the OEM value remains 1 all year. For the second scenario, the PV system size is increased to maximize generation while keeping the average OEM close to 1 (11 100 m² PV). This case shows which part of the total energy demand can be met by a PV-system without causing matching issues. The third PV system is sized to generate the same amount of energy yearly as the total energy consumption of the data center (51 750 m² PV). At this point matching issues occur because peaks in generation greatly exceed demand.

Table 2. Input variables for simulation exercise.

Simulation results

Out of the results found by this simulation a selection of three cases has been made for further analysis, these cases are shown in Table 3. They have been chosen as they represent three important strategies to reduce the energy impact of data centers. These are on-site generation, energy reuse and the use of geothermal energy.

Results of the PUE values for the chosen simulation cases are given in Figure 1. The PUE values are largely dependent on the type of cooling system, this became clear after analysis of the complete simulation results. The figure shows the previously described relationship between PUE and IT load, showing better PUE values for higher IT loads. It also shows the relationship between PUE and IT load, showing better PUE values for higher IT loads. It also shows the relationship between PUE and cooling temperature set point, with a higher cooling set point leading to lower PUE values.

Table 3. Three cases chosen for detailed analysis.

The most interesting results regarding the scope of PUE were found for the Sevilla case where a PV-system has been applied. In the following section this energy efficiency strategy is further analyzed to see whether the metrics put forward provide a better framework for reflecting the real performance of a data center than PUE alone. First the main characteristics of the other two cases are described.

Figure 1. PUE results for the twelve different simulations for the three chosen configurations.

Killarney – Energy reuse

Usable waste energy was defined as exhaust air with temperatures over 30°C. The ERF potential resulting from this is displayed in Figure 2. The potential found lies between 11–15% of total energy consumption for the different scenarios.

In cases where this waste energy can only be reused when there is simultaneous demand the actual reuse value can greatly decrease. Losses will also occur during the energy transport. Because of these reasons, it is impossible to translate the ERF potential in an actual value for ERF as the influence of these factors is unknown. To make the best use of the ERF potential the mentioned issues need to be addressed when designers implement energy reuse strategies. An interesting strategy is coupling energy recovery to an aquifer thermal storage system to avoid the necessity for simultaneous demand.

Figure 2. Energy recovery potential for the data center in Killarney.

Helsinki – Geothermal

For this case PUE reflects the real energy impact very well. As seen in Figure 1, the partial contribution of energy use for cooling to the PUE value has become very small, making this a very efficient design. Because PUE only assesses the use of electric energy, the thermal energy used for cooling the data center in this case does not increase the value for the total facility power, keeping the PUE value low.

Sevilla – PV

As previously stated, PUE doesn’t give insight into the positive contribution to the energy impact of on-site renewable energy generation. The OEF and OEM metrics can be used for assessing the amount and efficiency of on-site renewable generation. Results discussed in this section are for the Sevilla case with 75% average workload and 18°C inlet temperature. Figure 1 shows results from the other simulation setups follow a similar trend for the energy flows making the results discussed in this section also relevant for those cases.

Even though the data center industry is characterized by its high-energy density, Figure 3a shows on-site renewables can have an impact on its energy footprint. If we look at the roof sized PV system 4% of the total energy demand could be met, even for a high average utilization of 75% IT load. This impact will only increase as PV efficiency increases and therefore this benefit should be considered during performance assessment. When looking at the PV system sized for matching peak loads, the energy impact reduction further increases to 20% of the total energy demand. A larger site would be necessary or extra areas near the data center should be outfitted with PV panels. Nearby building or site owners might allow placement of PV panels for this purpose.

When the PV area is further increased problems will arise with energy matching. This is clear when annual energy generation by the PV system is equal to the annual energy demand of the data center. Figure 3a shows that for the simulated workload profile only 41% of the supply is matched by simultaneous demand meaning 59% cannot be used by the data center. Also, the electricity grid must balance this influx of energy, which is causing more and more problems as the adoption of renewable generation increases.

Figure 3b shows the average OEM value is still relatively high, because the value for OEM is 1 when there is no supply, this skews the average figure. Hourly values for OEM should be considered when interpreting results. One-year graphs containing hourly values for OEF and OEM can be found in appendix B[2]

There are ways to improve energy matching for on-site renewable generation. This can be done by matching generation to expected demand, save non-critical workload for periods of high on-site availability or by energy storage. To indicate with a single performance metric the benefit of using on-site renewables and to promote energy matching by showing to which extent the data center operates grid independent, it is proposed to introduce the Grid Usage Effectiveness (GUE) metric.

GUE

The GUE shows the grid dependence of the data center in relation to the IT load, it is defined as:

Figure 4 shows how the GUE is dependent of the OEF and OEM metrics. At first, the GUE value improves as the on-site generation and OEF increase, it is optimal when the OEF and OEM are both 1. At this point the data center operates independent of the grid as its demand is exactly matched by on-site generation. When the on-site generation starts to exceed the facility demand the GUE value increases again as the grid is being burdened with the excess electricity.

This accurately reflects to which extent the grid is being used, be it for supply or demand, and will promote energy balancing. GUE combines information concerning on-site (renewable) generation and energy matching with PUE. Though it is adding complexity, it’s giving a more complete picture of a data center’s energy impact without losing the clarity of the single metric. The average, minimum and maximum PUE and GUE values for the case from the previous section is shown in Figure 5.

When smart grids are introduced it is conceivable that weighing factors dependent on the momentary grid balance are introduced to the metric. In that case, data centers can help balance the grid by using energy from the grid when supply is abundant and they can supply energy to the grid when demand is high, without penalties to their GUE. This will add a further incentive to implement demand response strategies.

Figure 4. Relationship between GUE, OEF & OEM.

Figure 5. GUE vs. PUE for Sevilla case.

Figure 6 shows a simplified representation of the hourly PUE values and the GUE values for the three PV-system sizes. The full graphs can be found in appendix C[3]. The impact of the roof sized PV-system is subtle, whereas the positive impact of the peak size PV-system is very clear. The matching issues related to the demand size PV-system is also clearly illustrated, with the highest peak around noon during summer. This makes it immediately clear where the focus should lie for improvement.

Future data center designers will be able to use the GUE metric to design data centers which efficiently use on-site generation to reduce the impact they have on the electricity grid and the environment. As we are preparing for a future solely reliant on renewable energy this will be hugely important. The efficiency of demand management, supply matching and energy storage strategies can be assessed using this metric.

Discussion & conclusion

Though more situations are conceivable where the scope of PUE is too narrow to thoroughly assess the complete energy performance of a data center, the simulation exercise has provided information to answer the research question by reviewing case studies with energy reuse, geothermal energy use and on-site energy generations with PV panels.

‘How can complementary performance metrics to PUE help to better reflect the real energy performance of a data center?’

For the simulation case using geothermal energy PUE proved to accurately reflect the cooling systems energy impact. With help of the ERF metric, the simulation scenario for Killarney demonstrated a potential benefit for energy reuse of up to 15% of the total energy consumption. This scenario used a chiller as cooling system and the minimum exhaust air temperature for reusable waste heat was set to 30°C.

Using the OEM and OEF metrics, the simulation case for Sevilla showed a reduction of the total energy impact of 4% for a roof sized PV-system in Sevilla, increasing to 20% for a PV-system sized to maximize generation without causing matching issues. When further increasing the PV size energy matching issues arise that need to be mitigated.

Figure 6. Visualization of the hourly values for the PUE and different GUE values for one year.

To quickly assess the effectiveness of onsite energy generation, the GUE metric can be used. It shows the positive impact on-site renewable generation can have on the energy footprint and can also help to understand the challenges involved in energy matching. Evaluation of resulting GUE values can help find better strategies to tackle energy matching challenges. Suggested strategies for energy matching can involve demand management, supply matching and energy storage. Further research can provide information on effective use of these strategies to further reduce the energy impact of the data center industry.

In short, it’s necessary to broaden the scope of data center performance assessment beyond PUE to meet future challenges the upcoming energy transition will present. The metrics used in this paper, among others, are part of the tools required to meet these challenges. The next step will be to use this expanded framework of energy performance metrics for the creation and evaluation of a new generation of state-of-the-art energy efficient data centers.

Literature

Avelar, V., Azevedo, D., French, A., (2012), PUE™: A Comprehensive Examination of the Metric, The Green Grid.

Koomey, J.G., (2011), Growth in data center electricity use 2005 to 2010, Analytical Press, August 1, 2011.

Modoff, B., Bhagavath, V., Clifton, K., (2014), Not quite the Jetsons yet, but places to look Internet of Things (IOT) is a big idea but promise and reality are a distance apart, F.I.T.T. for investors, Deutsche Bank Securities.

Van de Voort, T., Zavrel, V., Torrens, J.I., Hensen, J.L.M, (2017), Analysis of performance metrics for data center efficiency: should PUE still be used as the main indicator, TBP, accepted by: The REHVA European HVAC Journal.

Van Schie, F.T., Zavřel, V., Torrens, J.I., Hundertmark, T.F.W., Hensen, J.L.M., (2015), Optimizing the total energy consumption and CO2 emissions by distributing computational workload, among worldwide dispersed data centers, in Proc. Building Simulation '15. Hyderabad: IBPSA.

Wang, L., Khan, S.U., (2011), Review of performance metrics for green data centers: a taxonomy study, J Supercomput, Volume 63, pp. 639-656.

Whitney, J., Delforge, P., (2014), America’s Data Centers Are Wasting Huge Amounts of Energy, Critical Action Needed to Save Billions of Dollars and Kilowatts, NRDC, August 2014.

APPENDIX A - Simulation model

1. Overview

2. Standard chiller

3. Chiller with free cooling

4. Indirect economizer

5. Seawater cooling

APPENDIX C – one year graphs of hourly values for OEM and OEF

1. Yearly profile of the OEF resulting from a roof size PV system

2. Yearly profile of the OEF resulting from a peak size PV system

3. Yearly profile of the OEM from a demand matching size PV system

4. Yearly profile of the OEM resulting from a peak matching size PV system

5. Yearly profile of the OEM resulting from a demand matching size PV system

APPENDIX D – one year graphs of hourly values for GUE and PUE

1. Yearly profile of the PUE values for the Sevilla case with 75% workload and 18°C inlet temperature

2. Yearly profile of the GUE values for the Sevilla case with a roof size PV-system

3. Yearly profile of the GUE values for the Sevilla case with a peak size PV-system

4. Yearly profile of the GUE values for the Sevilla case with a demand size PV-system