Memory Optimization

Module 9

© 2014 VMware Inc. All rights reserved

You Are Here

Course Introduction Storage Optimization

VMware Management Resources

CPU Optimization
Performance in a Virtualized
Environment
Memory Optimization
Network Scalability
Virtual Machine and Cluster
Network Optimization Optimization

Storage Scalability Host and Management Scalability

VMware vSphere: Optimize and Scale 9-2

© 2014 VMware Inc. All rights reserved

Importance

While VMware vSphere® employs various mechanisms to efficiently

allocate memory, you might still encounter a situation where virtual
machines are allocated insufficient physical memory.
You should know how to monitor memory usage of both the host and
virtual machines. You should also know how to troubleshoot
common memory performance problems, such as those involving an
excessive demand for memory.

VMware vSphere: Optimize and Scale 9-3

© 2014 VMware Inc. All rights reserved

Module Lessons

Lesson 1: Memory Virtualization Concepts

Lesson 2: Monitoring Memory Activity
Lesson 3: Troubleshooting Memory Performance Problems

VMware vSphere: Optimize and Scale 9-4

© 2014 VMware Inc. All rights reserved

 Lesson 1:
Memory Virtualization Concepts

VMware vSphere: Optimize and Scale 9-5

© 2014 VMware Inc. All rights reserved

Learner Objectives

By the end of this lesson, you should be able to meet the following
objectives:
 Describe how memory is used in a virtualized environment
 Explain each memory-reclamation technique
 Explain how memory overcommitment impacts performance

VMware vSphere: Optimize and Scale 9-6

© 2014 VMware Inc. All rights reserved

Virtual Memory Overview

virtual machine

guest virtual memory guest

applications

guest physical memory guest

operating system

host physical memory hypervisor

VMware vSphere: Optimize and Scale 9-7

© 2014 VMware Inc. All rights reserved

Application and Guest Operating System Memory Management

virtual machine

application

allocated list free list

guest
OS

VMware vSphere: Optimize and Scale 9-8

© 2014 VMware Inc. All rights reserved

Virtual Machine Memory Management

Virtual machine starts with no host application

physical memory allocated to it.
Host physical memory is allocated on
demand.
guest OS
 Guest operating system will not
explicitly allocate memory.
 Memory is allocated on the virtual
machine’s first access to memory (read
or write). hypervisor

VMware vSphere: Optimize and Scale 9-9

© 2014 VMware Inc. All rights reserved

Memory Reclamation

Guest physical memory is not freed in

the typical sense:
 Memory is moved to the free list. application
The hypervisor is not aware of when guest OS
the guest frees the memory: free list

 It has no access to the guest’s free list. guest OS

 The virtual machine can accrue lots of
host physical memory.
The hypervisor cannot directly reclaim
the host physical memory that is in the hypervisor
guest OS free list.

VMware vSphere: Optimize and Scale 9-10

© 2014 VMware Inc. All rights reserved

Guest Operating System Memory Terminology

memory size
total amount of memory
presented to a guest OS

allocated memory free memory

memory assigned to memory not assigned
applications and the OS

active memory idle memory

allocated memory recently allocated memory not
accessed or used by recently accessed or
applications and the OS used

VMware vSphere: Optimize and Scale 9-11

© 2014 VMware Inc. All rights reserved

Virtual Machine Memory-Reclamation Techniques

The hypervisor relies on the following memory-reclamation

techniques to free host physical memory:
 Transparent page sharing (enabled by default)
 Ballooning
 Memory compression
 Host-level (hypervisor) swapping

VMware vSphere: Optimize and Scale 9-12

© 2014 VMware Inc. All rights reserved

Transparent Page Sharing

Content-based page sharing reclaims memory with minimal

overhead.

VMware vSphere: Optimize and Scale 9-13

© 2014 VMware Inc. All rights reserved

Memory Ballooning in the Guest Operating System

The memory balloon driver (vmmemctl) collaborates with the virtual

machine to reclaim pages that are considered least valuable by the
guest operating system.

VMware vSphere: Optimize and Scale 9-14

© 2014 VMware Inc. All rights reserved

Reclaiming Memory with Ballooning

Ballooning preferentially selects free or idle virtual machine memory.

 This is because the guest operating system allocates from free
memory.
But if asked to reclaim too much, ballooning will eventually start
reclaiming active memory.
free memory idle memory active memory

guest
OS

hypervisor

VMware vSphere: Optimize and Scale 9-15

© 2014 VMware Inc. All rights reserved

Memory Compression

The VMware® ESXi™ host stores pages in a compression cache

rather than swap them out.
compression cache

A C
A B VM A B C VM D
B D

swap swap
A file B file

without compression cache with compression cache

VMware vSphere: Optimize and Scale 9-16

© 2014 VMware Inc. All rights reserved

Host Cache

Swap to host cache allows users to configure a special swap cache

on solid-state drive (SSD) storage. page table
0

1
1
0
SSD
1
0
0
1

main memory

VMware vSphere: Optimize and Scale 9-17

© 2014 VMware Inc. All rights reserved

Reclaiming Memory with Host Swapping

Host-level swapping randomly selects guest physical memory to

reclaim, which might be a virtual machine’s active memory.

free memory idle memory active memory

guest
OS

hypervisor

VMware vSphere: Optimize and Scale 9-18

© 2014 VMware Inc. All rights reserved

Reason for Reclaiming Memory

The hypervisor reclaims memory to support ESXi memory

overcommitment.
A host’s memory is overcommitted when the total amount of guest
physical memory is greater than the amount of host physical
memory.

VM1 (2GB) VM2 (2GB) VM3 (2GB)

VM
memory

hypervisor
(4GB)

VMware vSphere: Optimize and Scale 9-19

© 2014 VMware Inc. All rights reserved

Sliding Scale Mem.MinFreePct

Mem.MinFreePct is the amount of memory the VMkernel should

keep free.
 The VMkernel uses a sliding scale to determine the Mem.MinFreePct
threshold based on the amount of memory installed in the host.

Memory Installed Free State

Threshold Range

0 through 4GB (first 4GB) 6%

4 through 12GB (next 8GB) 4%

12 through 28GB (next 16GB) 2%

Remaining memory 1%

VMware vSphere: Optimize and Scale 9-20

© 2014 VMware Inc. All rights reserved

Criteria for Reclaiming Host Memory

Host physical memory is reclaimed based on four host free-memory

states, reflected by four thresholds.

If Host Free Memory Drops Use the Following

Toward the Following Reclamation Technique
Threshold
High None

Soft Ballooning

Hard Compression and swapping

Low All options are used as

necessary.

VMware vSphere: Optimize and Scale 9-21

© 2014 VMware Inc. All rights reserved

Memory Reclamation Review

The hypervisor uses memory-reclamation techniques to reclaim host

physical memory.
 Transparent page sharing (TPS) is enabled by default on hosts using small
pages.
• TPS is a low-overhead task.
 Ballooning (can cause guest operating system paging) is more efficient
than host-level swapping:
• Both ballooning and swapping can result in memory pages being written to disk.
• Because the guest operating system is aware of its memory usage, the guest
operating system can page more efficiently than the host.
 Memory compression:
• Compresses pages in memory rather than writing them to disk.
 Host-level swapping:
• This technique quickly reclaims memory.
• It has higher performance overhead than the other techniques.
Memory reclamation enables memory overcommitment.

VMware vSphere: Optimize and Scale 9-22

© 2014 VMware Inc. All rights reserved

Memory Overhead

Memory overhead has the following components:

 VMkernel: 100MB+
• Depends on the number of devices attached to the host
 Virtual machine memory overhead:
• Varies, based on size of guest memory and number of vCPUs

Examples:

1GB RAM 16GB RAM

2 vCPUs 4 vCPUs
=> ~30MB => ~151MB
overhead overhead

VMware vSphere: Optimize and Scale 9-23

© 2014 VMware Inc. All rights reserved

System Swap

System swap is a memory reclamation process that can take

advantage of unused memory resources across an entire system.
System swap is a tradeoff between the impact of reclaiming the
memory from another process and the ability to assign the memory
to a virtual machine that can use it.
 Space required is 1GB.
Available options for background storage are:
 Datastore: Allows the use of the datastore specified (except VMware®
Virtual SAN™ datastore).
 Host Cache: Allows the use of a part of the host cache.
 Preferred swap file location: Allows the use of the preferred swap file
location that ESXi has determined.

VMware vSphere: Optimize and Scale 9-24

© 2014 VMware Inc. All rights reserved

Configuring System Swap

Select host > Manage > Settings > System Swap.

VMware vSphere: Optimize and Scale 9-25

© 2014 VMware Inc. All rights reserved

Review of Learner Objectives

You should be able to meet the following objectives:

 Describe how memory is used in a virtualized environment
 Explain each memory-reclamation technique
 Explain how memory overcommitment impacts performance

VMware vSphere: Optimize and Scale 9-26

© 2014 VMware Inc. All rights reserved

 Lesson 2:
Monitoring Memory Activity

VMware vSphere: Optimize and Scale 9-27

© 2014 VMware Inc. All rights reserved

Learner Objectives

By the end of this lesson, you should be able to meet the following
objectives:
 Monitor host memory usage and virtual machine memory usage
 Monitor host swapping activity
 Monitor host ballooning activity

VMware vSphere: Optimize and Scale 9-28

© 2014 VMware Inc. All rights reserved

Monitoring Virtual Machine Memory Usage

Consumed host memory:

 Host physical memory allocated to a
guest
Active Guest Memory:
 Guest physical memory actively used
Monitoring the memory usage is useful for quickly
analyzing a virtual machine’s status.

VMware vSphere: Optimize and Scale 9-29

© 2014 VMware Inc. All rights reserved

Memory Usage Metrics Inside the Guest Operating System

Why is guest/host memory usage different than what I see inside the
guest operating system?
 Guest physical memory:
• Guest has better visibility while estimating active memory.
• ESXi active memory estimate technique can take time to converge.
 Host physical memory:
• Host memory usage does not correspond to any memory metric within the
guest.
• Host memory usage size is based on a virtual machine’s relative priority on
the physical host and memory usage by the guest.

VMware vSphere: Optimize and Scale 9-30

© 2014 VMware Inc. All rights reserved

Consumed Host Memory and Active Guest Memory

Consumed host memory > active guest memory:

 If memory is not overcommitted, this combination is good.
 The consumed host memory represents the highest amount of memory
used by the guest.
Consumed host memory <= active guest memory:
 Active guest memory does not completely reside in host physical
memory.
 This combination points to potential performance degradation.

When might active guest memory not reside

in host physical memory?

VMware vSphere: Optimize and Scale 9-31

© 2014 VMware Inc. All rights reserved

Monitoring Memory Usage Using resxtop

Possible states:
high, soft, hard,
and low

PCI hole
VMKMEM

Host Physical Memory (PMEM)

VMKMEM: Memory managed by the VMkernel

VMware vSphere: Optimize and Scale 9-32

© 2014 VMware Inc. All rights reserved

Monitoring Host Swapping in the vSphere Web Client

Host-level swapping severely affects the performance of the virtual

machine being swapped, as well as other virtual machines.

VMware vSphere: Optimize and Scale 9-33

© 2014 VMware Inc. All rights reserved

Host Swapping Activity in resxtop: Memory Screen

Memory screen (type m) for virtual machine worlds

Total memory
swapped for all
virtual machines on
host

swap reads swap writes

per second per second

swap space swap space

currently used target

VMware vSphere: Optimize and Scale 9-34

© 2014 VMware Inc. All rights reserved

Host Swapping Activity in resxtop: CPU Screen

CPU screen (type c) for virtual machine worlds

percentage of time
virtual machine has
waited for swap
activity

VMware vSphere: Optimize and Scale 9-35

© 2014 VMware Inc. All rights reserved

Host Ballooning Activity in vSphere Web Client

Ballooning has a lower performance penalty than swapping.

 High ballooning activity can affect performance (high guest operating
system paging).

VMware vSphere: Optimize and Scale 9-36

© 2014 VMware Inc. All rights reserved

Host Ballooning Activity in resxtop

memory balloon
statistics for the host
Is the balloon
driver installed?

physical memory physical

being held for memory
other VMs target to
reclaim

VMware vSphere: Optimize and Scale 9-37

© 2014 VMware Inc. All rights reserved

Introduction to Lab 13: Monitoring Memory Performance

Case 1: Case 2:
Baseline Data after
data workloads started

# ./starttest2 vApp: RAM-HOG (Limit 1,000MB)

Linux01
Linux01
Workload Workload
01 … 06

VM VM
128MB 128MB 1GB
1GB RAM RAM RAM RAM

your ESXi host

VMware vSphere: Optimize and Scale 9-38

© 2014 VMware Inc. All rights reserved

Lab 13: Monitoring Memory Performance

Use performance charts and the resxtop command to monitor

memory performance under load
1. Prepare for the Lab
2. Generate Database Activity in the Test Virtual Machine
3. Check for Overcommittment of Virtual Machine Memory
4. Configure an Advanced Performance Chart
5. Configure resxtop to Report Virtual Machine Memory Statistics.
6. Observe Baseline Memory Statistics
7. Import the RAM-HOG vApp
8. Start a Memory Test on Each RAM-HOG Workload
9. Record Memory Statistics
10. Clean Up for the Next Lab

VMware vSphere: Optimize and Scale 9-39

© 2014 VMware Inc. All rights reserved

Review of Lab 13: Monitoring Memory Performance

Review the lab questions:

 Did the values for MCTLSZ and MCTLTGT converge?
 Did the values for SWCUR and SWTGT converge?
 Which virtual machines do not have the balloon driver installed?
 Which virtual machines experienced swapping activity?
• Did the %SWPWT value exceed 5 percent for any of these virtual
machines?
 Based on the opm value that you recorded, did the performance of the
starttest2 script degrade?

VMware vSphere: Optimize and Scale 9-40

© 2014 VMware Inc. All rights reserved

Review of Learner Objectives

You should be able to meet the following objectives:

 Monitor host memory usage and virtual machine memory usage
 Monitor host swapping activity
 Monitor host ballooning activity

VMware vSphere: Optimize and Scale 9-41

© 2014 VMware Inc. All rights reserved

 Lesson 3:
Troubleshooting Memory Performance
Problems

VMware vSphere: Optimize and Scale 9-42

© 2014 VMware Inc. All rights reserved

Learner Objectives

By the end of this lesson, you should be able to meet the following
objectives:
 Describe various memory performance problems
 Discuss causes of memory performance problems
 Propose solutions to correct memory performance problems

VMware vSphere: Optimize and Scale 9-43

© 2014 VMware Inc. All rights reserved

Review: Basic Troubleshooting Checklist for ESXi Hosts

1. Check for VMware® Tools™ status.

2. Check for resource pool

CPU saturation. 11. Check for using only one vCPU
in an SMP virtual machine.
16. Check for low guest CPU
3. Check for host CPU saturation.
utilization.
12. Check for high CPU ready time
4. Check for guest CPU saturation. on virtual machines running in
17. Check for past virtual machine
under-utilized hosts.
memory swapping.
5. Check for active virtual machine
memory swapping. 13. Check for slow storage device. 18. Check for high memory
6. Check for virtual machine swap demand in a resource pool.
wait. 14. Check for unexpected increase
in I/O latency on a 19. Check for high memory
7. Check for active virtual machine
shared storage device. demand in a host.
memory compression.

8. Check for an overloaded 20. Check for high guest memory

15. Check for unexpected increase demand.
storage device.
in data transfer rate
on network controllers.
9. Check for dropped receive
packets.

10. Check for dropped transmit

packets.

Definite problems Likely problems Possible problems

VMware vSphere: Optimize and Scale 9-44

© 2014 VMware Inc. All rights reserved

Active Host-Level Swapping (1)

No

Yes

No Yes

Yes

VMware vSphere: Optimize and Scale 9-45

© 2014 VMware Inc. All rights reserved

Active Host-Level Swapping (2)

No

Yes

VMware vSphere: Optimize and Scale 9-46

© 2014 VMware Inc. All rights reserved

Causes of Active Host-Level Swapping

The basic cause of host-level swapping:

 Memory overcommitment from using memory-intensive virtual
machines whose combined configured memory is greater than the
amount of host physical memory available
Causes of active host-level swapping:
 Excessive memory overcommitment
 Memory overcommitment with memory reservations
 Balloon drivers in virtual machines not running or disabled

VMware vSphere: Optimize and Scale 9-47

© 2014 VMware Inc. All rights reserved

Resolving Host-Level Swapping

Host-level swapping can be resolved in the following ways:

 Reduce the level of memory overcommitment.
 Enable the balloon driver in all virtual machines.
 Add memory to the host.
 Reduce memory reservations.
 Use resource controls to dedicate memory to critical virtual machines.

VMware vSphere: Optimize and Scale 9-48

© 2014 VMware Inc. All rights reserved

Reducing Memory Overcommitment

Ways to reduce the level of memory overcommitment:

 Add physical memory to the ESXi host.
 Reduce the number of virtual machines running on the ESXi host.
 Increase available memory resources by adding the host to a VMware
vSphere® Distributed Resource Scheduler™ cluster.

VMware vSphere: Optimize and Scale 9-49

© 2014 VMware Inc. All rights reserved

Enabling Balloon Driver in Virtual Machines

To enable the balloon driver in a virtual machine, install VMware

Tools.
If a virtual machine has critical memory needs, use resource controls
to satisfy those needs.

A best practice is to always

keep the balloon driver enabled.

VMware vSphere: Optimize and Scale 9-50

© 2014 VMware Inc. All rights reserved

Memory Hot Add

Add memory resources to a powered-on virtual machine that meets

the following conditions:
 The virtual machine has a guest operating system that supports
memory hot add functionality.
 The virtual machine is using at least hardware version 7 (VMware®
ESX®/ESXi 4.0 or later compatibility).
 VMware Tools is installed.

VMware vSphere: Optimize and Scale 9-51

© 2014 VMware Inc. All rights reserved

Memory Hot Add Procedure

VMware vSphere: Optimize and Scale 9-52

© 2014 VMware Inc. All rights reserved

Reduce a Virtual Machine’s Memory Reservation

Reevaluate a virtual machine’s memory reservation if:

 That virtual machine’s reservation cause the host to swap virtual
machines without reservations.
Reduce a virtual machine’s memory reservation if:
 The virtual machine is not using its full reservation.
If the reservation cannot be reduced, memory overcommitment must
be reduced.

VMware vSphere: Optimize and Scale 9-53

© 2014 VMware Inc. All rights reserved

Dedicating Memory to Critical Virtual Machines

Use memory reservations to prevent critical virtual machines from

swapping:
 Use as a last resort.
 This might move the swapping problem to other virtual machines.

VMware vSphere: Optimize and Scale 9-54

© 2014 VMware Inc. All rights reserved

Guest Operating System Paging

Monitor the host’s ballooning activity.

 If ballooning > 0 for the host, look at ballooning activity for the virtual
machines with performance problems.
• If ballooning > 0 for those virtual machines, check for high paging activity
within the guest operating systems.

Cause Solution

If overall memory demand is Use resource controls to direct

high, the ballooning mechanism resources to critical VMs.
might reclaim active memory Reduce memory
from the application or guest OS. overcommitment on the host.

VMware vSphere: Optimize and Scale 9-55

© 2014 VMware Inc. All rights reserved

Example: Ballooning Versus Swapping

Swap target is
more for the virtual
machines without
the balloon driver.

Virtual
MCTL: N – Balloon machine with
memory-
driver not active, balloon driver
hog virtual
tools probably not swaps less.
machines
installed

VMware vSphere: Optimize and Scale 9-56

© 2014 VMware Inc. All rights reserved

High Guest Memory Demand

Monitor the virtual machine’s memory usage:

 If average > 80% or peaks > 90%, high guest memory demand might
be causing problems within the guest operating system.

Cause Solution

Application and guest operating Configure additional memory for

system are using a high the virtual machine.
percentage of memory Tune the application to reduce
allocated to them. its memory demand.

VMware vSphere: Optimize and Scale 9-57

© 2014 VMware Inc. All rights reserved

When Swapping Occurs Before Ballooning

Many virtual machines are powered on at the same time.

 Virtual machines might access a large portion of their allocated
memory.
 At the same time, balloon drivers have not yet started.
 This causes the host to swap virtual machines.
Host-level swapping slows the boot process but is not a problem
after the guest is booted up.
A virtual machine’s memory that is currently swapped out to disk will
not cause a performance problem if the memory is never accessed.

VMware vSphere: Optimize and Scale 9-58

© 2014 VMware Inc. All rights reserved

Memory Performance Best Practices

 Allocate enough memory to hold the working set of applications you will
run in the virtual machine, thus minimizing swapping.
 Never disable the balloon driver. Always keep it enabled.
 Keep transparent page sharing enabled
 Avoid overcommitting memory to the point that it results in heavy
memory reclamation.

VMware vSphere: Optimize and Scale 9-59

© 2014 VMware Inc. All rights reserved

Lab 14: Diagnosing Memory Performance Issues

Use a troubleshooting checklist to detect possible performance

problems related to memory swapping and paging
1. Generate Database Activity in the Test Virtual Machine
2. Configure Advanced Performance Charts
3. Configure resxtop to Report Virtual Machines Memory Statistics.
4. Check for Active Host-Level Memory Swapping
5. Determine Which Virtual Machines Are Affected by Host-Level Memory
Swapping
6. Check for Past Host-Level Swapping
7. Determine Which Virtual Machines Are Affected by Past Host-Level
Swapping
9. Check for Guest Operating System Paging at the Host Level
10. Check for Guest Operating System Paging in Each Virtual Machine
11. Check for High Guest Memory Demand
12. Clean Up for the Next Lab

VMware vSphere: Optimize and Scale 9-60

© 2014 VMware Inc. All rights reserved

Review of Learner Objectives

You should be able to meet the following objectives:

 Describe various memory performance problems
 Discuss causes of memory performance problems
 Propose solutions to correct memory performance problems

VMware vSphere: Optimize and Scale 9-61

© 2014 VMware Inc. All rights reserved

Key Points

 The hypervisor uses memory-reclamation techniques to reclaim host

physical memory. Transparent page sharing, ballooning, memory
compression, and host-level swapping are used to reclaim memory.
 A host’s swap rates and ballooning activity are key memory
performance metrics.
 The basic cause of memory swapping is memory overcommitment of
memory-intensive virtual machines.

Questions?

VMware vSphere: Optimize and Scale 9-62

© 2014 VMware Inc. All rights reserved