Introducing the

IBM Support Assistant (ISA)

for WebSphere on z/OS

John Hutchinson
IBM Washington Systems Center
Share 115 – Boston, August, 2010

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 1

Outline: ISA Tools for WebSphere on z/OS

Comments:
–Many work on z/OS,
Garbage Collection analysis:
–Some do not..
– GC and Memory Visualizer (GCMV) **
– Pattern Modeling Tool for Java GC (PMAT) –Some are Deprecated
Dump analysis: –Some are “Tech.Preview”
– Memory Analyzer (MAT) ** –Analysis vs. Visualization
– Dump Analyzer **
– HeapAnalyzer
– Memory Dump Diagnostic for Java (MDD4J)
– Thread & Monitor Dump Analyzer for Java (TMDA)
Trace analysis:
– Trace and Request Analyzer for WAS
– Log Analyzer
Real-time analysis: Health Center **
Configuration analysis: Visual Configuration Explorer (VCE)
** “IBM Monitoring and Diagnostic Tools for Java™”
Appendix
– Installing the IBM Support Assistant
– Other Diagnostic Tools for WebSphere on z/OS
– Working with Dumps & Traces
– Testing Tools & Sample Applications
– Documentation, Help & Web Resources
7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 2
 Diagnostic Data from WAS on z/OS
TSO/SDSF
• Browse logs
• MVS Modify cmds
• Turn tracing on
• Display WAS Servers
Other TSO/ISPF Apps
• RMF Monitors
• IPCS
• DB2 Admin
Telnet Client
• View logs
• UNIX commands
• FTP
Java tools
• Jextract, . . .
Web Browsers
• ISC - TPV
• z/OSMF Inc.Log Mgr
ISA Tools
• Health Center
• GC analyzers
• Dump analyzers
• Log Analyzers
• VCE

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 3

Which Tools for which Artifacts?

Artifact Tool
GCverbose xml (SYSOUT) GCMV
PMAT
Javacore TMDA
Heapdump Memory Analyzer
MDD4J
HeapAnalyzer
System Dump (jextracted) Memory Analyzer
Dump Analyzer
Logs & Traces TRA
Log Analyzer
Real Time Running Server Health Center
Configuration XML files VCE
7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 4
IBM Support Assistant (ISA) - Launch pad:
Use “Launch Activity” pull-down > “Analyze Problem”

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 5

GC Visualization and Analysis

<af type="tenured" id="13" timestamp="Jul 30 17:33:20 2009" intervalms="1.601">
<minimum requested_bytes="24" />
<time exclusiveaccessms="0.132" meanexclusiveaccessms="0.063" threads="4" lastthreadtid="0x43FA5
<refs soft="249" weak="11193" phantom="2" dynamicSoftReferenceThreshold="0" maxSoftReferenceThre
<tenured freebytes="0" totalbytes="536870400" percent="0" >
<soa freebytes="0" totalbytes="536870400" percent="0" />
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
<gc type="global" id="17" totalid="17" intervalms="1.684">
1. Enable verbose GC in your ISC:
<compaction movecount="6519" movebytes="1212976" reason="low free space (less than 4%)" />
<timesms mark="644.584" sweep="67.260" compact="430.816" total="1142.777" />
– AppServer -> Process defs -> Servant -> JVM, click on
<tenured freebytes="8168" totalbytes="536870400" percent="0" >
“Verbose garbage collection”.
<soa freebytes="8168" totalbytes="536870400" percent="0" />
– Click “OK”, Save the changes, and re-start server.
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
2. Verify the verbose GC setting using SDSF
</gc>
<tenured freebytes="6120" totalbytes="536870400" percent="0" >
– Look for <af> & <gc> xml structures in servant’s log.
<soa freebytes="6120" totalbytes="536870400" percent="0" />
3. Drive an application in your server.
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
– Use your own, or sample apps provided.
<refs soft="249" weak="11193" phantom="2" dynamicSoftReferenceThreshold="0" maxSoftReferenceThre
– Jmeter, MS Web AppStress tool or your own to drive load.
<time totalms="1142.955" />
</af>
4. Copy verboseGC output from SYSOUT to ASCII file
(on your workstation)
5. Launch GC verbose visualizers (GCMV or PMAT) and
browse to the file you uploaded.

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 6

 GC Tuning Suggestions
Do this before putting your applications into Production:
1. Install the application, test it.
2. Set JVM heap min/max size, & enable Verbose GC
3. Use a load driver to drive a representative, but constant load.
(Consider using a “peak” load.)
4. Extract verbose GC statistics and study with a visualizer.
5. Under peak load, the time between GCs should be:
• Constant, and about once every 10 seconds or more.
6. Look for Memory leaks – times between GC shorten.
7. Tune by experimenting independently with different heap sizes
and GC policies.
8. To study the JVM contents, take a heap, core, or SDump, and
use one of the Dump analysis tools.

(Don’t forget to also analyze your applications with a profiler such as

JinsightLive for System z.)

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 7

GC Visualization and Analysis tools

Launch one of the GC verbose visualizers (GCMV or PMAT) and
browse to the file you uploaded.

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 8

 GC and Memory Visualizer (GCMV) - IMDTJ
Uses verbose GC (xml doc)

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 9

Pattern Modeling & Analysis Tool for Java GC (PMAT)

Different perspective than GCMV - PMAT does not run in the ISA
workspace. When you click Next, a separate display appears.

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 10

 JVM Dumps - Many kinds . . .
• Java Dumps > /var/home/…
– Heap
– JavaCore
– JVM TDUMP:
• SVC Dumps, ABEND Dumps, SYSABENDs
• CEE Dumps
Analysis Tools:
• MDD4J – high level – good for “Leak Suspects”
• Memory Analyzer – More detailed
• Dump Analyzer
• Heap Analyzer
• Thread & Monitor Dump Analyzer for Java (TMDA)
See “Analyzing Dumps” in appendix for more details.
7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 11

Memory Dump Diagnostic for Java - V3 (MDD4J)

• Focus on data structures that may cause memory related problems.
• Analyze heap dumps:
– “Baseline” heap dump – take when application is started.
– “Primary” heap dump – when a problem occurs.
– “Comparative” heap dumps – analyze differences.

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 12

 MDD4J analysis (cont’d)
• Guided Activity Assistant
– “Summary” – Basic Heap Info & Contents Summary.
– “Reach Size Leak Objects” – Cumulative size of all objects.
– “Object/Class” leak suspects.
– “Data Structures view” – Understand relations & Track Changes.
• Big/Growing/Shrinking/Steady
• Yeti Reports: (HTML or Yeti zip file)
– Save as HTML file, or Yeti analysis zip file – extract & open index.html file.
– Analysis engine based on "Yeti" technology.
• Overview
• Health Report
• Content Graphs – Type & Field layout views
• Additional documentation:
– Techdoc WP101612 – “Getting started with GC, Heapdumps and Javacores for
WebSphere on z/OS”
– DeveloperWorks Summary doc:
www.ibm.com/developerworks/websphere/techjournal/0909_supauth/0909_supauth.html

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 13

Yeti Reports (from MDD4J)

Extract zip file created by MDD4J, & launch index.html

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 14

 Memory Analyzer (IMDTJ)
• Based on open source M.A.T. + DTFJ - www.eclipse.org/mat
• For java heapdumps, System Dumps (jextracted) & javacore*

Getting Started wizard:

• Overview ------------------Æ
• Leak report
• Component reports

Navigate: (TOC, Fly-over, Click)

• Histogram (tables)
• Context (In, Out Refs)
• Group (Pkg, Class, Loader)
• Dominator Tree
– Objects keeping others
• Path to GC roots
• Leak reports

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 15

Memory Analyzer samples

Navigate: (TOC, Fly-over, Click)
• Histogram (tables) Example: Biggest objects by Retained Size:
• Context (In, Out Refs)
• Group (Pkg, Class, Loader)
• Dominator Tree
– Objects keeping others
• Path to GC roots
• Leak reports

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 16

 Memory Analyzer sample use case
File>Open Heap Dump > Leak Suspects Report > Problem Suspect1 >
“Shortest Paths To the Accumulation Point” and “Accumulated Objects”
• There we found the suspect “MemoryLeak.”

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 17

Dump Analyzer - IMDTJ

• Based on JVM’s jdmpview tool
• System dumps processed by jextract
• FTP to workstation (binary)
• Import dump – “Browse”
• Select report, “Analyze”
– Click on WASBasicInfo . . . or . . .

Upper-rt corner of screen

• Save Report as…
– HTML, XML, txt
7/27/2010 • Save Log as …18
SHARE 115 - Boston © 2010 IBM Corporation
Dump Analyzer – Analysis Reports
- Save Report As: html, xml, or txt files
- Default Dump Report:
2. Analysis results
DumpAnalyzer V:2.2.2.20090926232659 : Start analysis of C:\AppTraces\SYS1.DUMP.D091014.zip
2.1. Results from Analyzer=com.ibm.dtfj.analyzer.jvm.DefaultDumpReport_Standard
Analyzer full name: com.ibm.dtfj.analyzer.jvm.DefaultDumpReport_Standard
Analyzer version: 1.3.0.20070812
Analyzer description: Report basic information from this JVM image (similar to javacore) - Standard version
2.1.1. Image and runtime information
Now reporting on runtime: 0.0.0
Image: (no identity)
Time of dump: Wed Oct 14 02:28:14 EDT 2009
System Type: z/OS System SubType: 01.10.00
Processor Type: s390x Processor SubType:
Number of Processors: 2
Installed Memory: 6442450944
Host Name: wsc2
IP address: /9.82.24.70
This Image contains: 1 address spaces; 1 processes; 1 runtimes
Process: PID:0x30208
Executable: main
Command line: [<null>]
Pointer size (bits): 64
Signal that triggered this dump: 0 ((no signal info available, or dump was not triggered by a signal))
Current Thread: 0x14b06000
Java Runtime: JavaVM@0x000000080B178548
Java Version: Java(TM) SE Runtime Environment(build pmz6460sr5ifx-20090623_02 (SR5)) IBM J9
2.1.2. JVM Initialization Arguments
JNI ignoreUnrecognized: False
JNI version: 65540
-Xthr:tw=HEAVY ExtraInfo=0x00000000
7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 19

HeapAnalyzer (New on ISA!)

• Originally provided on AlphaWorks
• Graphical analysis of javacores (heapdumps)
• Creates tables & graphs
• Heuristic engine to find memory leaks and excessive heap usage.

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 20

 HeapAnalyzer (cont’d)
• Analysis Views:
– Tree view
– Objects List
– Types List
– Root List
– Gaps by size

See detailed information of a node; search for total size drop between parent and child.
• Tree node format: TotalSize(Size/Heap%)[ObjSize] #ChildObj (# root obj) Name Addr
• Select a node and right click to find an address.

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 21

Thread & Monitor Dump Analyzer for Java (TMDA)

TMDA for Java analyzes javacore dumps and helps you find hangs, deadlocks,
resource contention, and bottlenecks in Java threads. (Tech preview.)
:
Webcast & Foils ibm.com/support/docview.wss?uid=swg27011855

Features provided & Color coding:

• Summary of Javacore
• Thread detail view
• Monitor detail view
• List of hang suspects
• Thread compare view
• Thread comparison summary
• Monitor lock compare view
• Garbage collector statistics for IBM JVM

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 22

 Working with Traces from WAS on z/OS

• STDOUT & STDERR traces usually go to JES logs

– Keep them off SYSLOG by specifying separate DD files
– May be re-directed to HFS files – TD103695
– Not suited for ISA trace analysis tools
• java tracing - enable dynamically:
– Modify (F) MVS commands:
• F server,tracejava=‘com.ibm.*=all’
• F server,tracedetail=(3,4)
• F server, TRACEINIT
• ffdc logs – automatically generated
• wsadmin.sh tracing – modify wsadmin.properties file
• JDBC tracing – db2.jcc.override.traceFile= in
DB2JccConfiguration.properties file

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 23

TRA - Trace and Request Analyzer for WAS

• Here’s a Gap Analysis for the addNode.log:
(Works with trace files from wsadmin logs.)

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 24

 Log Analyzer –
Importing Logs
Tivoli Log Analyzer opens in a new window
Click on File Æ Import Log from local system
Select log file Æ Symptom Catalog (WAS V.7) Æ Finish
wsadmin logs, verboseGC

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 25

Health Center:
Real-time Monitor with low overhead
• YouTube videos
1. Overview of Health Center features
www.youtube.com/watch?v=5Tcktcl0qxs
2. Install IBM Support Assistant and Health Center
www.youtube.com/watch?v=6WjE9U0jvEk
3. How to enable a Java application for live monitoring by the Health Center
www.youtube.com/watch?v=Hdp0mJ13NLQ
• Enabling a server (very easy!):
– Add –Xhealthcenter to servant JVM properties
• Updating the healthcenter.jar file
– Download, unpax & copy the z/OS agent pkg (mz64.pax) from ISA Help
• Using: Launch from ISA with host & port (1972)
• Views:
– Classes
– Environment
– Garbage Collection
– I/O (New!)
– Locking
– Native Memory
– Profiling

See also SHARE 113 (Denver) Presentation – Session 1153

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 26
 Health Center - Profiling
• Lower overhead than Jinsight Live for z, but not as rich.

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 27

Health Center - Classes

• Timeline shows Class Loading frequency

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 28

 Health Center - Environment
• Verify environmental variables, and Classpath setting

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 29

Health Center – Garbage Collection

• Graph shows Heap Use, Size, and Pause Times

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 30

 Health Center – I/O View
• Graph shows Number of Files Open over Time

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 31

Health Center - Locking

• Identify Lock Contention and avg. Lock Hold Times

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 32

 Visual Configuration Explorer (VCE)
Visualize, explore, and analyze configuration information
Correlate configuration information between different products
Collaborate with others during problem analysis
Also see SHARE 113 Session 1153 for more examples…

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 33

ISA VCE “Headless Installation”

Click on File Æ Preferences Æ VCE Æ Utilities Æ “Deploy” to create
VCEHeadlessRuntime_V1.0...zip file

Transfer zip file to your USS working directory, and unzip to ./vce dir.
jar –xvf VCEHeadlessRuntime_V1.0.15.20090624.0925.zip
Run the exporter from ./vce/ directory after setting WAS_HOME:
Java jar startup.jar -buildfile wasexporter/wasexport.xml -Dwas.root=$WAS_HOME
FTP the wasexport.xml configuration file to your workstation and re-name it.
7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 34
 VCE Quick Start
• Create Workspace
• Add Configuration(s)
• Create New Diagram
• Drag objects to diagram
• Show Parents or
Children

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 35

Comparing Object Properties

• Right-Click on “Show Properties Node”
• Right-Click on “Compare Objects”

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 36

 Appendix
• Installing the ISA
• Other Diagnostic Tools for WebSphere on z/OS
– MVS Modify (F) commands

• Working with Dumps & Traces

– Triggering dumps & Traces
– Extracting system dumps for ISA tools

• Testing Tools & Sample Applications

• Documentation, Help & Web Resources

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 37

Installing the IBM Support Assistant (ISA)

• Download & Install ISA
– http://www.ibm.com/software/support/isa/
– See IEA (IBM Education Assistant) for tutorials:
• http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/index.jsp?
topic=/com.ibm.iea.selfassist/selfassist/ISAv41_Task.html
– Install Product Add-ons: (100’s)
• ISA: Update > Find New… > Tools Add-ons
• http://www.ibm.com/support/docview.wss?uid=swg27013279

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 38

 Other Diagnostic Tools for WebSphere
• WebSphere “built-in” tools (See InfoCenter)
– Display server - MVS Modify commands
– Trace Dynamically - MVS Modify (F) commands
– Performance Monitor (PMI) & Viewer
– threadmonitor (WAS V7)
– dumpNameSpace
– wsadmin
– ISC
• Other:
– JinsightLive for IBM System z
http://www.alphaworks.ibm.com/tech/jinsightlive

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 39

The all-powerful MVS Modify command

F <server_name>,HELP
• TRACEALL - Set overall trace level
• TRACEBASIC - Set basic trace components
• TRACEDETAIL - Set detailed trace components
• TRACESPECIFIC - Set specific trace points
• TRACEINIT - Reset to initial trace settings
• TRACENONE - Turn off all tracing
• JAVACORE - Generate jvm core dump
• HEAPDUMP - Generate jvm heap dump
• JAVATDUMP - Generate jvm tdump
• TRACEJAVA - Set java trace options

F <server_name>,DISPLAY,HELP
• LISTENERS - Display listeners
• CONNECTIONS - Display connection information
• TRACE - Display information about trace settings
• JVMHEAP - Display jvm heap statistics
• WORK - Display work elements
• ERRLOG - Display the last 10 entries in the error log
• THREADS - Display thread status
7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 40
 Working with Dumps
• Java Dumps > /var/home/… MVS console commands
– Heap: MVS command: F <server_name>,HEAPDUMP
– Core: F <server_name>,JAVACORE
– JVM TDUMP: F <server_name>,JAVATDUMP
Heapdumps (and javacore) written to home dir
(Default = /var/WebSphere/home/cell/<servant Group ID>

• WebSphere ABEND dumps >/var/home/…

• SYSABEND & CEE Dumps > JES SPOOL
• SVC Dumps, ABEND Dumps > SYS1.DUMP.Dyy
– Setup Dump Options:
CD SET,SDUMP=(RGN)
D DUMP,O
SDUMP- ADD OPTIONS (LSQA,RGN,TRT)
– MVS Dump console command:
Dump COMM=‘title’
R nn,jobname=H7SR01AS

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 41

jextract - Convert SVC dump for Dump Analyzer

• jextract is shipped with the IBM JVM in the ../java/bin/ directory
• Point directly to the Dump dataset:
/shared/zWebSphere/V7R0/java64/J6.0_64/bin/jextract SYS1.DUMP.D091014.T102814.SYSB.S00003
Loading dump file...
Read memory image from SYS1.DUMP.D091014.T102814.SYSB.S00003
Set debug scratch space size to 8 MB
VM set to 000000080B178548
Dumping JExtract file to SYS1.DUMP.D091014.T102814.SYSB.S00003.xml
<!-- extracting gpf state --> <!-- 5ms -->
<!-- extracting host network data --> <!-- 9ms -->
<!-- extracting classes --> <!-- 183200ms -->
<!-- extracting monitors --> ..<!-- 188234ms -->
<!-- extracting threads --> .<!-- 195727ms -->
<!-- extracting trace buffers --> <!-- 195740ms -->
<!-- extracting roots -->.<!-- 199500ms -->
<!-- extracting objects --> .<!-- 268266ms -->
Finished writing jextract XML file in 268268ms
Warning: found 14 inconsistencies in the dump file. Further information has been written to the
jextract XML file
Creating archive file: SYS1.DUMP.D091014.T102814.SYSB.S00003.zip
Adding "SYS1.DUMP.D091014.T102814.SYSB.S00003"
Adding "SYS1.DUMP.D091014.T102814.SYSB.S00003.xml"
Adding "/shared/zWebSphere/V7R0/java64/J6.0_64/lib/TraceFormat.dat"
Adding "/shared/zWebSphere/V7R0/java64/J6.0_64/lib/J9TraceFormat.dat"
jextract complete..

• FTP the resulting .zip file to your workstation.

• May have to specify these arguments:
export J9DBGEXT_SCRATCH_SIZE=8
jextract -J-DJavaio.tmpdir=/u/hutch/largezfs-Xmx1024 SYS1.DUMP.D091014.
jdmpview - Replaced by the ISA Dump Analyzer or Memory Analyzer?

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 42

 Testing Tools
• Load Drivers
– JMeter
– Microsoft ™ Web Application Stress Tool
– LoadRunner, Silk, …
• Sample Applications
– Techdoc WP101612 “Getting started with analysis of GC,
Heapdumps and Javacores For WebSphere on z/OS”
• PD Application Tools: causeOutOfMemory, Sleeper

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 43

ISA Built-in Documentation (local eclipse):

“Find Information” - Search

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 44

 ISA Help
http://localhost:1260/help/index.jsp
– Highlight a tool
(e.g., HeapAnalyzer)
– Click on the “Help” tab:

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 45

ISA Resources
ISA website:
www.ibm.com/support/docview.wss?rs=3455&uid=swg27012682
www.ibm.com/support/docview.wss?rs=3455&context=SSLLVC&q1=vce&uid=swg27013116

ISA Download website www.ibm.com/software/support/isa/download.html

ISA V4.1.1 ReadMe www.ibm.com/support/docview.wss?uid=swg27016886
IBM Support Assistant Team www.ibm.com/software/support/isa

ISA Forum www.ibm.com/developerworks/forums/forum.jspa?forumID=935

Comments/feedback to: IBMSA@us.ibm.com
IBM Education Assistant (IEA) tutorials:
publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/index.jsp?topic=/com.ibm.iea.selfassist
/selfassist/ISAv41_Task.html

IBM Diagnostics Guides www.ibm.com/developerworks/java/jdk/diagnosis/index.html

IBM Techdocs at www.ibm.com/support/techdocs: WP101575 & WP101612

7/27/2010 SHARE 115 - Boston © 2010 IBM Corporation 46