Expert Workshop E143 Chapter 8
Troubleshooting 800xA
Service Framework
Extended Automation System 800xA
© ABB Consult IT - 1
Revision C
Troubleshooting
Service Framework Topics
Summary of Service
Service Framework
File set distribution
Affinity
Error Scenarios
© ABB Consult IT - 2
Revision C
Troubleshooting
Service Framework List of Services
Service Process name Run in Type of
Redundancy
Alarm Manager AdvAeAlarmManager.exe AS Service/Service
Aspect Directory AfwADServer.exe AS Service/Service
File Set Distribution [Client] AfwFsdNode.exe All Nodes
File Set Distribution [Server] AfwFsdSrv.exe AS Service/Service
Lock Server AfwLockSrv.exe AS Service/Standby
Service Manager [Client] AfwSMClient.exe All Nodes
Service Manager [Server] AfwServiceManager.exe Server
Nodes
© ABB Consult IT - 3
Cross Reference Service AfwXRefSrv.exe AS Service/Service
Revision C
Troubleshooting
Service Framework List of Services
Service Process name Run in Type of
Redundancy
Soft Alarms AdvAeSoftAlarms.exe AS
Event Collector AdvAeEventCollector.exe Usually CS Service/Standby
OPC DA AdvDSOPCConnector.exe CS Service/Service
Basic History AdvHtHistorySrv.exe CS Service/Service
Time Server AfwTimeSrv.exe CS Service/Standby
Time Server Handler AfwTimeSrvHdlr.exe All Nodes
Tray Icon Service AfwTrayIcon.exe All Nodes
© ABB Consult IT - 4
Revision C
Troubleshooting
Service Framework Service Framework
Affinity can be configured
for service providers in
Service/Service relation
You must belong to
software developer group
Service to change this info.
Service Group
Defines the priority order of the
Service Provider Aspect Servers. The first one is
the Master Canditate
Service Provider
Service provider = task
Service Provider
which runs service.
© ABB Consult IT - 5
Depending on redundancy
you will have 2 or 3 service
Revision C
providers.
Troubleshooting
Service Framework Redundancy
Aspect
Server 2
4. Client
3.
Aspect 1.
Server 1 2. 3. Aspect 5.
Master Server 3
Candidate
4.
Aspect directory services
1. Client makes a transaction to its Aspect Server (AS3)
2. AS3 sends transaction to Master Candidate (AS1)
3. AS1 sends the complete transaction to AS2 and a subset back to AS3
© ABB Consult IT - 6
4. When both AS2 and AS3 send a complete transaction AS1 replies
Revision C
with a commit transaction back to AS2 and AS3
5. AS3 sends commit back to Client transaction complete
Troubleshooting
Service Framework States of Services
Init
InitializeState
ErrorState
ReadyState
SyncState MigrationState
BackgroundState ServiceState TerminationState
© ABB Consult IT - 7
Termination
Revision C
Troubleshooting
Service Framework Commands on Services
Command Description
Cold Reset Cold Reset is the same as starting over with an
empty system. For the aspect directory it
means to remove all aspect directory data and
do a total synchronization.
New Session The New session command will start create a new
Session identifier. The command is necessary
to perform in some error situations. See the
support cases for details.
Reset Force synchronization, but only for changed data.
Run Go to the target state specified.
Suspend Makes possible to get a new sub set of commands
© ABB Consult IT - 8
Revision C
Troubleshooting
Service Framework File Set Distribution
1.Client Node 1.Server Node
Components using FSD
1.SM Client 1.AfwFsdNode 1.AfwFsdSrv
•Graphics
1.CS-Lib
•Operator notes
•System synchronization 1.AfwFsdNodeControl 1.AfwFsdAccess
Priority determines
1.Client Process, 1.Disk I/O
the distribution
e.g. Workplace
1.Disk
I/O
1.Client Process,
e.g. Backup 1.NTFS
1.NTFS
1.AfwFsdNodeConfig 1.Aspect
, Aspect Directory Aspect Server
Admin Structure Inventory
Objects Priorities > 5 means
always copied to client
© ABB Consult IT - 9
Revision C
Troubleshooting
Service Framework CS Lib
Client/Server Library is a support library that helps an application
communicate with another application over a socket based protocol.
Shows the connections to all
services for the specified
application
Shows the same information
© ABB Consult IT - 10
An example where a client is
Revision C
connected to different servers,
should be corrected by setting
up affinity
Troubleshooting
Service Framework CS Lib Useful Operations
Operation Description
Clients Clients connected to the Service; IP and Port should be the same as in
AfwServiceMemTblViewer
Servers The port that the service is listening to, first hand information from windows,
AfwServiceMemTblViewer should show the same
Back End Shows the connection from other client to this server.
AddressCache All DNS queries done by CS-Lib; Unresolved => bad
Summary Summary of the service; lots of dead => bad
© ABB Consult IT - 11
Revision C
Troubleshooting
Service Framework Service Manager and SMClient
Client Node Client Node
Listen to changes from Service
Shared Shared
Manager. Writes changes to Memory
Service
Client
Memory
Service
Client
Table Table
Shared Memory Table. Info written:
• Server node name and address
SMClient SMClient
• Port number
• Id’s of related objects in the Service Structure
Servers are found in:
HKEY_LOCAL_MACHINE\SOFTWARE\ABB\AFW\Systems\<system
id>\$Servers
Server Node Server Node
Service Service
Starts Stops and supervises Manager
Service
Manager
Service
services. Information about
services to start in
Shared Shared
SMDATA.dat SMClient Memory SMClient Memory
Table Table
© ABB Consult IT - 12
Revision C
Troubleshooting
Service Framework
Service Connection with Affinity Configured
The nodes in the defined node group
gets the affinity defined for the node
group. The client will be connected to
the same machine for all applicable
services.
© ABB Consult IT - 13
Revision C
Troubleshooting
Service Framework Behavior by a Restart
Shutdown of AS2 Æ
Switch over to Aspect
Server in 2nd Pos
AS2 is back the
concerning affinity
connection goes back
© ABB Consult IT - 14
Revision C
Troubleshooting
Service Framework Caution When Removing Server
Removing a server with configuration wizard removes the server
everywhere, that means it is not present in the affinity configuration
Scenario:
1. Remove AS2
2. Add AS2 again and start of AS2
3. Stop of AS1 Æ clients have no more
connection to the system
© ABB Consult IT - 15
Conclusion: Be aware of the consequences when
Revision C
you are removing a node from the system!
Troubleshooting
Service Framework Error Scenarios
How to check service status
Read only mode
Error state
Check of synchronization
Recover a node
© ABB Consult IT - 16
Revision C
Troubleshooting
Service Framework How to Check the Service Status?
1. Add More columns e.g. Long message
2. Call up detail from context menu
3. Copy to Excel or Note pad for further
analyze
4. Use GUID search
© ABB Consult IT - 17
Revision C
Troubleshooting
Service Framework
© ABB Consult IT - 18
Revision C
AfwServiceStatus
Troubleshooting
Service Framework
Check Resources with Task Manager
Commit Charge Total
< Physical Total
Peak should never reach Limit
Æ Crash of processes
© ABB Consult IT - 19
Revision C
Troubleshooting
Service Framework
Check Individual Processes with Task Manager
Add columns in the process tabs especially Virtual Memory size,
Handle count, Thread count, User Objects and GDI Objects
Take a screen shot of the process list in each node when the
system runs normal. Use this as a reference.
© ABB Consult IT - 20
Revision C
Troubleshooting
Service Framework Read-only Mode
When does it happen?
What to do?
© ABB Consult IT - 21
Revision C
Troubleshooting
Service Framework
© ABB Consult IT - 22
Revision C
Read-only Mode
Troubleshooting
Service Framework Read-only Mode
How to bring AS2 and AS3 back?
Select the service provider for the Aspect directory
AS2 and AS3
Run the command Suspend, Cold Reset and Run in
a sequence
Aspect directory will now make a complete
synchronization
© ABB Consult IT - 23
Revision C
Troubleshooting
Service Framework Cold Start of Aspect Directory
The suspend command brings up The cold reset command deletes
a new dialog, The status changes the MainDB.afwdb file on the
© ABB Consult IT - 24
from Service Æ ready node
Revision C
Troubleshooting
Service Framework Cold Start of Aspect Directory
The Run commands makes a total
synchronization of the MainDB.afwdb files
© ABB Consult IT - 25
Revision C
Troubleshooting
Service Framework Error State in 1 out of 2 Redundancy
Objects are changed after
communication failure.
If the “wrong” AD service is in
error state after a restart,
© ABB Consult IT - 26
Restart this one first, make a
cold start of the other one.
Revision C
Troubleshooting
Service Framework
Aspect Directory Synchronization Analyze Bad Case
Afwsysinfo –csd
shows checksum
differences between
aspect directories
© ABB Consult IT - 27
Determine if it is normal or repair action should be executed.
Revision C
Troubleshooting
Service Framework
Aspect Directory Synchronization Analyze Good Case
Afwsysinfo –csd
shows checksum
differences between
aspect directories,
good case shows
also statistic about
the number of
objects
© ABB Consult IT - 28
Revision C
Troubleshooting
Service Framework Recover Node
Determine node type
Example for a Redundant Aspect Server main actions
Remove the failed node from the Domain.
Repair the failed node and restore the failed node
Disconnect and connect the node to the domain
Remove system registration and aspect directory files by using the
Configuration Wizard
Connect the node from Configuration Wizard
The aspect server will now synchronize again with the main aspect
server
Example for a Connectivity Server AC 800M
The same as for Redundant Aspect server +
Restore *.cfg file, OPC Server Panel
© ABB Consult IT - 29
Restore History from file system if backed up
Revision C
Troubleshooting
Service Framework
© ABB Consult IT - 30
Revision C
System Registration and Aspect Directory Files
Troubleshooting
Service Framework
© ABB Consult IT - 31
Revision C
Connect node to system again
Troubleshooting
Service Framework
© ABB Consult IT - 32
Revision C
ABB Logo
Maintenenance Expert workshop
Exercise 8.1 System 800xA Troubleshooting Service Framework
8.1.1 Description
This exercise deals with trouble shooting the Service Framework. In a redundant
configuration of the 800xA system.
8.1.2 Objectives
Upon completion of this exercise you will be able to:
• Understand how affinity works.
• Find out how the startup works.
8.1.3 Reference documentation
• Documentation in workshop binder.
• Administration and Security
1/4
E143-08 Exercise 8.1 - RevB.doc
8.1.4 Exercise Steps
1. Check the connection to services for the client with the Service Connection
Viewer.
2. Stop the server, which the client is connected to. If the client is connected to
different services stop the service with the Aspect Directory Service connection.
What happens?
Check the license status. What measurements do you need to take if the server is
broken for a longer time?
3. Start the server again what happens to the connections?
4. Make an affinity configuration.
2/4
Maintenenance Expert workshop
5. Use the service connection viewer in order to check the connection of client. Is the
client connected according to the affinity configuration?
6. Drag the client from the odd client node group to the even client node group.
Make reboot of the client. Inspect the connections with service connection viewer.
What happens?
7. Stop the second aspect server DCASCSx2 from configuration wizard. Check the
service connection viewer, did you get a switch over to first aspect server?
3/4
E143-08 Exercise 8.1 - RevB.doc
8. Remove the stopped Aspect Server from the configuration wizard of the running
Aspect Server. Add the server again. Wait until server is up and running. Make a
reboot of client. Why hasn’t the client connected to the original server as shown in
bullet 6?
9. Remove all nodes in the Affinity configuration. Shutdown the Aspect Server
DCASCSx1. Do you get any connection from the client after a reboot of the
client? What happens? How can you correct it?
4/4
Maintenenance Expert workshop
Exercise 8.2 System 800xA Troubleshooting 1 oof 2 Redundancy
8.2.1 Description
This exercise deals with trouble shooting 1 oof 2 redundancy. The student will be
aware how the 1 oof 2 redundancy works in a system where the network is lost.
8.2.2 Objectives
Upon completion of this exercise you will be able to:
• Recover from a situation where you have lost the network in a 1 oof 2
redundancy configuration.
8.2.3 Reference documentation
• Documentation in workshop binder.
• Administration and Security
1/2
E143-08 Exercise 8.2 - RevA.doc
8.2.4 Exercise Steps
1. Stop both network by removing the power from the switches.
2. On DCASCSx1 add a generic object in the Functional Structure. On DCASCSx2
add an other object in the Functional structure.
3. Start up the switches again. What happens to the services?
4. You want to keep the data on the machine which is in error state. What can you
do?
5. Get the Aspect Directory service in service state on both machines. Before you do
the cold init on the Aspect Directory service open up a windows explorer and
navigate to the \OperateITData\AspDir directory. Start the System Event list on
the Service object as well. Check the synchronization messages.
2/2
