Introduction
Jan 24, 2014 This document was generated from CDN thread Created by: Abdul Rasheed on 11:22:03 AM Dear All, Its with great worry that i am posting this thread, we are working on the issue for more than a week and still not able to find a solution and if it is delayed further, we will loose a big order which will in turn put me in a very bad position (.
This document describes the steps to efficiently guide you through the trace collection process of Cisco Unified Communications Manager (CUCM/CallManager). A common scenario is used for illustration.
Prerequisites
Requirements
Cisco recommends that you have knowledge of these topics:
- Real-Time Monitoring Tool (RTMT) application
- Detailed tracing for the CallManager service
- Detailed tracing for the computer-telephony integration (CTI) Manager service
Components Used
The information in this document is based on current versions of CUCM.
For versions earlier than CUCM 9.x, see Collecting CUCM Traces from CUCM 8.6.2 for a TAC SR.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Background Information
If you work with a Technical Assistance Engineer (TAC) engineer on a Communications Manager issue, you might need to collect CUCM traces. This could be a task you do infrequently or have never done before.
In this scenario, you troubleshoot a call that has not been recorded even though the CUCM-side configuration appears to be correct. The administrator receives an alarm message for each call that failed to record, so the TAC engineer has asked you to reproduce the issue and gather detailed CallManager traces, detailed CTI Manager traces, and Event Viewer logs from the CUCM side. These logs will capture the call signaling events, the CTI messages that are exchanged with the server that records the calls, and the alarms from the call that failed to be recorded.
In order to complete this task, you will need to:
- Install the RTMT application
- Configure or confirm detailed traces for the CallManager service
- Configure or confirm detailed traces for the CTI Manager service
- Reproduce the issue and take notes
- Gather the requested traces
- Verify trace file coverage
- Attach a trace package to your service request
Install the RTMT Application
In CUCM, the RTMT application is used to gather traces for most types of issues. Every major and minor version of CUCM has an associated version of the RTMT application. If, on your PC, you do not see a Unified RTMT program group under Start > Programs > Cisco, or if the RTMT version does not match your CUCM cluster, you must install the RTMT tool for your CUCM version before you move forward.
- Log in to the Cisco Unified CM Administration Page.
- Choose Application > Plugins.
- Click Find.
- Click the Download link for the Cisco Unified Real-Time Monitoring Tool – Windows plugin.
- Save the CcmServRtmtPlugin.exe file.
- Run CcmServRtmtPlugin.exe.
- Follow the directions of the InstallAnywhere installation wizard, accept the License Agreement and install to the default folder location. If you had an old version of RTMT installed, the installation wizard will direct you to uninstall the old version before you proceed.
- Choose Start > Programs > Cisco > Unified RTMT and launch Cisco Unified Real-Time Monitoring Tool.
- In the Real-Time Monitoring Tool Login window, enter the IP address for your CUCM Publisher.
- In the Add the Certificate to Trust Store window, click Accept.
- In the Authentication Required window, enter the same User Name and Password as you do to log on to the CUCM Administration page.
If you are unable to log on, add theRealtimeandTraceCollection permission to your user account, or use the application administrator account created when the system was installed and try again. - The application will open, and you will see a Select Configuration window. Click OK in order to select the Default configuration. The System Summary page opens.
You have now verified that RTMT is installed, and that you are able to log on to your CUCM cluster with the tool.
Configure or Confirm Detailed Tracing for the CallManager Service
In CUCM 9.x and later, detailed tracing is enabled by default for the CallManager service. Before you proceed, confirm that detailed tracing is still configured. If not, configure it.
- Log on to the Cisco Unified Serviceability page.
- Choose Trace > Configuration.
- From the Server drop-down list, choose the CUCM Publisher and click Go.
- From the Service Group drop-down list, choose CM Services and click Go.
- From the Service drop-down list, choose Cisco CallManager and click Go.The system is configured for the default detailed tracing, as shown in this image:
- Trace On is enabled.
- The Debug Trace Level is set to Detailed.
- All Trace Filters are enabled, with the exception of:
- Enable Miscellaneous Trace
- Enable SoftKey Trace
- Enable Route or Hunt List Trace
- Enable All GateWay Trace
- Enable SCCP Keep Alive Trace
- Enable SpeedDial Trace
- Enable SIP Keep Alive (REGISTER Refresh) Trace
- If the system is not configured with at least the default detailed tracing settings, and if you use CUCM 9.x or later:
- Click Set Default. This reverts trace configuration for this service to default settings.
- Choose Apply to All Nodes.
- Click Save.
- Confirm trace configuration on the other servers in the cluster.
If you use an earlier version of CUCM, you will need to manually configure your trace settings to match the illustration. The Set Default button on earlier versions will set the Debug Trace Level to Error, not Detailed.
Configure or Confirm Detailed Tracing for the CTI Manager Service
In CUCM 9.x and later, detailed tracing is also enabled by default for the CTI Manager service. As this could have been changed, confirm or configure this before you proceed.
- Log in to the Cisco Unified Serviceability page.
- Choose Trace > Configuration.
- From the Server drop-down list, choose the CUCM Publisher and click Go.
- From the Service Group drop-down list, choose CM Services and click Go.
- From the Service drop-down list, choose Cisco CallManager and click Go.The system is configured for the default detailed tracing, as shown in this image:
- Trace On is enabled.
- The Debug Trace Level is set to Detailed.
- Enable All Trace is enabled.
- If these settings were modified, and if you use CUCM version 9.x or later:
- Click Set Default in order to revert trace configuration to default settings.
- Check the Apply to All Nodes check box.
- Click Save.
- Confirm trace configuration on the other servers in the cluster.
As with the CallManager trace settings, if you use an earlier version of CUCM than this document is intended for, you will need to manually configure your trace settings to match the illustration. Click Set Default on earlier versions if you need to set the Debug Trace Level to Error.
Note: But what about the Event Viewer logs? You do not need to change debug levels for either the Event Viewer - Application logs or the Event Viewer - System logs. You will proceed with issue reproduction.
Reproduce the Issue and Take Notes
In this scenario, you will place test calls to generate a failure. However, if you give the TAC engineer a set of traces with no information about the test calls, they might not be able to find the calls to analyze. You also risk the collection of data for the wrong time frame and will have to start over.
For each test call, record this information:
- Calling party phone number
- Called party phone number
- Call start time
- Call end time
- Time and description for any issues you experienced during the call
As CUCM traces can be very lengthy, TAC will need those call details in order to find your test calls in the data.
Gather the Requested Traces
After the issue was reproduced, gather the traces requested by TAC immediately. This prevents the trace files from being overwritten before you can gather them.
In this scenario, you need to collect CallManager traces, CTI Manager traces, and all Event Viewer logs. Unless TAC has given you other instructions, you will collect those files from all servers for the complete time range that covers your test call or calls. This prevents you from missing traces from a server that you did not know was in the call flow.
- Launch the RTMT.
- Connect to the IP address of your CUCM Publisher.
- Log on with the same credentials you use for the CUCM Administration web page.
- Choose System > Tools > Trace & Log Central.
- Double-click Collect Files. The Collect Files window opens to Select UCM Services/Applications.
- In Select UCM Services/Applications, click the check box in the All Servers column for:
- Cisco CTIManager
- Cisco CallManager
- Click Next. The Collect Files window advances to Select System Services/Applications.
- In Select System Services/Applications, click the check box in the All Servers column for:
- Event Viewer-Application Log
- Event Viewer-System Log
- Click Next. The Collect Files window advances to the Collect File Options screen.
- Configure your Collection Time:
- As you have timestamps for your test call(s), click the Absolute Range radio button.
- From the From Date/Time drop-down list, choose the time for one minute before your first test call.
- From the To Date/Time drop-down list, choose the time for one minute after your last test call.
- Configure your Download File Options.
- Click Browse in order to configure your Download File Directory and specify a new directory for each trace file collection. For example, you might want to download these files to DesktopTACcallrectest01. Traces for a later test could go to DesktopTACcallrectest02. This keeps the collected file sets for each issue reproduction organized and separated.
- Leave all other settings set to the defaults.
- Click Finish.
You will observe that the Collect Files window updates with the status of trace collection. While trace collection is ongoing, you will see a Cancel button is available. When collection is complete, the Cancel button will be grayed out.
Verify Trace File Coverage
Review the files that you have gathered in order to ensure they cover the problem time frame. The simplest way to do this is to review the TraceCollectionResult*.xml files.
When RTMT collects a set of files, it writes a TraceCollectionResult*.xml file to the download file directory for each server that it collects data from. You will see these files along with subdirectories for each CUCM server. The TraceCollectionResult*.xml files state which files were successfully downloaded from each server . The subdirectories contain the actual trace and log files.
Open each TraceCollectionResult file and observe whether the modified date for the listed file or files map to the date and time range for your trace collection. If trace files could not be collected, for example due to their being overwritten, they will be missing.
One thing you will note if you are familiar with earlier versions of CUCM is that the Cisco CallManager traces will be a single set of SDL* traces, not a set of SDL* traces and a set of ccm* traces. This is because, in CUCM 9.x and later, traces are interleaved into a single set of files for more ease of troubleshooting. The same is true for the Cisco CTI Manager service. Instead of there being both SDL* traces and cti* traces, all of the data will be in the SDL* traces for that service.
Trace collection issues can usually be avoided if you collect traces immediately after an issue reproduction.
Note: The TraceCollectionResult*.xml files simply contain a list of files which were successfully collected from each CUCM server. TAC will need to review the actual trace and log files that were collected.
Attach a Trace Package to Your Service Request
Now that you have a complete set of traces for your issue reproduction call, you need to get them to your TAC engineer.
When you downloaded the traces, you specified a new download file directory. This directory now contains all of the log and trace files, as well as the TraceCollectionResult*.xml files. TAC will generally want all of the contents of the download file directory, not just one or two files.
In order to make this simple, upload a single .zip file using our Case File Uploader tool.
- Compress the entire download file directory to a single .zip archive file.
- Browse to https://cway.cisco.com/csc. You will be redirected to a log in page. Log in with your CCO username and password.This brings you to the Case File Uploader tool.
- Enter your service request number.
- Add your .zip file.
- Add a file description for your TAC engineer. This is a good opportunity to communicate your issue reproduction notes.
- Click Upload. The Case File Uploader tool will display upload status. Wait for the upload to complete.
- Close the browser window.
- Lastly, ensure that you have communicated to your TAC engineer all of your issue reproduction notes, whether this was through the upload tool, via email, or verbally. This allows them to start analyzing the data for you.
Analysis
The Cisco CallManager/CTI Manager traces related to the specific call can be analyzed by the Collaboration Solutions Analyzer tool (ladder diagram/annotations/filtered logs/diagnostic signatures). Check the documentation on how to use the tool:
Summary
This document explained how to use the RTMT in order to collect the most commonly needed types of detailed traces for troubleshooting CUCM with your TAC engineer, install RTMT, trace configuration in CUCM, reproduce issues, gather and verify trace files, and efficiently attach those files to a service request.
I'm currently hosting several dozen websites in Azure and recently started seeing a 'Memory Resource Exhausted' warning within the portal blade for each web app:
I'm hosting my sites across two S3 Standard (Large) app service plans, I'm getting the warning on all sites regardless of which app service plan they are on.
Interestingly enough when looking at memory usage for either app service plan I am always below 40%, memory usage is actually rather consistent. I never see spikes or anything remotely close to 85% memory usage I'm being warned about.
My question is, am I misinterpreting the warning message? Is there a different memory resource that I need to be monitoring?
Andy MehalickAndy Mehalick
1 Answer
Check working set and private set memory from kudu. There could be a problem in the private set.
Rıfat Erdem SahinRıfat Erdem Sahin
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.