Turn up the Logs!

Whenever there is a problem in Maximo and you describe it to one of your co-workers the first thing they say to you is “what do the logs say?” Logs are the first place to look when you have a problem. There are some small issues with logs in that they are all over the place and some you will need to activate to access what can be logged. When you install Maximo (with WebSphere as it’s middleware) a couple of directories will become very familiar to you. They are the SMP and WebSphere directories. The logs are in there. Deep down in there. So, lets talk about making life a little bit easier. When you set up any environment you will go through some initial setup that we would call the foundational setup. Setting up your foundation is critical for managing and maintaining an environment that runs well and does not frustrate the hell out of you because everything is all over the place. Logs are almost always overlooked, and you end up having to go to all the standard locations to dig up the information that you are looking for when things go wrong. In this article I will show you how to set up your logs so that you can retrieve them in a more polished, sophisticated way that will exemplify how a well-built environment runs.

While researching information for this blog article I came across the usual suspects and for the Maximo users out there you know what I’m talking about. Log locations, log file sizes, amount/level of what will be logged. How Maximo uses the Log4j logger. Those are all great and form the foundations of how the application can be checked. I get the most info from the WebSphere logs though. Specifically, the SystemOut.log and the SystemErr.log. These are WebSphere’s logs that are writing information about the application that it is hosting (in our case Maximo).

The first thing you would consider when setting up your logs is where they are located.  The default path is going to be something like this (C:\IBM\WebSphere\AppServer\profiles\ctgAppSrv01\logs\MXServer). I’ve gotten quite used to that being the log location, but you may want to make life a bit easier and put it into a directory like C:\WebSphere_logs. The path (location) of the SystemOut.log and SystemErr.log can be changed. If you go to the servers and click on your server there will be a link on a right-hand navigation that will be called Logging and tracing under the Troubleshooting section.

When that page opens you will see a link called JVM logs. When clicked this will be a page most Maximo workers will know, and it gives you some very good options as to how you want your WebSphere logs set up.  As you can see from my screen capture below the log files were increased from the 1 MB standard to 5 MB, because even in environments that have very little traffic a 1 MB log will fill up quickly (like get your timer out quickly).

What size your log files should be is going to be a call you’ll have to make.  Take into consideration things like your server environment, your usage of cron tasks, automation scripts, escalations, and integrations.  These are all likely to be written to the SystemOut.log file if you are running them or increasing log levels to find out what they are doing.  You can also set how many “rolling” logs you want to keep under the Maximum Number of Historical Log Files field.  On this same page you can click on the Runtime tab and you will have the option to change the path the SystemOut and SystemErr files are written to.  This is where you can change the default repository for your logs to something that is not as buried as the original location.

Now, let’s go a bit deeper into what WebSphere can do for us when logging application information.  When the standard information that the SystemOut and SystemErr is not showing anything from the errors you are receiving it may be time to “turn up” logging. As I mentioned before Maximo is an application hosted by WebSphere, and as any developer knows tracing an application will provide step by step information as to what the application is doing. WebSphere allows you to turn on tracing of its applications. This is not on by default and you will want to turn it off once you are finished due to the amount of resources that will get chewed up while tracing an application. On the same server page (Application servers > MXServer > MXserver) where we went into to access the JVM logs there is a link for Change Log Detail Levels.  Here we can “turn up” tracing. This will allow for deeper diving into the actual applications that WebSphere is running.  The default setting is *=info.  This is going to be plenty good enough for 90% of your troubleshooting. Now, let’s say strange problems are going on with your hosted applications (Maximo) you may want to “trace” the applications inputs and outputs and as you know that’s all programs are. An interface for input and it spits an output as a result of what it was given.  If you increase the level to *=fine, *=finer, or *=finest you will now get a trace.log to pop up in that good old default directory. What this will relate to you will be a trace of what the application is doing.  Depending on the components you add the more “granular” a set of detail you can log.

IBM has a very good article on their Knowledge Center entitled Log Level settings which details levels that can be traced.  (https://www.ibm.com/support/knowledgecenter/SSEQTP_9.0.5/com.ibm.websphere.base.doc/ae/utrb_loglevel.html).  You are not going to want to leave your WebSphere system in this state because this of course will chew up resources at a much greater rate than just a few rolling logs that are written to every minute (the SystemErr.log doesn’t necessarily write every minute.)

Let’s take into account when systems need to run lean and clean and have to push the system’s resources (otherwise known as processor and available memory) to their maximum potential.  So, you’re hosting Maximo on your raspberry pi and you know you will have 200 concurrent connections at any given time editing work orders, creating purchase requisitions, firing automation scripts from custom built apps and you know that 1.2GHz quad core processor is going to be working hard, you may benefit from altering your logging system. WebSphere has the ability to log in HPEL mode.  High Performance Extensibility Logs are the exact same thing as your SystemOut, SystemErr, and trace logs, but they are not written to a nice easily accessible text file with the .log extension. These logs are written in binary and I am not even geeky enough to read that. I’ll need a translator to read binary. Lucky for me the HPEL logs can be accessed with a tool called logViewer.bat (or in the case of your raspberry pi logViewer.sh). This tool can translate the binary and bring back the information to you in either a shell scripting tool or command prompt as well as give you the ability to take the retrieved data and have a text file created out of it. Is it hardcore? The answer is absolutely, but as an old saying goes extreme situations require extreme measures.

When you go to set this up you will see a statement from WebSphere saying “Advised for most installations.” I would venture to say this is not recommended for most installations. Your choice is to have the system use some memory and processor threads to write to a text file that you can set to roll over and be readily available to check out or use a command line tool to extract the binary data and present it to you or create the text file on demand. I’m going to go with having a nicer system that has more memory and bigger processor so that I can easily read my log file.

In a final parting statement, it is a best practice to return the logs back to the state you found them in. If you have “turned up” logging, “turn it down”. It’s just going to convolute the next person that has to see a lot of excess info when they are opening the logs to troubleshoot. This topic can be extensive, and I encourage everyone to read up on what logs you may need to use in your endeavors. I could keep rambling and rambling, but my goal is to pique your interest not put you to sleep.  Stay tuned for more blog articles and go get those problems fixed by seeing what it says in the logs!