Renewing SSL Certificates for IBM WebSphere and Apache Tomcat

Renewing a certificate for IBM WebSphere or Apache Tomcat is a relatively straightforward process.  There are, however, some subtle differences between renewing and installing a fresh certificate that I would like to document here primarily so I make it easier on myself the next time I need to simply renew a certificate.  If you are looking for more advanced information please consider these articles on configuring IBM WebSphere for SSL and installing or importing certificates into a WebSphere Trust Store.

If an IBM WebSphere or Apache Tomcat application server is nearing the validation end of their SSL certificates you can follow these steps to ensure that your servers remain secure and your user’s experiences are not interrupted.

IBM WebSphere

Step 1: Backup the existing certificate in case you need to revert.

  • Connect to the server where IBM WebSphere HTTP Server is installed. Navigate to the folder [IBM_HTTPServer_Home]\.
  • Backup or move the existing SSL certificate to an expired_certs folder Ex: C:\IBM\HTTPServer

renewing_ssl_certificate_websphere_apache_tomcat_ibm_maximo_update_ugrade_support_blog_help_tip_troubleshoot

Step 2: Copy the new certificate to IBM WebSphere HTTP Server home folder.

  • Copy the new Maximo SSL certificate to the [IBM_HTTPServer_Home]\ folder. If you need help requesting a new certificate, please refer to our article configuring IBM WebSphere for SSL.

Step 3: Launch IBM Key Management application to import new certificate

  • Launch the IBM Key Management application.
  • Click Export/Import button from the right-side button menu.
  • Select the Import Key radio button.
  • Change the Key file type to PKCS S12 using the drop-down menu.
  • Browser to the file name of the certificate that was copied in Step 3

ibm_maximo_blog_updating_ssl_certificate_websphere_server_security_renew_maximosecrets

  • Click the OK button. Supply the password associated with this certificate.

ibm_maximo_blog_renew_ssl_certificate_security_update_credentials_managed_service_self_help_how to

  • Click the OK button.
  • Select the label under Select a label to change. In the Enter a new label textbox provide a new label for this certificate.

  • Click the OK button.
  • Double-click the certificate that was just added with the new label.

ibm_maximo_blog_supprt_troubleshoot_fix_security_renew_ssl_certificate_update_websphere_apache_tomcat

  • Click the Set the certificate as the default checkbox at the bottom left.

ibm_maximo_blog_validation_of_ssl_certificate_update_websphere_apache_tomcat_troubleshoot_security

  • Click the OK button.
  • The certificate that was just added with the new label should now have an * next to its name.

ibm_maximo_websphere_update_apache_tomcat_renew_ssl_certificate_blog_troubleshoot_support

  • Click the OK button.
  • The certificate that was just added with the new label should now have an * next to its name.

 

Apache Tomcat

Step 1: Backup the existing certificate in case you need to revert.

  • Connect to the server where Apache Tomcat server is installed. Navigate to the folder [Apache_Tomcat_Home]\conf. Ex: C:\Apache\Tomcat\conf
  • Backup or move the existing SSL certificate to a certs folder.
  • Open the server xml file using a text editor.

ibm_maximo_suppprt_renew_new_ssl_cerificate_server_security_validation_websphere_apache_tomcat_blog

 

Step 2: Edit the server.xml file.

  • Locate the section <Connector port=”443” scheme=”https” . Change the name of the certificate if necessary.  Update the keystorePass value to the new password for the certificate.
  • NOTE: This is an XML document. Consequently, any ampersand or quote characters will need to be replaced.  For example, if your password is 1234”& then the value for keystorePass would be keystorePass=”1234&quot;&amp;”

How_to_renew_ssl_certificate_websphere_apache_tomcat_ibm_maximo

  • Save the server xml document and restart the Apache Tomcat service.

Turn up the Logs!

Whenever there is a problem in Maximo and you describe it to one of your co-workers the first thing they say to you is “what do the logs say?” Logs are the first place to look when you have a problem. There are some small issues with logs in that they are all over the place and some you will need to activate to access what can be logged. When you install Maximo (with WebSphere as it’s middleware) a couple of directories will become very familiar to you. They are the SMP and WebSphere directories. The logs are in there. Deep down in there. So, lets talk about making life a little bit easier. When you set up any environment you will go through some initial setup that we would call the foundational setup. Setting up your foundation is critical for managing and maintaining an environment that runs well and does not frustrate the hell out of you because everything is all over the place. Logs are almost always overlooked, and you end up having to go to all the standard locations to dig up the information that you are looking for when things go wrong. In this article I will show you how to set up your logs so that you can retrieve them in a more polished, sophisticated way that will exemplify how a well-built environment runs.

While researching information for this blog article I came across the usual suspects and for the Maximo users out there you know what I’m talking about. Log locations, log file sizes, amount/level of what will be logged. How Maximo uses the Log4j logger. Those are all great and form the foundations of how the application can be checked. I get the most info from the WebSphere logs though. Specifically, the SystemOut.log and the SystemErr.log. These are WebSphere’s logs that are writing information about the application that it is hosting (in our case Maximo).

The first thing you would consider when setting up your logs is where they are located.  The default path is going to be something like this (C:\IBM\WebSphere\AppServer\profiles\ctgAppSrv01\logs\MXServer). I’ve gotten quite used to that being the log location, but you may want to make life a bit easier and put it into a directory like C:\WebSphere_logs. The path (location) of the SystemOut.log and SystemErr.log can be changed. If you go to the servers and click on your server there will be a link on a right-hand navigation that will be called Logging and tracing under the Troubleshooting section.

When that page opens you will see a link called JVM logs. When clicked this will be a page most Maximo workers will know, and it gives you some very good options as to how you want your WebSphere logs set up.  As you can see from my screen capture below the log files were increased from the 1 MB standard to 5 MB, because even in environments that have very little traffic a 1 MB log will fill up quickly (like get your timer out quickly).

What size your log files should be is going to be a call you’ll have to make.  Take into consideration things like your server environment, your usage of cron tasks, automation scripts, escalations, and integrations.  These are all likely to be written to the SystemOut.log file if you are running them or increasing log levels to find out what they are doing.  You can also set how many “rolling” logs you want to keep under the Maximum Number of Historical Log Files field.  On this same page you can click on the Runtime tab and you will have the option to change the path the SystemOut and SystemErr files are written to.  This is where you can change the default repository for your logs to something that is not as buried as the original location.

Now, let’s go a bit deeper into what WebSphere can do for us when logging application information.  When the standard information that the SystemOut and SystemErr is not showing anything from the errors you are receiving it may be time to “turn up” logging. As I mentioned before Maximo is an application hosted by WebSphere, and as any developer knows tracing an application will provide step by step information as to what the application is doing. WebSphere allows you to turn on tracing of its applications. This is not on by default and you will want to turn it off once you are finished due to the amount of resources that will get chewed up while tracing an application. On the same server page (Application servers > MXServer > MXserver) where we went into to access the JVM logs there is a link for Change Log Detail Levels.  Here we can “turn up” tracing. This will allow for deeper diving into the actual applications that WebSphere is running.  The default setting is *=info.  This is going to be plenty good enough for 90% of your troubleshooting. Now, let’s say strange problems are going on with your hosted applications (Maximo) you may want to “trace” the applications inputs and outputs and as you know that’s all programs are. An interface for input and it spits an output as a result of what it was given.  If you increase the level to *=fine, *=finer, or *=finest you will now get a trace.log to pop up in that good old default directory. What this will relate to you will be a trace of what the application is doing.  Depending on the components you add the more “granular” a set of detail you can log.

 

IBM has a very good article on their Knowledge Center entitled Log Level settings which details levels that can be traced.  (https://www.ibm.com/support/knowledgecenter/SSEQTP_9.0.5/com.ibm.websphere.base.doc/ae/utrb_loglevel.html).  You are not going to want to leave your WebSphere system in this state because this of course will chew up resources at a much greater rate than just a few rolling logs that are written to every minute (the SystemErr.log doesn’t necessarily write every minute.)

Let’s take into account when systems need to run lean and clean and have to push the system’s resources (otherwise known as processor and available memory) to their maximum potential.  So, you’re hosting Maximo on your raspberry pi and you know you will have 200 concurrent connections at any given time editing work orders, creating purchase requisitions, firing automation scripts from custom built apps and you know that 1.2GHz quad core processor is going to be working hard, you may benefit from altering your logging system. WebSphere has the ability to log in HPEL mode.  High Performance Extensibility Logs are the exact same thing as your SystemOut, SystemErr, and trace logs, but they are not written to a nice easily accessible text file with the .log extension. These logs are written in binary and I am not even geeky enough to read that. I’ll need a translator to read binary. Lucky for me the HPEL logs can be accessed with a tool called logViewer.bat (or in the case of your raspberry pi logViewer.sh). This tool can translate the binary and bring back the information to you in either a shell scripting tool or command prompt as well as give you the ability to take the retrieved data and have a text file created out of it. Is it hardcore? The answer is absolutely, but as an old saying goes extreme situations require extreme measures.

When you go to set this up you will see a statement from WebSphere saying “Advised for most installations.” I would venture to say this is not recommended for most installations. Your choice is to have the system use some memory and processor threads to write to a text file that you can set to roll over and be readily available to check out or use a command line tool to extract the binary data and present it to you or create the text file on demand. I’m going to go with having a nicer system that has more memory and bigger processor so that I can easily read my log file.

In a final parting statement, it is a best practice to return the logs back to the state you found them in. If you have “turned up” logging, “turn it down”. It’s just going to convolute the next person that has to see a lot of excess info when they are opening the logs to troubleshoot. This topic can be extensive, and I encourage everyone to read up on what logs you may need to use in your endeavors. I could keep rambling and rambling, but my goal is to pique your interest not put you to sleep.  Stay tuned for more blog articles and go get those problems fixed by seeing what it says in the logs!

Import SSL Certificates into WebSphere Trust Store

Ever tried to connect your Maximo system with an external secured URL? By default, WebSphere is designed not to trust secured external URLs. It will only allow the connection if an administrator specifically instructs WebSphere to do so by importing the certificate into its Trust Store.

Here are some examples of where this may be useful:

  • Connection to a GIS REST service for integrating GIS data with Maximo
  • Connection to a secured Office 365 Email server
  • Connection to a financial system, such as SAP, that uses secured APIs to communicate
  • Connection to an SMS service for texting users when certain system events occur

… and there are many more.

Here is the message you may encounter:


BMXAA1477E - The connection failed to the HTTP handler for the endpoint. Review the error and server log files for information to indicate the cause of the issue, for example, incorrect properties in the DefaultHTTPExit.java handler class. 
com.ibm.jsse2.util.j: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is: 
java.security.cert.CertPathValidatorException: The certificate issued by CN=GlobalSign, O=GlobalSign, OU=GlobalSign Root CA - R2 is not trusted; internal cause is: 
java.security.cert.CertPathValidatorException: Certificate chaining

Look familiar? Let’s fix it.

  1. Log into WebSphere as an administrative user.
  2. Click on the Security > SSL Certificate and Key Management link in the left navigation pane.
  3. Click on the Related Items > Key stores and certificates link on the right side of the main pane.
  4. Click on the CellDefaultTrustStore item in the table.
  5. Click on the Additional Properties > Signer certificates link on the right side.
  6. Click on the Retrieve from Port button.
  7. Fill out the Host, Port and Alias fields. For example:
    1. Host: www.google.com
    2. Port: 443
    3. Alias: www.google.com

  8. Press the Retrieve signer information button. Ensure that the values seem reasonably correct (i.e. you don’t get an error back.)
  9. Press the OK button.
  10. Click the Save to Master Configuration link.
  11. Press the OK button after the changes have been synchronized with all of the nodes.

At this point you’ll need to restart Maximo, your node agents, and your deployment manager for the changes to take effect. From here forward, Maximo will now trust that URL and allow the connection.

 

Changing the hostname of a Maximo WebSphere Application Server

There may come a time when you need to change the hostname of your IBM Maximo WebSphere Application Server. This could be due to migrating the configuration to another machine, changing domain/subnet information, or copying a virtual machine for development purposes. Whatever the reason, this article aims to help get Maximo back up and running after a hostname change on the IBM WebSphere Application Server.

For example, let’s assume the old hostname is 172.31.36.36, and we want to change it to maximo.a3jgroup.com in a WebSphere v8.5.5 environment.

Step 1: Change the Host Name for the Cell Manager and Node

  1. Sign into the IBM WebSphere Application Server machine as an administrative user.
  2. Stop the IBM Cell Manager and Node Agent Windows Services if they are running.
  3. Open a command prompt as an administrator and navigate to the C:\IBM\WebSphere\AppServer\bin directory (substitute your WebSphere folder if necessary).
  4. Run the following command: wsadmin.bat -conntype NONE -lang jython
  5. At the wsadmin> prompt, issue the following command: AdminConfig.list('ServerIndex')
    This will list out the existing servers (typically the Cell Manager and Node). The results will look like:
    '(cells/ctgCell01/nodes/ctgCellManager01|serverindex.xml#ServerIndex_1)'
    '(cells/ctgCell01/nodes/ctgNode01|serverindex.xml#ServerIndex_1)'
  6. For each of the entries returned in the previous step, issue the following command:
    AdminConfig.modify('(cells/ctgCell01/nodes/ctgCellManager01|serverindex.xml#ServerIndex_1)', "[[hostName maximo.a3jgroup.com]]")
    AdminConfig.modify('(cells/ctgCell01/nodes/ctgNode01|serverindex.xml#ServerIndex_1)', "[[hostName maximo.a3jgroup.com]]")
  7. Save the changes by running: AdminConfig.save()
  8. Exit the wsadmin> prompt by running: exit
  9. Start the IBM Cell Manager Windows Service. This should enable you to log into the IBM WebSphere Console.

Step 2: Change the Host Name for the various Ports

  1. Sign into the IBM WebSphere Console as the wasadmin user.
  2. Open the Servers > Server Types > WebSphere application servers page.
  3. Click on the MXServer application server.
  4. Click on the Ports link under Communications.
  5. Click on each address that has the old hostname and change it to the new hostname. Click OK.
  6. Open the System administration > Deployment manager page.
  7. Click on the Ports link under Additional Properties
  8. Click on each address that has the old hostname and change it to the new hostname. Click OK.
  9. Open the System administration > Node agents page.
  10. Click on the nodeagent Node agent.
  11. Click on the Ports link under Additional Properties
  12. Click on each address that has the old hostname and change it to the new hostname. Click OK.
  13. Click the Save to master configuration file link.
  14. Synchronize changes back to the cell.
  15. Log out of the IBM WebSphere Console
  16. Restart the IBM Cell Manager Windows Service.
  17. Start the IBM Node Agent Windows Service.
  18. Depending on whether the MXServer application server is set to start automatically, you may need to log back into the IBM WebSphere Console to start it.

Maximo should now be running on a new host name!

Maximo SSL (HTTPS) Configuration

I won’t get into a lengthy discussion about why you should use SSL to secure browser traffic. I’ll simply offer that your Maximo environment, especially if the environment can be accessed over the public internet, should be secured. Here are a few reasons:

  • SSL Encrypts Sensitive Information
  • SSL Provides Trust and Authentication
  • SSL Can Provide Compliance in certain Industries

The disadvantages are that it costs money to obtain and verify a certificate from a Certificate Authority, and it takes some time to configure. Hopefully, this blog post will help to reduce the amount of time necessary to configure SSL with Maximo.

Please note that you can also create a self-signed certificate which will suffice in forcing the browser traffic to be encrypted. However, users will see that the certificate is not trusted in the browser, and it might lead them to ask questions or not trust the connection.

This article will take you through setting up Maximo with SSL through a GoDaddy SSL Certificate, which is currently ranging anywhere from $50.00 to $300.00 annually based on the level of encryption, features, and number of sub-domains you want to secure. We will use the most basic single-domain option for this exercise.

Step 1: Create a Key Database and Generate a new Certificate Signing Request

Run the IBM Key Management Utility

  • Sign on to the server where the IBM HTTP Server is installed as an administrator.
  • Run the IBM Key Management Utility program from your Start Menu > IBM HTTP Server V8.5

Create a Key Database

  • Create a new Key Database File. This file will hold your private keys and must be kept secure:
    • Key database type: CMS
    • File Name: maximo.kdb
    • Location: C:\IBM\HTTPServer\ (provide an appropriate path for your system)
    • Click OK

Secure the Key Database

  • You will be prompted for a password to secure your new key database:
    • Enter a password
    • Confirm the password
    • Check the Stash password to a file checkbox
    • Click OK

Create a New Certificate Signing Request

  • Create a New Certificate Request from the menu options. This request, also known as a Certificate Signing Request (CSR), is what will be sent to the Certificate Authority for verification. With the information provided in the request, the CA will do their homework on you to make sure you are who you say you are, including possibly attempting to contact you or your business by email or phone. It’s important that this information is as accurate as possible:
    • Key Label: MAX_SSL_KEY
    • Key Size: 2048
    • Common Name: www.mycompany.com (this is the host name that you want to secure and must be correct!)
    • Organization: My Company Inc
    • Locality: Your city
    • State/Province: Your state
    • Zipcode: Your zip
    • Country or region: Your country
    • Email Address: admin@mycompany.com
    • Enter the file name: C:\IBM\HTTPServer\maximo.arm
    • Click OK. We’re going to leave the IBM Key Management Utility open for now. We’ll come back to it later.

Step 2: Purchase an SSL Certificate, Upload the CSR, and Download the Certificate

Purchase an SSL Certificate

Request an SSL certificate

After you purchase an SSL certificate, you need to request it for the website’s domain name (or “common name”) you want to use.

Activate your credit

  1. Log in to your GoDaddy account.
  2. Click SSL Certificates.
  3. Next to the SSL certificate credit you want to use, click Set up.
  4. If you have multiple credits, select the credit you want to use, and then click Set up.
  5. Refresh the page; you should see a New Certificate. If you don’t, continue to refresh the page until you do.

Request your certificate

  1. Next to your New Certificate, click Manage.
  2. Select Provide a CSR, and then enter the CSR from your server. Back on the server where the IBM HTTP Server is installed, open the C:\IBM\HTTPServer\maximo.arm file in your favorite text editor. Copy its contents. This is your Certificate Signing Request (CSR).
  3. Click Request Certificate.

Verification Process

  • GoDaddy will verify your certificate request. How long this takes depends on the type of certificate (typically between 1 and 7 days). Once we have that certificate we can continue with the process. Until then, we wait.

Download the Certificate

  • After GoDaddy approves your SSL certificate request, you can download your primary and intermediate certificate from within the SSL application on the GoDaddy website.
  • On your SSL certificate home page, click Download.
  • Select the Apache server type.
  • Click Download ZIP file.

Copy the Certificate to your IBM HTTP Server

  • Copy the ZIP file to the C:\IBM\HTTPServer\ folder on the server where the IBM HTTP Server is installed.
  • Extract the contents of the ZIP file to the C:\IBM\HTTPServer\ folder.
  • Note that there should be two files: one that represents your certificate, and one that represents GoDaddy’s intermediate certificate. You will need both to install the certificate in the next step.

Step 3: Install the Certificate

Run the IBM Key Management Utility

  • Sign on to the server where the IBM HTTP Server is installed as an administrator.
  • Run the IBM Key Management Utility program from your Start Menu > IBM HTTP Server V8.5
  • Open your maximo.kdb key database created earlier, using the password you created earlier to unlock the key database.

Install the Intermediate Certificate

  • Change the Key database content drop-down to Signer Certificates.
  • Click the Add… button
  • Choose the intermediate certificate file that you extracted in the previous step. Please note that the file name is likely to be of the nature gd_bundle_*.crt, and that the CRT file extension is not in the default list in the Key Management Utility file browser. Simply change the file extension drop-down to All Files or paste the exact file name into the window.
  • Click OK
  • You should see the intermediate GoDaddy certificates listed.

Install the Certificate

  • Change the Key database content drop-down to Personal Certificates.
  • You should see the MAX_SSL_KEY record that was created earlier when we created our CSR. Highlight that record. Note that the * in front of the MAX_SSL_KEY record indicates it is the default key.
  • Press the Receive button.
  • Choose the certificate file that you extracted in the previous step. Please note that the file name is likely to be some has sequence with a file extension of .crt, and that the CRT file extension is not in the default list in the Key Management Utility file browser. Simply change the file extension drop-down to All Files or paste the exact file name into the window.
  • Click OK
  • If all goes well you should see a Validation Successful message in the Key Management Utility! This means that your key database is now validated with a signed certificate from a Certificate Authority (GoDaddy). We’re almost done!

Step 4: Update the HTTP Server, WebSphere, and Maximo

Update the IBM HTTP Server

  • Sign on to the server where the IBM HTTP Server is installed as an administrator.
  • Edit the C:\IBM\HTTPServer\conf\httpd.conf file. Please note that the directory path may be different based upon your installation (e.g. C:\Program Files\IBM\HTTPServer\).
  • Add the following snippet to the file:
    # Maximo SSL Config
    LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
    Listen 0.0.0.0:443
    ## IPv6 support:
    #Listen [::]:443
    <VirtualHost *:443>
    SSLEnable
    </VirtualHost>
    KeyFile "C:/IBM/HTTPServer/maximo.kdb"
    
  • Save the file.
  • Restart the IBM HTTP Server V8.5 Windows Service.

Update the WebSphere Virtual Host

  • Log into the IBM WebSphere Console as an administrator.
  • Navigate to Environment > Virtual Hosts.
  • Click on the maximo_host Virtual Host.
  • Click on Host Aliases under Additional Properties.
  • Click on the New… button.
  • Add the host name that you used for the Common Name (e.g. www.mycompany.com) in your certificate request as the Host Name. Specify 443 as the Port.
  • Click OK and save the configuration to the Master File.
  • Restart your IBM WebSphere Windows Services (Cell Manager and Node Agent).

Update your Maximo System Properties

  • At this point, you should be able to access Maximo over SSL (HTTPS). There are just a few System Properties that should be updated in order to have everything buttoned up.
  • Log into Maximo as an administrator.
  • Navigate to the System Configuration > Platform Configuration > System Properties application.
  • Update the following properties, which should just involve changing the existing property value to change the http:// to https://
    • mxe.doclink.path01: This is for attached documents.
    • mxe.help.protocol: This is for application help. Please note that in 7.6.0.4 the help is now pointing to IBM’s website. In either case, the property value should be changed to https.
    • mxe.int.webappurl: This is for Integration Framework web service and HTTP calls.
    • mxe.oslc.restwebappurl: This is for OSLC calls through the REST interface.
    • mxe.oslc.webappurl: This is for OSLC calls through the standard interface.

 

That’s it! I hope this has been helpful. Please feel free to leave feedback in the comments section below.

 

Securing Maximo by Forcing Users to SSL (HTTPS)

You’ve taken the step of securing your Maximo environment by implementing SSL in your WebSphere environment. However, just because you’ve implemented the SSL configurations doesn’t mean users must use them. How do you force users to append that little “S” to the back of the HTTP when they navigate to Maximo?

There are passive options, to be sure, but why not force users to the HTTPS address? If the Maximo environment is open to the internet do you really want your data passing through un-encrypted? One method of forcing SSL/HTTPS is by using Apache’s Rewrite Module which we’ll describe below. This way, if a user forgets to use the proper address, they will be automatically re-routed to the correct address.

  1. Ensure that your system is properly setup to handle SSL (HTTPS). I can’t stress this enough. Before forcing users to use secure protocols, make sure that those protocols are working properly. If you need assistance, visit our blog post on Configuring SSL with Maximo.
  2. Backup the httpd.conf file, normally located in the C:\IBM\HTTPServer\conf directory.
  3. Open the httpd.conf in your favorite text editor.
  4. Add the following lines to the file, substituting the appropriate path for “C:/IBM/HTTPServer” for your file system:
    LoadModule rewrite_module modules/mod_rewrite.so
    # Rewrite Rule for SSL. Ensure traffic on SSL.
    RewriteEngine On
    # If it's not 443 (SSL Port) ...
    RewriteCond %{SERVER_PORT} !^443$
    #...redirect it to the same address but make it SSL
    RewriteRule ^(.*) https://%{SERVER_NAME}%{REQUEST_URI} [L,R]
  5. Restart your IBM HTTP Service.
  6. Test your solution!