How to make #ActiveMq a cluster ready service using #bash scripts and #API calls

ActiveMQ is a very nice and powerful open source, multi-protocol, Java-based message broker.

It is a very popular software used to bridge between loose components of an environment that consists of several services.

To make it really useful in a modern architecture of a complex software product we must prepare ActiveMQ to be very easily deployed in a cluster environment.

The simplest way to do it is to write some basic Bash scripts that use the ActiveMQ API and implement the most basic control functions needed by a cluster orchestrator to control the service.

This basic functions are:

• A start function that will start our service
• A stop function that will graciously stop the service
• A health check function that will be used by the cluster orchestrator to asses the health of the service and decide if the service needs to be restarted
• A status check function that will be used by the orchestrator to asses the detailed status of the service and decide if additional action should be taken: recovery , scaling the service to additional instances , downscaling the service instances etc.

Note that in the case of ActiveMQ the scaling or downscaling may not be of interest as usually we need just one instance.

Start function

To start the ActiveMQ instance we can use the following script mqStart.sh:

#!/bin/bash
source $(dirname "$0")/config.sh

function runConditionalActiveMQ {
        STATUS=`./mqHealth.sh`
        if [ $STATUS == 'UP' ] || [ $STATUS == 'DEGRADED' ]
        then
                echo ActiveMQ is already running with STATUS $STATUS
        else
                echo ActiveMQ Starting...
                $ACTIVEMQ_HOME/bin/activemq console &> /dev/null &
                sleep 10
        fi
}
runConditionalActiveMQ

Notes:

• config.sh declares the environment variables like $ACTIVEMQ_HOME
• the whole start method uses the standard way to start ActiveMQ
• note that the conditional used to check if we can start is based on the health check script mqHealth.sh

Stop function

To stop the ActiveMQ instance we can use the following script mqStop.sh:

#!/bin/bash
source $(dirname "$0")/config.sh

echo "Stopping...  " $ACTIVEMQ_HOME
$ACTIVEMQ_HOME/bin/activemq stop

Notes:

• config.sh declares the environment variables like $ACTIVEMQ_HOME
• the whole stop method uses the standard way to stop ActiveMQ

Status function

To check the status of ActiveMQ we have a very useful and already available API call, but first some extra configuration to activate it must be done.

We need to allow API calls from localhost in our case or from the host that is monitoring the service.

Change the default configuration file from apache-activemq/webapps/api/WEB-INF/classes/jolokia-access.xml to look like the following:

<?xml version="1.0" encoding="UTF-8"?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<restrict>
  <cors>
   <!-- Allow cross origin access from localhost -->
   <allow-origin>http://localhost</allow-origin>

   <!--  and all servers from localhost with any protocol -->
   <allow-origin>*://*.localhost</allow-origin>

   <!-- Check for the proper origin on the server side, too -->
   <strict-checking/>
</cors>

  <!-- deny calling operations or getting attributes from these mbeans -->
  <deny>
    <mbean>
      <name>com.sun.management:type=DiagnosticCommand</name>
      <attribute>*</attribute>
      <operation>*</operation>
    </mbean>
    <mbean>
      <name>com.sun.management:type=HotSpotDiagnostic</name>
      <attribute>*</attribute>
      <operation>*</operation>
    </mbean>
  </deny>

</restrict>

After a restart of ActiveMQ with the new configuration we can check the status with the following script mqStatus.sh:

#!/bin/bash
curl -u admin:admin -s http://localhost:8161/api/jolokia/exec/org.apache.activemq:type=Broker,brokerName=activemq,service=Health/healthStatus -H "Origin:http://localhost" | python -m json.tool 2> /dev/null

Notes:

• curl is used to call the API endpoint
• due to CORS we must specify the header -H “Origin:http://localhost”
• brokerName must be set to the name of the broker defined when we configured ActiveMQ
• admin:admin must be changed to the real user:pass pair
• python is used to beautify the output json
• status information will be available in a json format that can be further processed by the cluster orchestrator and make decisions based on the reported information.

Health Check function

To check the health of the service we use the above status API call but we reduce the status information to a simple selection of health statuses: UP/DOWN/DEGRADED

The following script mqHealth.sh implements that.

#!/bin/bash

LIST=$(curl -u admin:admin -s http://localhost:8161/api/jolokia/exec/org.apache.activemq:type=Broker,brokerName=activemq,service=Health/healthStatus -H "Origin:http://localhost" -H 'cache-co
ntrol: no-cache' \
| python -m json.tool 2> /dev/null \
| grep '\"value\"' \
| cut -d ':' -f 2 \
| tr -d '"' \
| tr -d ',')

if  [ -z "$LIST" ]; then
    echo "DOWN"
else
    UP="true"
    for value in $LIST
    do
        if [ $value != "Good" ] && [ $value != "Getting" ]; then
            UP="false"
        fi
    done
    if [ $UP == "true" ]; then
        echo "UP"
    else
        echo "DEGRADED"
    fi
fi

Notes:

• curl is used to call the API endpoint
• due to CORS we must specify the header -H “Origin:http://localhost”
• brokerName must be set to the name of the broker defined when we configured ActiveMQ
• admin:admin must be changed to the real user:pass pair
• “Getting” comes from the standard warning messages returned by ActiveMq that start with “Getting Worried “. We do not want WARNING conditions to trigger a degraded status.

Update:

In case python is not available on the platform the script has to run the following is a grep only variant.

#!/bin/bash

LIST=$(curl -u admin:admin -s http://localhost:8161/api/jolokia/exec/org.apache.activemq:type=Broker,brokerName=activemq,service=Health/healthStatus -H "Origin:http://localhost" -H 'cache-co
ntrol: no-cache' \
| grep -o '"value":[^*][^" ]*' \
| cut -d ':' -f 2 \
| tr -d '"' \
| tr -d ',')

if  [ -z "$LIST" ]; then
    echo "DOWN"
else
    UP="true"
    for value in $LIST
    do
        if [ $value != "Good" ] && [ $value != "Getting" ]; then
            UP="false"
        fi
    done
    if [ $UP == "true" ]; then
        echo "UP"
    else
        echo "DEGRADED"
    fi
fi