Ad Astra Per Aspera

Automating shelving of Jenkins jobs

The current version of the shelve plugin does not support shelving multiple projects. This can be a pain if you have a large Jenkins deployment. Luckily, it is possible to shelve multiple projects using a simple bash script.

If you have the jenkis-cli jar on your system, this can be can in two steps:

First generate a list of jobs to shelve:

java -jar jenkins-cli.jar -noCertificateCheck -http -user <your user> -i <key> -s http://jenkins.host.com:8080/ list-jobs | grep jobs-I-want-to-remove > ~/tmp/shelve-list.txt

1	java -jar jenkins-cli.jar -noCertificateCheck -http -user <your user> -i <key> -s http://jenkins.host.com:8080/ list-jobs \| grep jobs-I-want-to-remove > ~/tmp/shelve-list.txt

Iterate over the list to shelve the projects:

i=0 ; for job in $(cat ~/tmp/shelve-list.txt) ; do curl -u <user>:<password> -X POST -D "{}" http://jenkins.host.com:8080/job/$job/shelve/shelveProject ; ((i++)) ; [[ $i == 10 ]] && echo "Waiting ... " && sleep 10 && export i=0; done

1	i=0 ; for job in $(cat ~/tmp/shelve-list.txt) ; do curl -u <user>:<password> -X POST -D "{}" http://jenkins.host.com:8080/job/$job/shelve/shelveProject ; ((i++)) ; [[ $i == 10 ]] && echo "Waiting ... " && sleep 10 && export i=0; done

This script uses a small delay between every couple shelve requests. This is useful if your jobs contain a lot of data and Jenkins will take too long to shelve them.

Running Apache Artemis with 32Mb of JVM heap memory

As part of the effort to run Maestro on the Minicloud, I decided to tweak the Maestro broker used by the test cluster. The broker component provides the infrastructure for the test nodes to exchange messages. It is used, for example, to synchronize the start and stop of the test execution and share success and failure notifications.

Given the resource restrictions I mentioned on the previous post, I decided to setup a dual purpose node, serving both as a Maestro client and as as Maestro broker. The virtual machine I setup for this purpose has only 1 vCPU and 512 Mb of RAM. At first sight is looks pretty small, but it should be sufficient for running both of the tasks allocated for it.

To accommodate for these resources, I decided to deploy Apache Artemis as the Maestro Broker. One the cool things about Artemis is that it can run with ridiculously small amount of memory. The low scale of the Maestro Broker means that I can go pretty aggressive with the resource limitations and, as such, I decided go with as little as 32Mb of heap.

The first step to achieve this, is to adjust the artemis command-line for the download package. In its default setting, the heap is set to vary between 512Mb to 1Gb, which would exceed the capacity of the virtual machine I configured. This script runs the code that creates the broker instance, therefore, it’s not necessary to waste too much time tweaking it. Simply adjusting it for creating the broker instance should be sufficient.

This setting is adjusted on ${install.directory}/bin/artemis. The line to be changes is the one with JAVA_ARGS. I have adjusted the heap to a fixed size of 256Mb.

# Set Defaults Properties
JAVA_ARGS="-XX:+UseParallelGC -XX:+AggressiveOpts -Xms256M -Xmx256M"
CLASSPATH="$ARTEMIS_HOME/lib/artemis-boot.jar"

# Set Defaults Properties

JAVA_ARGS="-XX:+UseParallelGC -XX:+AggressiveOpts -Xms256M -Xmx256M"

CLASSPATH="$ARTEMIS_HOME/lib/artemis-boot.jar"

With that setting in place, the instance can be created. For example:

./artemis create --nio --allow-anonymous --http-host 0.0.0.0 --user admin --password "admin" --role amq /home/maestro/apache-artemis-instance

1	./artemis create --nio --allow-anonymous --http-host 0.0.0.0 --user admin --password "admin" --role amq /home/maestro/apache-artemis-instance

To start tweaking the instance, the java heap memory for the instance can be adjusted. That is done on the ${instance.directory}/etc/artemis.profile. Once again, the line to be adjusted is the one with JAVA_ARGS:

JAVA_ARGS=" -XX:+PrintClassHistogram -XX:+UseG1GC -XX:+AggressiveOpts -Xms32M -Xmx32M -Dhawtio.realm=activemq -Dhawtio.offline="true" -Dhawtio.role=amq -Dhawtio.rolePrincipalClasses=org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal -Djolokia.policyLocation=${ARTEMIS_INSTANCE_ETC_URI}jolokia-access.xml"

JAVA_ARGS=" -XX:+PrintClassHistogram -XX:+UseG1GC -XX:+AggressiveOpts -Xms32M -Xmx32M -Dhawtio.realm=activemq -Dhawtio.offline="true" -Dhawtio.role=amq -Dhawtio.rolePrincipalClasses=org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal -Djolokia.policyLocation=${ARTEMIS_INSTANCE_ETC_URI}jolokia-access.xml"

That alone should be sufficient for the very low scale of the Maestro Broker. However, given the very low memory settings, it is a good idea to disable a couple of extra things: the console and unused acceptors.

To disable the console we can modify the ${instance.directory}/etc/bootstrap.xml. The trick is to remove (or comment) the web element and all its contents. This is the content that should be removed:

<web bind="http://0.0.0.0:8161" path="web">
<app url="activemq-branding" war="activemq-branding.war"/>
<app url="artemis-plugin" war="artemis-plugin.war"/>
<app url="console" war="console.war"/>
</web>

</web>

Then, the last trick is to leave only the acceptors that are intended to be used in this broker instance. Since Maestro broker relies on MQTT, that’s the only one that can be left there. The acceptors configuration for Apache Artemis should look like this:

<acceptors>
<acceptor name="mqtt">tcp://0.0.0.0:1884?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=MQTT;useEpoll=true</acceptor>
</acceptors>

<acceptor name="mqtt">tcp://0.0.0.0:1884?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=MQTT;useEpoll=true</acceptor>

</acceptors>

After that, the broker can be started. If you are unsure about the configuration, you can check the command lines using ps or jcmd.

In addition to these changes, the Artemis documentation mentions a lot of other tunnables that can be adjusted. For such a slow scale broker as this, I did not find the reason to adjust anything else other than what is mentioned here.

Minicloud: the Free OpenPower cloud

I recently got access to the Minicloud, a Free OpenPower Cloud hosted by Unicamp, one of the most prominent universities in Brazil. This cloud is a good way to ensure that your application provides good support for the Power architecture.

Having access to this cloud, I can help ensure that some of the projects I am interested about can be ran and supported on this architecture. Additionally, it offers me a Power environment to test the messaging-related packages I maintain on Fedora and CentOS – entirely on my free time.

To start with this cloud, I decided to go big and deploy Maestro there. Having only 4Gb, it means that I need to adjust very well the resources of my VMs so that I can maximize the usability of the cloud environment. I will write some posts about this.

The other project that I am interested in playing with in the Minicloud is Paho C, the MQTT client from Eclipse Foundation. As the Fedora maintainer for the package, having access to this environment means that I have a personal environment where I can ensure the quality of the packages.

Making it easy to play with Maestro

Maestro is a collection of tools that can be used for distributed performance testing. It is oriented towards testing of messaging oriented middleware (MOM), such as Apache ActiveMQ and its successor Apache Artemis , but can be extended to other performance test use cases.

Because Maestro forms a test cluster, it requires several components to run and deploying it can be tricky.

In order to simplify the development process as well as simplify it for users hoping to play with and run local tests, I created a set of containers (handled by Docker Compose) that can deploy a Maestro Test Cluster for local testing.

The process is extremely simple and you can go from 0 to the actual test execution in 3 commands (supposing docker-compose is already installed on your system):

1. Build and run the containers:

docker-compose -f docker-artemis-compose.yml up -d

1	docker-compose -f docker-artemis-compose.yml up -d

2.Run the client:

docker start -a -i maestro-client

1	docker start -a -i maestro-client

3. Execute a test (within the client):

cd /opt/maestro/maestro-test-scripts/scripts/groovy/singlepoint
groovy FixedRateTest.groovy /maestro/reports/

1 2	cd /opt/maestro/maestro-test-scripts/scripts/groovy/singlepoint groovy FixedRateTest.groovy /maestro/reports/

After the test is completed, just point the browser to http://localhost:8000/ to read the report and view the details.

Additional details about this, such as how to tweak the tests or use a different messaging project, are available in this document within the Maestro Java source code.

Garmin Connect IQ on Fedora

Garmin Connect IQ is the SDK used to build applications and widgets for Garmin Devices such as the awesome Garmin Vivoactive 3.

Unfortunately, the package does not work on Fedora out of the box. In order to make it run, you have to take the following steps:

1. Install Gnucash (it is required because there’s no package for a compatible webkitgtk):

sudo dnf install gnucash.

1	sudo dnf install gnucash.

2. Export the path to the webkit library required (you might want to leave this on your shell profile or similar):

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64/gnucash

1	export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64/gnucash

3. Install libjpeg8.

3.1 (Update) Enable this COPR and install libjpeg8 from there.

~~3.1 If you are using Fedora 27, you can install this RPM~~

~~3.2 If you are using Fedora 26, luck is on your side and you can just use the package from this COPR.~~

GDB – A lot more than you knew

This talk covers a lot of interesting things about GDB.

How to find the IP address of Linux guests

There’s a lot of examples in the internet showing how you can obtain the IP address of a libvirt guest virtual machine. Most of the examples show how to do that when the network is using NAT forwarding (aka “Virtual Networks”).

However, how to do that if your setup is using Bridged Networking and your guests receive an IP address from an external DHCP server?

One idea is to install the read the DHCP leases file straight from the guest FS. You can do that by using the libguestfs-tools to cat the file within the image. For RHEL 6 (and CentOS 6 and similar distributions) the process is similar to this:

virt-cat <domain-name> /var/lib/dhclient/dhclient-eth1.leases | tail -17 | grep "fixed-address" | sed 's/;//' | awk '{ print $2 }'

1	virt-cat <domain-name> /var/lib/dhclient/dhclient-eth1.leases \| tail -17 \| grep "fixed-address" \| sed 's/;//' \| awk '{ print $2 }'

For RHEL 7 (as well as CentOS 7 and similar) the process is slightly different due to the way NetworkManager stores the leases file:

leases_file=$(virt-ls <domain-name> /var/lib/NetworkManager/ | grep lease)
virt-cat <domain-name> /var/lib/NetworkManager/$leases_file | tail -17 | grep "fixed-address" | sed 's/;//' | awk '{ print $2 }'

1 2	leases_file=$(virt-ls <domain-name> /var/lib/NetworkManager/ \| grep lease) virt-cat <domain-name> /var/lib/NetworkManager/$leases_file \| tail -17 \| grep "fixed-address" \| sed 's/;//' \| awk '{ print $2 }'

In this Gist there’s a modified script I used to update the IP address of all guests in one shot.