Adding Application Test Suite to a CI/CD pipeline | CLEVR

Published 07-06-2019, last updated 06-07-2023 2 min read

There are many great resources on the topic so just a short explanation: Continuous integration refers to the practice of creating a build as new code is committed and running automated tests against that build to verify that everything works correctly. Continuous delivery is the ability to automatically release any new code, usually with a single click.

Common stages in the CI/CD process, borrowed from plutora

In a way continuous integration is the basis for continuous delivery by 1B requiring that the application can be automatically deployed in order to run test which in turn 2B provide evidence that new code does not break the application. The primary focus of CI/CD is automation.

How does ATS fit in CI/CD?

The Application Test Suite (ATS) is a low-code tool for automated testing on GUI level, specialized for Mendix applications. As such it can provide great value to development teams by frequently running a so called regression test suite. ATS has two options for setting this up: 1B schedules and 2B the ATS API.

Schedules is a light-weight alternative for CI/CD that is meant for small companies. It requires minimal set up and no external infrastructure. The scheduling and configuration is all done on the ATS platform itself and alerts can be sent when the tests finish. On the other hand, the ATS API offers a much more versatile approach for automating the testing process. As such it is geared towards large companies that have multiple apps and require more control and a centralized place to manage and monitor the whole CI/CD pipeline.

Smart Digital Factory

To help companies that want to incorporate CI/CD in the development process Mansystems has launched the concept of the Smart Digital Factory which comes with a set of tools specifically geared to Mendix development such as ACR, ATS, AMS and APD. The toolkit includes a CI/CD pipeline solution that comes with out of the box support for many Mendix related task such as creating deployment packages, creating and restoring backups and runningautomated tests with ATS. The toolkit is great for companies that are committed to the Mendix platform and do not want to invest resources to set up and maintain their own infrastructure.

ATS Jenkins plugin

That all said, as part of the dev team for ATS, I am aware that many users of ATS already have existing infrastructure for CI/CD. So in this blog post I want to show how to integrate ATS into your existing pipeline, with specific focus on Jenkins. I choose Jenkins because of its ubiquity.

CICD_ tools_market_ shareCI/CD tools market share according to datanyze

Although Jenkins is very popular it can be quite overwhelming and not easy to extend especially for people with no coding experience. In order to enable everyone to use ATS in combination with Jenkins I decided to write a custom Jenkins plugin. On Github you can find both the source code and the plugin package.

In order to run an ATS test case or test suite with this plugin you only need to add a single build step to Jenkins and specify the application id, an API token and a job template ID. No additional configuration and no explicit HTTP requests or XML parsing.

ATS tests can be run with a single build step without the need to explicitly trigger HTTP requests or parse XML.

Optionally, failed tests can be automatically rerun. Forcing tests to fail at least twice is a good strategy to deal with flakiness. To understand why, I recommend reading this great post. When executing an ATS build step, if the tests fail the Jenkins build will also be marked as failed. Additionally, ATS will log information to the console, which relates the build to jobs in ATS.

The Jenkins log file contains id's of ATS jobs that can be used to look up the respective job in ATS for all the details

Installation

This plugin is installed manually. To do this, go to your Jenkins dashboard and navigate to Jenkins_manage_Jenkins

Under Advanced you should see the option to upload a file. Click Choose file and then Upload.

Jenkins_Upload_Plugin

Now you can add the build step Run ATS tests to your Jenkins Build.

Summary

The motivation for this blog post was twofold. First, I wanted to develop a Jenkins plugin that is simple to use and requires no coding skills. The plugin package is free to use for everyone. Second, by making the plugin open source I hope to provide a starting point and guide for people looking to integrate ATS into other CI/CD tools.

Thanks for taking the time to read this blog post, and I hope it helps you develop great apps!