How to Run Cucumber Tests using Docker in parallel


651
2 comments, 651 points
Cucumber Tests using Docker in parallel
Cucumber Tests using Docker in parallel

How to Run Cucumber Tests using Docker

It is crucial once you have an automated test suite to run it often , get feedback as quick as possible and make the most of your tests to increase your team productivity and product quality. In our previous article we covered how to run selenium cucumber tests in chromedriver headless  but in order to have that done in a proper way we need to know how to run cucumber tests using docker in parallel .

Assuming we have docker setup already done in jenkins and your local machine if not here is how your docker file should look like we are going to use jenkins pipeline scripts to kick off our tests in parallel.

Create your jenkins job

Go to Jenkins top page, select “New Job“, then choose “Build a free-style softwareproject“. This job type consists of the following elements: optional SCM, such as CVS or Subversion where your source code resides. optional triggers to control when Jenkins will perform builds.

Go to configure tab job : https://jenkins.yourjenkinshostname.com/view/{ViewName}/job/{JobName}/configure

Paste your pipeline script

stage 'Build'
node('QA-Slave') {
                try {
                   currentBuild.description="Building Image"
                    git credentialsId: 'gitkey', url: 'kit project url'
                    docker.build "docker.yourhost.com/cucumber-autotest:${env.BUILD_NUMBER}"

                }
                catch (e) {
                    currentBuild.result="FAILED"
                    throw(e)
                } finally {
                notifier.notifyBuild(env.BUILD_NUMBER, currentBuild.description, currentBuild.result)
                }
            }

stage 'Test'
def runjob(String testexecutor,String profile){

    currentBuild.description="Testing Image"
    dockerContainerName="${testexecutor}-${env.BUILD_NUMBER}"

    sh """
    ./bin/run_tests.sh ${dockerContainerName} ${"docker.yourhost.com/cucumber-autotest:${env.BUILD_NUMBER}"} -p ${profile} THREAD=${profile} -f pretty -f ReportPortal::Cucumber::Formatter -o 'output/test_report.txt';
    """
}

node('QA-Slave') {
    timeout(time: 3, unit: 'HOURS'){
    try{
            parallel 'test1': {runjob('test1','thread1')},
                     'test2': {runjob('test2','thread2')},
                     'test3': {runjob('test3','thread3')},
                     'test4': {runjob('test4','thread4')},
                     'test5': {runjob('test5','thread5')},
                     'test6': {runjob('test6','thread6')},
                     'test7': {runjob('test7','thread7')},
                     'test8': {runjob('test8', 'thread8')},
                     'test9': {runjob('test9', 'thread9')},
                     'test10':{runjob('test10', 'thread10')}

    }
    catch (e) {


    } finally {
        step([$class: 'ArtifactArchiver', artifacts: 'output/**/*', excludes: null])
        step([$class: 'JUnitResultArchiver', allowEmptyResults: true, keepLongStdio: true, testResults: 'output/*.xml'])
        dockerCleanup();
        if(notifier.getResults() >= 98){
           currentBuild.result="SUCCESS"
        sh "exit 0"
        } else{
           currentBuild.result="FAILED"
        sh "exit 1"
           throw new Exception("Pass rate lower than 98%")
        }
        deleteDir()
    }
    }
}

@NonCPS
def dockerCleanup() {
    // Inspired by
    // https://gitlab.zoopla.co.uk/devops/jenkins-libraries/blob/master/src/org/zpg/BasePipeline.groovy
    // ... but 'mvn-smoke-tests' has made us cautious about over-cleansing
    // So for now we'll do a simple `docker rmi`

    // remove the *image* we created this build
    // the container should auto-exit
    sh """
       docker rmi ${"docker.yourhost.com/cucumber-autotest:${env.BUILD_NUMBER}"};
    """
}

Lets try to understand some bits from this script:

 -f pretty -f ReportPortal::Cucumber::Formatter -o 'output/test_report.txt'

This line is to push your test results to an external report system as in this case is reportportal .

Threads : thread1 , thread2 …threadn is a cucumber profile that sores one or more cucumber tags and should be added in your cucumber yaml file.

 notifier.notifyBuild(env.BUILD_NUMBER, currentBuild.description, currentBuild.result)

You need it only if you use a notification system for your build test results , Im going to cover it in a separate article how you can send slack notifications using jenkins pipeline. I will do it separated as is a big topic.

Advertisements

In test stage you can see I used a function getresults()

    if(notifier.getResults() >= 95){
           currentBuild.result="SUCCESS"
        sh "exit 0"
        } else{
           currentBuild.result="FAILED"
        sh "exit 1"
           throw new Exception("Pass rate lower than 95%")
        }

Basically is our way to set a threshold in jenkins when to fail the build based in the test results. We automatically pass the build if fail rate is less than 2 %. I will cover how to do that in a further article.

Your run tests sh script

Inside runjob() function you’ve seem ./run_tests.sh , bellow you can see the code from it.

 

#!/bin/sh
# run_test.sh <container_name> <image_id> <features>
(
    CONTAINER_NAME=$1
    IMAGE_ID=$2
    echo "cont: " $CONTAINER_NAME;
    echo "image: " $IMAGE_ID;
    echo "args: " ${@:3};
    docker run --cap-add sys_admin  -v /dev/shm:/dev/shm -i -e BROWSER -e TEST_ENV --name $CONTAINER_NAME $IMAGE_ID -f junit -o output -f pretty ${@:3};
    exit_code=$(docker inspect -f "{{ .State.ExitCode }}" $CONTAINER_NAME)
    docker cp $CONTAINER_NAME:/code/output .;
    docker rm $CONTAINER_NAME
    echo "exit:" $exit_code
    exit $exit_code
)

Using this jenkins pipeline script you can customise your cucumber tests to run on whatever environment you like , browser , brand ( if you have a big framework that incorporates multiple internal projects.)

Now there might be voices to say that we are defining the number of threads and we don’t leave  jenkins to split the objects dynamically . You can run jobs in parallel using the pipeline script provided by Jenkins but the problem with this will run all your tests sequentially first and only on the second run will be running in parallel.

I found it a bit weird because as you are going to increase the number of tests this job will run again and again your tests sequentially in order to figure out how to split the time equally.

// in this array we'll place the jobs that we wish to run
def branches = [:]

//running the job 4 times concurrently
//the dummy parameter is for preventing mutation of the parameter before the execution of the closure.
//we have to assign it outside the closure or it will run the job multiple times with the same parameter "4"
//and jenkins will unite them into a single run of the job

for (int i = 0; i < 4; i++) {
  def index = i //if we tried to use i below, it would equal 4 in each job execution.
  branches["branch${i}"] = {
//Parameters:
//param1 : an example string parameter for the triggered job.
//dummy: a parameter used to prevent triggering the job with the same parameters value. this parameter has to accept a different value
//each time the job is triggered.
    build job: 'test_jobs', parameters: [[$class: 'StringParameterValue', name: 'param1', value:
      'test_param'], [$class: 'StringParameterValue', name:'dummy', value: "${index}"]]
    }
}
parallel branches

It’s up to you what solution you wanna choose , the top one works well for me.

If you have any queries please do not hesitate to add it in the comment bellow.

Happy testing!


Like it? Share with your friends!

651
2 comments, 651 points
Test engineer

2 Comments

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

  1. I have created a jenkins pipeline where in one of the stages I am running the scripts which runs the maven test cases then I copy that to the workspace the complete cucumber folder ,some of the cases are failing still build are marked as sucess

    1. hi , it should not be the case

      In the example I have a notified.getRresults()

      My bad I didn’t published the code for the getResults method

      here we go
      @NonCPS
      def getResults(){
      AbstractTestResultAction testResultAction = currentBuild.rawBuild.getAction(AbstractTestResultAction.class)
      def failCount = null
      def failureDiffString = null
      def totalCount = null
      def passed = null
      def passrate = null
      if (testResultAction != null) {
      failCount = testResultAction.failCount
      totalCount = testResultAction.totalCount
      passed = totalCount - failCount
      passrate = (passed/totalCount*100).toInteger()
      }

      return passrate
      }

      also if you will have this code in a separate groovy file you need to create it

      and load it in your pipeline
      notifier = null

      and inside your build after you checkout your code :
      notifier = load 'jenkins/notify.groovy'

      I will update the article with this informations