Continuous Integration / Gitlab Cx

stages:
	- build
	- test

build:
	stage: build
	script:
		- export NUM_CORES=$(nproc --all)
		- mkdir $CI_PROJECT_DIR/build
		- cd $CI_PROJECT_DIR/build
		- cmake ..
		- make -j $NUM_CORES
	tags:
		- testcluster
	artifacts:
		paths:
			- build

test:
	stage: test
	variables: 
		SLURM_TIMELIMIT: 30
	script:
		- cd $CI_PROJECT_DIR/build
		- ./test
	tags:
		- testcluster

variables:
	SLURM_CONSTRAINT: "hwperf"

stages:
	- prepare
	- build
	- test

prepare:
	stage: prepare
	script:
		- echo "Preparing on frontend node..."
	variables:
		NO_SLURM_SUBMIT: 1
	tags:
		- testcluster

build:
	stage: build
	script:
		- export NUM_CORES=$(nproc --all)
		- mkdir $CI_PROJECT_DIR/build
		- cd $CI_PROJECT_DIR/build
		- cmake ..
		- make -j $NUM_CORES
	tags:
		- testcluster
	artifacts:
		paths:
			- build

test:
	stage: test
	variables: 
		SLURM_TIMELIMIT: 30
	script:
		- cd $CI_PROJECT_DIR/build
		- ./test
	tags:
		- testcluster

variables:
	SLURM_NODELIST: broadep2
	SLURM_TIMELIMIT: 10

stages:
	- build
	- test

build:
	stage: build
	script:
		- export NUM_CORES=$(nproc --all)
		- mkdir $CI_PROJECT_DIR/build
		- cd $CI_PROJECT_DIR/build
		- cmake ..
		- make -j $NUM_CORES
	tags:
		- testcluster
	artifacts:
		paths:
			- build

test:
	stage: test
	variables: 
		SLURM_TIMELIMIT: 30
	script:
		- cd $CI_PROJECT_DIR/build
		- ./test
	tags:
		- testcluster

stages:
	- build
	- benchmark

.build:
	stage: build
	script:
		- export NUM_CORES=$(nproc --all)
		- mkdir $CI_PROJECT_DIR/build
		- cd $CI_PROJECT_DIR/build
		- cmake ..
		- make -j $NUM_CORES
	tags:
		- testcluster
	variables: 
		SLURM_TIMELIMIT: 10
	artifacts:
		paths:
			- build

.benchmark:
	stage: benchmark
	variables: 
		SLURM_TIMELIMIT: 20
	script:
		- cd $CI_PROJECT_DIR/build
		- ./benchmark
	tags:
		- testcluster

# broadep2

build-broadep2:
   extends: .build
   variables:
      SLURM_NODELIST: broadep2

benchmark-broadep2:
   extends: .benchmark
   dependencies:
      - build-broadep2
   variables:
      SLURM_NODELIST: broadep2

# naples1

build-naples1:
   extends: .build
   variables:
      SLURM_NODELIST: naples1

benchmark-naples1:
   extends: .benchmark
   dependencies:
      - build-naples1
   variables:
      SLURM_NODELIST: naples1

In order to create a child pipeline we have to dynamically create a YAML file that is compatible for the Gitlab-CI system. The dynamically created file is only valid for the current Cx execution. The YAML file can be either created by a script that is part of the repository like the .ci/generate_jobs.sh script in the example below. There are different methods to create the YAML file for the child pipeline (multi-line script entry, templated job with variable overrides, …).

$ cat .ci/generate_jobs.sh
#!/bin/bash -l
# Get list of modules
MODLIST=$(module avail -t intel 2>&1 | grep -E "^intel" | awk '{print $1}')
# Alternative: Get list of idle hosts in the testcluster (requires NO_SLURM_SUBMIT=1)
#HOSTLIST=$(sinfo -t idle -h --partition=work -o "%n %t" | grep "idle" | cut -d ' ' -f 1)
for MOD in ${MODLIST}; do
  MODVER=${MOD/\//-}   # replace '/' in module name with '-' for job name
  cat << EOF
build-$MODVER:
  stage: build
  variables:
    CUDA_MODULE: $MOD
  script:
    - module load "\$CUDA_MODULE"
    - make
    - ./run_tests
  tags:
    - testcluster
  EOF
done

With this script, we can generate and execute the child pipeline in the parent configuration. We use NO_SLURM_SUBMIT=1 to generate the pipeline on the frontend node. In some cases, you have to use a specific system (e.g. CUDA modules only usable on the host medusa), then just use the SLURM_NODELIST variable. We store the generated YAML file as artifact in the generator job and include it as trigger in the executor. If you want to use artifacts in the child pipeline that are created in the parent pipeline (like differently configured builds), you have to specify the variable PARENT_PIPELINE_ID=$CI_PIPELINE_ID and specify the pipeline in the child job (job -> needs -> pipeline: $PARENT_PIPELINE_ID).

generate_child_pipeline:
  stage: build
  tags:
    - testcluster
  variables:
    - NO_SLURM_SUBMIT: 1
  script: 
   - .ci/generate_jobs.sh > child-pipeline.yml
  artifacts:
    paths:
      - child-pipeline.yml

execute_child_pipeline:
  stage: test
  trigger:
    include:
      - artifact: child-pipeline.yml
        job: generate_child_pipeline
    strategy: depend
  variables:
    PARENT_PIPELINE_ID: $CI_PIPELINE_ID