casjay/JenkinsSetup.md

🚀 JenkinsSetup — Multi-Arch Jenkins + Incus Build System

A complete Jenkins build system setup using Incus containers, designed for seamless
cross-architecture CI pipelines across two hosts: amd64.us (x86_64) and arm64.us (aarch64).
Supports Docker, Podman, cross-compilation, multi-registry Docker pushes, and
dynamic configuration driven from Git metadata.

🌐 Overview

Component	Host	Architecture	Purpose
Jenkins Main	casjay.cc	x86_64	Central Jenkins Controller
Build Node	amd64.us	amd64 / x86_64	Docker/Podman agents, cross-agent
Build Node	arm64.us	arm64 / aarch64	Docker/Podman agents, cross-agent

🖥️ Main Node Setup (`casjay.cc`)

0. Host Configuration

Ensure required cgroup settings for nested containers:

if grep -q '^GRUB_CMDLINE_LINUX=' /etc/default/grub; then
sed -i 's/^GRUB_CMDLINE_LINUX="*/GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0 cgroup_enable=cpuset cgroup_enable=memory swapaccount=1 /' /etc/default/grub
else
echo 'GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0 cgroup_enable=cpuset cgroup_enable=memory swapaccount=1"' >> /etc/default/grub
fi

grub2-mkconfig -o /boot/grub2/grub.cfg

for grub_efi in $(ls -A /boot/efi/EFI/*/grub.cfg 2>/dev/null); do
if [ -n "$grub_efi" ] && [ -f "$grub_efi" ]; then
grub2-mkconfig -o "$grub_efi"
fi
done

reboot

stat -fc %T /sys/fs/cgroup 2>/dev/null| grep -q "cgroup2fs" && echo "cgroup2fs is enabled" || echo "cgroup2fs is disabled"

1. Install Jenkins & Dependencies

dnf install -y java-17-openjdk git jenkins

systemctl enable --now jenkins

2. Generate SSH Key for Agent Access

sudo -u jenkins ssh-keygen -t ed25519 -f /var/lib/jenkins/.ssh/id_ed25519 -N ""

3. Set Global Environment Variables

Configure under Manage Jenkins → Configure System → Global properties → Environment variables: http://casjay.cc/configure

Variable	Example Value	Description
`REGISTRIES`	`docker.io ghcr.io dockersrc.us`	List of Docker registries
`REGISTERY_ORG_USER`	`casjaysdev`	Default organization/user prefix
`ORG_PREFIX_DOCKER_IO`	`casjaysdev`	Override for docker.io
`ORG_PREFIX_GHCR_IO`	`casjaysdev`	Override for ghcr.io
`ORG_PREFIX_DOCKERSRC_US`	`casjaysdev`	Override for dockersrc.us
`TAG_LATEST`	`latest`	“latest” tag
`IMAGE_AUTHORS`	`CasjaysDev CI/CD <ci@casjay.cc>`	OCI image authors metadata
`IMAGE_LICENSES`	`WTFPL`	OCI image license metadata

4. Add SSH Credential in Jenkins UI

Navigate to Manage Jenkins > Credentials > Global: https://casjay.cc/credentials/store/system/domain/_/
Add credential:
- Kind: SSH Username with Private Key
- Username: jenkins
- Private Key: Paste content of /var/lib/jenkins/.ssh/id_ed25519
- ID: jenkins-ssh

Use settingg

https://casjay.cc/user/administrator/security/

Install Plugins

https://casjay.cc/manage/pluginManager/available

Plugin list

docker-plugin
docker-workflow 
docker-commons

🖧 Build Node Setup

For `amd64.us` (amd64 / x86_64)

0. Host Configuration

Ensure required cgroup settings for nested containers:

if grep -q '^GRUB_CMDLINE_LINUX=' /etc/default/grub; then
sed -i 's/^GRUB_CMDLINE_LINUX="*/GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0 cgroup_enable=cpuset cgroup_enable=memory swapaccount=1 /' /etc/default/grub
else
echo 'GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0 cgroup_enable=cpuset cgroup_enable=memory swapaccount=1"' >> /etc/default/grub
fi

grub2-mkconfig -o /boot/grub2/grub.cfg

for grub_efi in $(ls -A /boot/efi/EFI/*/grub.cfg 2>/dev/null); do
if [ -n "$grub_efi" ] && [ -f "$grub_efi" ]; then
grub2-mkconfig -o "$grub_efi"
fi
done

reboot

stat -fc %T /sys/fs/cgroup 2>/dev/null| grep -q "cgroup2fs" && echo "cgroup2fs is enabled" || echo "cgroup2fs is disabled"

1. System Preparation & SSH Setup

dnf install -y java-17-openjdk git openssh-server

systemctl enable --now sshd

useradd --system --create-home --shell /bin/bash jenkins

mkdir -p /home/jenkins/.ssh

chmod 700 /home/jenkins/.ssh

chown -Rf jenkins:jenkins /home/jenkins

passwd jenkins

passwd -u jenkins

2. Copy SSH Key from Main

ssh -l root casjay.cc 'ssh-copy-id -i /var/lib/jenkins/.ssh/id_ed25519.pub jenkins@amd64.us'

usermod -aG docker jenkins

3. Container Agents

ALPINE_VERSION="${ALPINE_VERSION:-3.20}"

DEBIAN_VERSION="${DEBIAN_VERSION:-bookworm}"

Docker Agent (native):

incus launch images:alpine/${ALPINE_VERSION} docker-agent

incus config set docker-agent security.nesting true

incus config set docker-agent security.privileged true

incus exec docker-agent -- apk add --no-cache docker openrc

incus exec docker-agent -- rc-update add docker default

incus exec docker-agent -- service docker start

Podman Agent:

incus launch images:alpine/${ALPINE_VERSION} podman-agent

incus config set podman-agent security.nesting true

incus exec podman-agent -- apk add --no-cache podman fuse-overlayfs shadow

incus exec podman-agent -- rc-service cgroups start

Cross Agent:

incus launch images:debian/${DEBIAN_VERSION} cross-agent

incus exec cross-agent -- bash -c "apt-get update && apt-get update -yy && apt-get dist-upgrade -yy"

incus exec cross-agent -- bash -c "apt-get install -y build-essential clang cmake ninja-build gcc-mingw-w64 g++-mingw-w64 gcc-arm-linux-gnueabihf g++-arm-linux-gnueabihf gcc-aarch64-linux-gnu g++-aarch64-linux-gnu binfmt-support qemu-user-static rustc cargo golang-go python3 python3-pip ruby ruby-dev openjdk-17-jdk maven gradle mono-complete php-cli composer"

For `arm64.us` (aarch64 / arm64)

0. Host Configuration

Ensure required cgroup settings for nested containers:

if grep -q '^GRUB_CMDLINE_LINUX=' /etc/default/grub; then
sed -i 's/^GRUB_CMDLINE_LINUX="*/GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0 cgroup_enable=cpuset cgroup_enable=memory swapaccount=1 /' /etc/default/grub
else
echo 'GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0 cgroup_enable=cpuset cgroup_enable=memory swapaccount=1"' >> /etc/default/grub
fi

grub2-mkconfig -o /boot/grub2/grub.cfg

for grub_efi in $(ls -A /boot/efi/EFI/*/grub.cfg 2>/dev/null); do
if [ -n "$grub_efi" ] && [ -f "$grub_efi" ]; then
grub2-mkconfig -o "$grub_efi"
fi
done

reboot

stat -fc %T /sys/fs/cgroup 2>/dev/null| grep -q "cgroup2fs" && echo "cgroup2fs is enabled" || echo "cgroup2fs is disabled"

1. System Preparation & SSH Setup

dnf install -y java-17-openjdk git openssh-server

systemctl enable --now sshd

useradd --system --create-home --shell /bin/bash jenkins

mkdir -p /home/jenkins/.ssh

chmod 700 /home/jenkins/.ssh

chown -Rf jenkins:jenkins /home/jenkins

passwd jenkins

passwd -u jenkins

usermod -aG docker jenkins

curl -L -o install_plugins.sh https://raw.githubusercontent.com/jenkinsci/plugin-installation-manager-tool/master/install-plugins.sh -O /usr/local/jenkins_plugins && \
chmod +x /usr/local/jenkins_plugins

2. Copy SSH Key from Main

ssh -l root casjay.cc 'ssh-copy-id -i /var/lib/jenkins/.ssh/id_ed25519.pub jenkins@arm64.us'

3. Container Agents

ALPINE_VERSION="${ALPINE_VERSION:-3.20}"

DEBIAN_VERSION="${DEBIAN_VERSION:-bookworm}"

Docker Agent (native):

incus launch images:alpine/${ALPINE_VERSION} docker-agent

incus config set docker-agent security.nesting true

incus config set docker-agent security.privileged true

incus exec docker-agent -- apk add --no-cache docker openrc

incus exec docker-agent -- rc-update add docker default

incus exec docker-agent -- service docker start

Podman Agent:

incus launch images:alpine/${ALPINE_VERSION} podman-agent

incus config set podman-agent security.nesting true

incus exec podman-agent -- apk add --no-cache podman fuse-overlayfs shadow

incus exec podman-agent -- rc-service cgroups start

Cross Agent:

incus launch images:debian/${DEBIAN_VERSION} cross-agent

incus exec cross-agent -- bash -c "apt-get update && apt-get update -yy && apt-get dist-upgrade -yy"

incus exec cross-agent -- bash -c "apt-get install -y build-essential clang cmake ninja-build gcc-mingw-w64 g++-mingw-w64 gcc-arm-linux-gnueabihf g++-arm-linux-gnueabihf gcc-aarch64-linux-gnu g++-aarch64-linux-gnu binfmt-support qemu-user-static rustc cargo golang-go python3 python3-pip ruby ruby-dev openjdk-17-jdk maven gradle mono-complete php-cli composer"

🐳 Dockerfile for Cross-Compiling Base

FROM debian:bookworm

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential clang cmake ninja-build \
    gcc-mingw-w64 g++-mingw-w64 \
    gcc-arm-linux-gnueabihf g++-arm-linux-gnueabihf \
    gcc-aarch64-linux-gnu g++-aarch64-linux-gnu \
    qemu-user-static binfmt-support \
    rustc cargo golang-go python3 python3-pip \
    ruby ruby-dev openjdk-17-jdk maven \
    gradle dotnet-sdk-7.0 mono-complete \
    php-cli composer zig \
  && apt-get clean && rm -rf /var/lib/apt/lists/*

WORKDIR /build
CMD ["/bin/bash"]

🧪 Jenkinsfile for Cross-Compiling

pipeline {
  agent any

  environment {
    IMAGE_REPO = ""
    TAG_LATEST = "${env.TAG_LATEST ?: 'latest'}"
    TAG_DATE = "${env.TAG_DATE ?: new Date().format('yyMM')}"
    TAG_COMMIT = "${env.TAG_COMMIT ?: (env.GIT_COMMIT?.substring(0,7) ?: 'dev')}"
    IMAGE_LICENSES = "${env.IMAGE_LICENSES ?: 'WTFPL'}"
    IMAGE_AUTHORS = "${env.IMAGE_AUTHORS ?: 'Jason Hempstead <dev@casjay.cc>'}"
    REGISTRIES = "${env.REGISTRIES ?: 'docker.io ghcr.io dockersrc.us'}"
    REGISTERY_ORG_USER = "${env.REGISTERY_ORG_USER ?: 'casjaysdev'}"
    ORG_PREFIX_GHCR_IO = "${env.ORG_PREFIX_GHCR_IO ?: 'casjaysdev'}"
    ORG_PREFIX_DOCKER_IO = "${env.ORG_PREFIX_DOCKER_IO ?: 'casjaysdev'}"
    ORG_PREFIX_DOCKERSRC_US = "${env.ORG_PREFIX_DOCKERSRC_US ?: 'library'}"
  }

  stages {
    stage('Checkout') {
      steps {
        checkout scm
      }
    }

    stage('Set Dynamic Repo and Org/User') {
      steps {
        script {
          def gitUrl = env.GIT_URL ?: sh(script: "git config --get remote.origin.url", returnStdout: true).trim()
          def repoName = gitUrl.tokenize('/').last().replace('.git', '')

          def orgUser = ''
          if (gitUrl.contains('@')) {
            orgUser = gitUrl.tokenize(':').last().tokenize('/')[0]
          } else if (gitUrl.contains('://')) {
            orgUser = gitUrl.tokenize('/')[3]
          }

          env.IMAGE_REPO = env.IMAGE_REPO ?: repoName
          env.DEFAULT_ORG_USER = env.REGISTERY_ORG_USER ?: orgUser

          echo "Determined IMAGE_REPO = ${env.IMAGE_REPO}"
          echo "Using DEFAULT_ORG_USER = ${env.DEFAULT_ORG_USER}"
        }
      }
    }

    stage('Build, Push & Manifest') {
      steps {
        script {
          def registries = env.REGISTRIES.split()
          def tags = [env.TAG_LATEST, env.TAG_DATE, env.TAG_COMMIT]

          registries.each { registry ->
            def envVarName = "ORG_PREFIX_" + registry.replaceAll(/[\.\-]/, "_").toUpperCase()
            def prefix = env[envVarName] ?: env.DEFAULT_ORG_USER

            if (!prefix) {
              error("No org/user prefix found for registry '${registry}' and no REGISTERY_ORG_USER set.")
            }

            tags.each { tag ->
              def fullTagAmd64 = "${registry}/${prefix}/${env.IMAGE_REPO}:${tag}-amd64"
              def fullTagArm64 = "${registry}/${prefix}/${env.IMAGE_REPO}:${tag}-aarch64"

              echo "Building and pushing ${fullTagAmd64} and ${fullTagArm64}"

              sh """
                docker build --platform linux/amd64 -t ${fullTagAmd64} .
                docker push ${fullTagAmd64}

                docker build --platform linux/arm64 -t ${fullTagArm64} .
                docker push ${fullTagArm64}
              """

              def wikiUrl = ""
              if (gitUrl) {
                def baseGitUrl = gitUrl.replaceFirst(/\.git$/, '')
                wikiUrl = baseGitUrl + "/wiki"
              }

              sh """
                docker manifest create ${registry}/${prefix}/${env.IMAGE_REPO}:${tag} \
                  ${fullTagAmd64} \
                  ${fullTagArm64}
              """

              sh """
                docker manifest annotate ${registry}/${prefix}/${env.IMAGE_REPO}:${tag} ${fullTagAmd64} \
                  --os linux \
                  --arch amd64 \
                  --annotation org.opencontainers.image.title="${env.IMAGE_REPO}" \
                  --annotation org.opencontainers.image.version="${tag}" \
                  --annotation org.opencontainers.image.revision="${env.TAG_COMMIT}" \
                  --annotation org.opencontainers.image.created="${new Date().format('yyyy-MM-dd\\'T\\'HH:mm:ssXXX')}" \
                  --annotation org.opencontainers.image.authors="${env.IMAGE_AUTHORS}" \
                  --annotation org.opencontainers.image.documentation="${wikiUrl}" \
                  --annotation org.opencontainers.image.url="${gitUrl ?: ''}" \
                  --annotation org.opencontainers.image.source="${gitUrl ?: ''}" \
                  --annotation org.opencontainers.image.licenses="${env.IMAGE_LICENSES}"
              """

              sh """
                docker manifest annotate ${registry}/${prefix}/${env.IMAGE_REPO}:${tag} ${fullTagArm64} \
                  --os linux \
                  --arch arm64 \
                  --annotation org.opencontainers.image.title="${env.IMAGE_REPO}" \
                  --annotation org.opencontainers.image.version="${tag}" \
                  --annotation org.opencontainers.image.revision="${env.TAG_COMMIT}" \
                  --annotation org.opencontainers.image.created="${new Date().format('yyyy-MM-dd\\'T\\'HH:mm:ssXXX')}" \
                  --annotation org.opencontainers.image.authors="${env.IMAGE_AUTHORS}" \
                  --annotation org.opencontainers.image.documentation="${wikiUrl}" \
                  --annotation org.opencontainers.image.url="${gitUrl ?: ''}" \
                  --annotation org.opencontainers.image.source="${gitUrl ?: ''}" \
                  --annotation org.opencontainers.image.licenses="${env.IMAGE_LICENSES}"
              """

              sh "docker manifest push ${registry}/${prefix}/${env.IMAGE_REPO}:${tag}"
            }
          }
        }
      }
    }
  }
}

🧪 Jenkins Pipeline: Dynamic Multi-Registry, Multi-Arch Build & Push with OCI Labels

pipeline {
  agent any

  environment {
    IMAGE_REPO = ""
    TAG_LATEST = "${env.TAG_LATEST ?: 'latest'}"
    TAG_DATE = "${env.TAG_DATE ?: new Date().format('yyMM')}"
    TAG_COMMIT = "${env.TAG_COMMIT ?: (env.GIT_COMMIT?.substring(0,7) ?: 'dev')}"
    IMAGE_LICENSES = "${env.IMAGE_LICENSES ?: 'WTFPL'}"
    IMAGE_AUTHORS = "${env.IMAGE_AUTHORS ?: 'Jason Hempstead <dev@casjay.cc>'}"
    REGISTRIES = "${env.REGISTRIES ?: 'docker.io ghcr.io dockersrc.us'}"
    REGISTERY_ORG_USER = "${env.REGISTERY_ORG_USER ?: 'casjaysdevdocker'}"
  }

  stages {
    stage('Checkout') {
      steps {
        checkout scm
      }
    }

    stage('Set Dynamic Repo and Org/User') {
      steps {
        script {
          def gitUrl = env.GIT_URL ?: sh(script: "git config --get remote.origin.url", returnStdout: true).trim()
          def repoName = gitUrl.tokenize('/').last().replace('.git', '')

          def orgUser = ''
          if (gitUrl.contains('@')) {
            orgUser = gitUrl.tokenize(':').last().tokenize('/')[0]
          } else if (gitUrl.contains('://')) {
            orgUser = gitUrl.tokenize('/')[3]
          }

          env.IMAGE_REPO = env.IMAGE_REPO ?: repoName
          env.DEFAULT_ORG_USER = env.REGISTERY_ORG_USER ?: orgUser

          echo "Determined IMAGE_REPO = ${env.IMAGE_REPO}"
          echo "Using DEFAULT_ORG_USER = ${env.DEFAULT_ORG_USER}"
        }
      }
    }

    stage('Build, Push & Manifest') {
      steps {
        script {
          def registries = env.REGISTRIES.split()
          def tags = [env.TAG_LATEST, env.TAG_DATE, env.TAG_COMMIT]

          registries.each { registry ->
            def envVarName = "ORG_PREFIX_" + registry.replaceAll(/[\.\-]/, "_").toUpperCase()
            def prefix = env[envVarName] ?: env.DEFAULT_ORG_USER

            if (!prefix) {
              error("No org/user prefix found for registry '${registry}' and no REGISTERY_ORG_USER set.")
            }

            tags.each { tag ->
              def fullTagAmd64 = "${registry}/${prefix}/${env.IMAGE_REPO}:${tag}-amd64"
              def fullTagArm64 = "${registry}/${prefix}/${env.IMAGE_REPO}:${tag}-aarch64"

              echo "Building and pushing ${fullTagAmd64} and ${fullTagArm64}"

              sh """
                docker build --platform linux/amd64 -t ${fullTagAmd64} .
                docker push ${fullTagAmd64}

                docker build --platform linux/arm64 -t ${fullTagArm64} .
                docker push ${fullTagArm64}
              """

              def wikiUrl = ""
              if (gitUrl) {
                def baseGitUrl = gitUrl.replaceFirst(/\.git$/, '')
                wikiUrl = baseGitUrl + "/wiki"
              }

              sh """
                docker manifest create ${registry}/${prefix}/${env.IMAGE_REPO}:${tag} \
                  ${fullTagAmd64} \
                  ${fullTagArm64}
              """

              sh """
                docker manifest annotate ${registry}/${prefix}/${env.IMAGE_REPO}:${tag} ${fullTagAmd64} \
                  --os linux \
                  --arch amd64 \
                  --annotation org.opencontainers.image.title="${env.IMAGE_REPO}" \
                  --annotation org.opencontainers.image.version="${tag}" \
                  --annotation org.opencontainers.image.revision="${env.TAG_COMMIT}" \
                  --annotation org.opencontainers.image.created="${new Date().format('yyyy-MM-dd\\'T\\'HH:mm:ssXXX')}" \
                  --annotation org.opencontainers.image.authors="${env.IMAGE_AUTHORS}" \
                  --annotation org.opencontainers.image.documentation="${wikiUrl}" \
                  --annotation org.opencontainers.image.url="${gitUrl ?: ''}" \
                  --annotation org.opencontainers.image.source="${gitUrl ?: ''}" \
                  --annotation org.opencontainers.image.licenses="${env.IMAGE_LICENSES}"
              """

              sh """
                docker manifest annotate ${registry}/${prefix}/${env.IMAGE_REPO}:${tag} ${fullTagArm64} \
                  --os linux \
                  --arch arm64 \
                  --annotation org.opencontainers.image.title="${env.IMAGE_REPO}" \
                  --annotation org.opencontainers.image.version="${tag}" \
                  --annotation org.opencontainers.image.revision="${env.TAG_COMMIT}" \
                  --annotation org.opencontainers.image.created="${new Date().format('yyyy-MM-dd\\'T\\'HH:mm:ssXXX')}" \
                  --annotation org.opencontainers.image.authors="${env.IMAGE_AUTHORS}" \
                  --annotation org.opencontainers.image.documentation="${wikiUrl}" \
                  --annotation org.opencontainers.image.url="${gitUrl ?: ''}" \
                  --annotation org.opencontainers.image.source="${gitUrl ?: ''}" \
                  --annotation org.opencontainers.image.licenses="${env.IMAGE_LICENSES}"
              """

              sh "docker manifest push ${registry}/${prefix}/${env.IMAGE_REPO}:${tag}"
            }
          }
        }
      }
    }
  }
}

casjay/JenkinsSetup.md

Select an option

No results found

Select an option

No results found

🚀 JenkinsSetup — Multi-Arch Jenkins + Incus Build System

🌐 Overview

🖥️ Main Node Setup (`casjay.cc`)

0. Host Configuration

1. Install Jenkins & Dependencies

2. Generate SSH Key for Agent Access

3. Set Global Environment Variables

4. Add SSH Credential in Jenkins UI

Use settingg

Install Plugins

Plugin list

🖧 Build Node Setup

For `amd64.us` (amd64 / x86_64)

0. Host Configuration

1. System Preparation & SSH Setup

2. Copy SSH Key from Main

3. Container Agents

For `arm64.us` (aarch64 / arm64)

0. Host Configuration

1. System Preparation & SSH Setup

2. Copy SSH Key from Main

3. Container Agents

🐳 Dockerfile for Cross-Compiling Base

🧪 Jenkinsfile for Cross-Compiling

🧪 Jenkins Pipeline: Dynamic Multi-Registry, Multi-Arch Build & Push with OCI Labels

casjay/JenkinsSetup.md

🚀 JenkinsSetup — Multi-Arch Jenkins + Incus Build System

🌐 Overview

🖥️ Main Node Setup (casjay.cc)

0. Host Configuration

1. Install Jenkins & Dependencies

2. Generate SSH Key for Agent Access

3. Set Global Environment Variables

4. Add SSH Credential in Jenkins UI

Use settingg

Install Plugins

Plugin list

🖧 Build Node Setup

For amd64.us (amd64 / x86_64)

0. Host Configuration

1. System Preparation & SSH Setup

2. Copy SSH Key from Main

3. Container Agents

For arm64.us (aarch64 / arm64)

0. Host Configuration

1. System Preparation & SSH Setup

2. Copy SSH Key from Main

3. Container Agents

🐳 Dockerfile for Cross-Compiling Base

🧪 Jenkinsfile for Cross-Compiling

🧪 Jenkins Pipeline: Dynamic Multi-Registry, Multi-Arch Build & Push with OCI Labels

🖥️ Main Node Setup (`casjay.cc`)

For `amd64.us` (amd64 / x86_64)

For `arm64.us` (aarch64 / arm64)