Skip to content

Instantly share code, notes, and snippets.

@gmr

gmr/0-auto-manged-chef-repo.md

Last active February 8, 2018 22:45
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save gmr/6107050 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save gmr/6107050 to your computer and use it in GitHub Desktop.
Download ZIP
This gist has files that are used to automate chef repository submodules used for roles, data bags, cookbooks and environments, ultimately uploading the maintained chef repository to chef via knife using Jenkins and Github.
Raw
0-auto-manged-chef-repo.md

The files in this gist are for having Jenkins automatically manage a chef repository using git submodules. This allows for clean, clutter free management of individual cookbooks, and individual respositories for roles, environments and data bags.

It relies on using Github (we use Github Enterprise) and Jenkins in combination with the Jenkins Github plugin to notify Jenkins when a repository has changed.

Our chef-repo directory looks something like:

chef-repo
    - cookbooks
          - Each cookbook is a git submodule managed by Jenkins
    - data_bags (git submodule managed by Jenkins)
    - environments (git submodule managed by Jenkins)
    - roles (git submodule managed by Jenkins)
    - All other normal files and directories

Each cookbook has a Jenkins job that recives a github callback when changes are pushed into it. These jobs all ove two build steps:

Raw
0-dependencies-json-notes.md

The dependencies.json file allows for more granularity than metadata.rb, specifying the remote repository and optionally the branch and revision to use. It should be placed in the root directory for each cookbook you create, but can be empty:

{}
Raw
0-dependencies.json
{
"cookbooks": {
"build-essential": {
"url": "git://github.com/opscode-cookbooks/build-essential.git"
"branch": "master",
"revision": "2991fc6e5448e5b2378eb6882e9c1ce557d828f0"
},
"cron": {
"url": "git://github.com/opscode-cookbooks/cron.git"
},
"hp_server": {
"url": "git://github.meetmecorp.com/chef/hp_server.git"
},
"ohai": {
"url": "git://github.com/opscode-cookbooks/ohai.git"
},
"python": {
"url": "git://github.com/opscode-cookbooks/python.git"
},
"resolver": {
"url": "git://github.com/opscode-cookbooks/resolver.git"
},
"sudo": {
"url": "git://github.com/opscode-cookbooks/sudo.git"
},
"supervisor": {
"url": "git://github.com/opscode-cookbooks/supervisor.git"
},
"yum": {
"url": "git://github.com/opscode-cookbooks/yum.git"
}
}
}
Raw
0-knife.rb
# Goes in /var/lib/jenkins/.chef/
log_level :info
log_location STDOUT
syntax_check_cache_path "#{ENV['HOME']}/.chef/syntax_check_cache"
chef_server_url "#{ENV['KNIFE_CHEF_SERVER']}"
client_key "#{ENV['KNIFE_CLIENT_KEY']}"
node_name "#{ENV['KNIFE_NODE_NAME']}"
validation_client_name "#{ENV['KNIFE_VALIDATION_CLIENT_NAME']}"
validation_key "#{ENV['KNIFE_VALIDATION_CLIENT_KEY']}"
cookbook_path "#{ENV['KNIFE_COOKBOOK_PATH']}"
Raw
1-cookbook-update.sh
# This is the first build step for a cookbook repo update which is
# triggered by a GitHub callback to Jenkins on commit
# Reset the workspace, we don't want the git checkout of $JOB_NAME
cd $WORKSPACE/..
rm -rf workspace
mkdir workspace
cd workspace
# Pull down repo
git clone git@github.com:YOUR/chef-repo.git
cd repo
# Add or update the cookbook
if [ ! -d "cookbooks/${JOB_NAME}" ]; then
git submodule add $GIT_URL cookbooks/$JOB_NAME
git submodule init cookbooks/$JOB_NAME
git commit -m "Adding cookbook ${JOB_NAME}" .gitmodules cookbooks/$JOB_NAME
else
git submodule update --init cookbooks/$JOB_NAME
cd cookbooks/$JOB_NAME
git pull -f -u origin master
cd ../..
git commit -m "Updating cookbook ${JOB_NAME} to ${GIT_COMMIT}" .gitmodules cookbooks/$JOB_NAME
fi
Raw
1-other-build.sh
# This is the only build step for updates to the data bag, role and
# environment repositories which are triggered by a GitHub callback
# to Jenkins on commit
# Reset the workspace, we don't want the git checkout of $JOB_NAME
cd $WORKSPACE/..
rm -rf workspace
mkdir workspace
cd workspace
# Pull down repo
git clone git@github.com:YOUR/chef-repo.git
cd repo
# Update the $JOB_NAME
git submodule update --init $JOB_NAME
cd $JOB_NAME
git pull -f -u origin master
cd ..
git commit -m "Updating ${JOB_NAME} to ${GIT_COMMIT}" .gitmodules $JOB_NAME
# Push the change
git push origin master
Raw
2-cookbook-dependency.py
# This script processes the dependencies.json file in a cookbook and either adds
# or updates the submodule information in the master repo, this is run in a job
# per cookbook "Execute Python" build step *after* the cookbook-update.sh build
# step
import json
import os
import subprocess
import sys
base_path = os.environ['WORKSPACE'] + '/' + 'repo'
cookbook = os.environ['JOB_NAME']
cookbook_path = '%s/cookbooks/%s' % (base_path, cookbook)
dependencies = '%s/dependencies.json' % cookbook_path
def execute(command):
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
if p.returncode:
print 'Command failure [%s] returned %s' % (' '.join(command),
p.returncode)
print
print stderr
print
sys.exit(p.returncode)
return stdout
os.chdir(base_path)
if os.path.exists(dependencies):
with open(dependencies, 'r') as handle:
data = json.load(handle)
cookbooks = data.get('cookbooks', dict())
for name in cookbooks:
print 'Processing dependency cookbook %s' % name
dependency_path = '%s/cookbooks/%s' % (base_path, name)
if not os.path.exists(dependency_path):
print 'Adding dependency cookbook %s' % name
execute(['git', 'submodule', 'add', cookbooks[name]['url'], 'cookbooks/%s' % name])
print 'Committing addition of dependency cookbook %s' % name
execute(['git', 'commit', '-m', 'Adding dependency cookbook %s' % name,
'.gitmodules', 'cookbooks/%s' % name])
print 'Updating dependency cookbook %s' % name
execute(['git', 'submodule', 'update', '--init', 'cookbooks/%s' % name])
os.chdir(dependency_path)
if cookbooks[name].get('branch'):
execute(['git', 'checkout', '-b', cookbooks[name].get('branch', 'master')])
execute(['git', 'pull', '-f', '-u', 'origin', cookbooks[name].get('branch', 'master')])
os.chdir(base_path)
if 'revision' in cookbooks[name]:
# Change to that revision
os.chdir(dependency_path)
execute(['git', 'reset', '--hard', cookbooks[name]['revision']])
# Commit the specific revision for that cookbook
os.chdir(base_path)
stdout = execute(['git', 'status', '--porcelain'])
if stdout:
print 'Committing updates to dependency cookbook "%s"' % name
revision = cookbooks[name].get('revision', 'HEAD')
execute(['git', 'commit', '-m', 'Updating dependency cookbook %s to %s' % (name, revision),
'.gitmodules', 'cookbooks/%s' % name])
else:
print 'No changes to commit for dependency cookbook %s' % name
Raw
2-knife_production_upload.sh
# Used for the job that is notified when the master chef repository is updated when
# a cookbook, role, environment or data bag repository updates chef-repo. When
# cookbook-dependency.py, cookbook-update.sh or other-buid.sh are run, they will
# update chef-repo and push the changes, causing GitHub to trigger Jenkins to run
# the job with this shell script as its build step.
# Update the submodules to the proper revisions
git submodule update --init --recursive
# Knife configuration
export KNIFE_CHEF_SERVER="YOUR_CHEF_SERVER"
export KNIFE_CLIENT_KEY="$HOME/.chef/client.pem"
export KNIFE_NODE_NAME="jenkins"
export KNIFE_VALIDATION_CLIENT_NAME="chef-validator"
export KNIFE_VALIDATION_CLIENT_KEY="$HOME/.chef/validation.pem"
export KNIFE_COOKBOOK_PATH=$WORKSPACE/cookbooks
OBJECT=`git log --pretty=oneline -n 1 | perl -n -e '/^(\w{40})\s+(Adding|Updating)\s(cookbook|dependency\scookbook|environments|data_bags|roles)/ && {print "$3"}'`
if [ "$OBJECT" == "cookbook" ] || [ "$OBJECT" == "dependency cookbook" ]; then
# Extract the cookbook to upload
COOKBOOK=`git log --pretty=oneline -n 1 | perl -n -e '/^\w{40}\s+((Adding|Updating)\sdependency\scookbook\s([\w-]+)\sto.*$|(Adding|Updating)\scookbook\s([\w-_]+)\sto.*$)/m && {print "$3$5\n"}'`
# Upload the cookbook
knife cookbook upload $COOKBOOK
elif [ "$OBJECT" == "data_bags" ]; then
cd $WORKSPACE/data_bags
FILES=`git log --pretty=oneline --name-only -n 1 | perl -n -e '/^([\w\-\_]+\/[\w\-\_]+\.json)$/m && {print "$1\n"}'`
cd $WORKSPACE
for data_bag in $FILES
do
knife data bag from file ${data_bag/\// }
done
elif [ "$OBJECT" == "environments" ]; then
cd $WORKSPACE/environments
FILES=`git log --pretty=oneline --name-only -n 1 | perl -n -e '/^([\w\-_]+\.json)$/m && {print "$1\n"}'`
cd $WORKSPACE
for environment in $FILES
do
knife environment from file $environment
done
elif [ "$OBJECT" == "roles" ]; then
cd $WORKSPACE/roles
FILES=`git log --pretty=oneline --name-only -n 1 | perl -n -e '/^([\w\-_]+\.json)$/m && {print "$1\n"}'`
cd $WORKSPACE
for role in FILES
do
knife role from file $f
done
fi
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment