Blueprint is a simple configuration management tool that reverse-engineers servers. It figures out what you’ve done manually, stores it locally in a Git repository, generates code that’s able to recreate your efforts, and helps you deploy those changes to production.
blueprint-list(1): list all blueprints.blueprint-create(1): create a blueprint.blueprint-rules(1): create a blueprint from a blueprint-rules file.blueprint-show(1): generate code from a blueprint.blueprint-diff(1): save the difference between two blueprints.blueprint-split(1): split one blueprint into two others interactively.blueprint-prune(1): select a subset of resources interactively.blueprint-template(1): render mustache.sh templates locally.blueprint-apply(1): run a blueprint’s generated shell code.blueprint-push(1): push a blueprint to the Internet.blueprint-pull(1): pull a blueprint from the Internet.blueprint-destroy(1): destroy a blueprint.blueprint(5): Blueprint JSON format.blueprintignore(5): ignore specific files when creating blueprints.blueprint-rules(5): enumerate resources in blueprints.blueprint.cfg(5): centralized blueprint service configuration.blueprint-template(5): mustache.sh template language syntax.blueprint-template(7): built-in template data.blueprint(7): Blueprint Python library.blueprint-git(1): low-level access to blueprints.blueprint-show-files(1): show files in a blueprint.blueprint-show-ignore(1): show blueprintignore(5) rules from a blueprint.blueprint-show-packages(1): show packages in a blueprint.blueprint-show-services(1): show services in a blueprint.blueprint-show-sources(1): show source tarballs in a blueprint.Blueprint was born out of frustration with development environments, deployment processes, and the complexity of configuration management systems.
Blueprint insists development environments realistically model production and that starts with using Linux. Blueprint only works on Debian- or Red Hat-based Linux systems. We recommend VirtualBox, Vagrant, Rackspace Cloud, or AWS EC2 for development systems that use the same operating system (and version) as is used in production.
On top of the operating system, we recommend using the same web servers, databases, message queue brokers, and other software in development and production. This brings development visibility to entire classes of bugs that only occur due to interactions between production components.
When development and production share the same operating system and software stack, they also share the same interactive management tools, meaning developers and operators alike don’t need to maintain two vocabularies. Well-understood tools like apt-get/dpkg, yum/rpm, and the whole collection of Linux system tools are available everywhere. Blueprint is unique relative to other configuration management in encouraging use of these tools.
What’s common to all configuration management tools is the desire to manage the whole stack: from the operating system packages and services through language-specific packages, all the way to your applications. We need to span all of these across all our systems. To pick on RubyGems arbitrarily: RubyGems is purposely ignorant of its relationship to the underlying system, favoring compatibility with Windows and a wide variety of UNIX-like operating systems. Blueprint understands the macro-dependencies between RubyGems itself and the underlying system and is able to predictably reinstall a selection of gems on top of a properly configured operating system.
When constructing this predictable order-of-operations used to reinstall files, packages, services, and source installations, Blueprint, along with other configuration management tools, takes great care in performing idempotent actions. Thus Blueprint prefers to manage the entire contents of a file rather than a diff or a line to append. Idempotency means you can apply a blueprint over and over again with confidence that nothing will change if nothing needs to change.
Because Blueprint can reverse-engineer systems, it is of particular use migrating legacy systems into configuration management. It doesn’t matter when you install Blueprint: changes made to the system even before Blueprint is installed will be taken into account.
Prerequisites:
echo "deb http://packages.devstructure.com $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/devstructure.list
sudo wget -O /etc/apt/trusted.gpg.d/devstructure.gpg http://packages.devstructure.com/keyring.gpg
sudo apt-get update
sudo apt-get -y install blueprint
pip install blueprint
Make sure pip is using Python >= 2.6, otherwise the installation will succeed but Blueprint will not run.
git clone git://github.com/devstructure/blueprint.git
cd blueprint
git submodule update --init
make && sudo make install
rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm
yum install python26
git clone git://github.com/devstructure/blueprint.git
cd blueprint
git submodule update --init
make && sudo make install PYTHON=/usr/bin/python26
This installs Python 2.6 from EPEL side-by-side with Python 2.4 and so won’t break Yum.
By now you’ve hopefully created a development environment running the same operating system (and version) that you use in production and installed your production stack. If this is your first time configuring production databases and web servers, don’t be shy about copying-and-pasting from the guy at the next desk or a staging or production system (perhaps even multiple systems).
Once Blueprint itself is installed you can reverse-engineer your development system with a single command.
blueprint create nameBlueprint will list packages managed by APT, Yum, RubyGems, Python’s easy_install and pip, PHP’s PEAR and PECL, and Node.js’ NPM. It will also determine which configuration files in /etc have been added or modified from their packaged versions and collect files in /usr/local that are part of any software packages installed from source (typically via GNU make(1)). Finally, it will build a list of conditions under which System V init or Upstart services should be restarted, including package upgrades and configuration changes.
All of this information is encoded in a blueprint(5) JSON document and zero or more tar(5) archives and stored in a branch called name in the local Git repository ~/.blueprints.git.
Any blueprint in the local Git repository may be applied to the system with blueprint-apply(1):
blueprint apply nameSuppose you want to install a basic Ruby stack for running a Sinatra application called example proxied by Nginx on Debian-based Linux. Install the prerequisite packages:
sudo apt-get install build-essential nginx-light ruby ruby-dev rubygems
sudo gem install sinatra unicorn
Configure Nginx in /etc/nginx/sites-available/example:
server {
listen 80 default;
location /static { root /usr/local/share/rack/example/static; }
location / { proxy_pass http://127.0.0.1:8080; }
}
Enable the Nginx configuration:
ln -s /etc/nginx/sites-{available,enabled}/example
Create the application itself. /usr/local/share/rack/example/app.rb:
require 'sinatra'
get '/hi' do
"Hello, world!\n"
end
/usr/local/share/rack/example/config.ru:
require 'app'
run Sinatra::Application
Configure Unicorn in /etc/unicorn.conf.rb:
worker_processes 4
Configure Upstart to run the application:
description "Example"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
chdir /usr/local/share/rack/example
exec unicorn -c /etc/unicorn.conf.rb
Now create a blueprint and call it example.
sudo blueprint create example
We’ll pick this example up in the next chapter.
Once a blueprint has been created and stored in the local Git repository, it’s easy to inspect what’s been included both in its raw JSON form and in more friendly formats.
First, print the entire blueprint(5) JSON document:
blueprint show name | lessThe output of blueprint-show(1) can be a bit overwhelming, which is one of the reasons the following four commands were created.
blueprint-show-files(1) lists the pathname of each file included in the blueprint on its own line.
pathnameblueprint-show-packages(1) lists the manager (“apt”, “yum”, “rubygems1.8”, etc.), name, and version number for each package included in the blueprint on its own line. If two versions of the same package are included (as is possibly with RubyGems and some other package managers), each will be printed on its own line.
manager package versionblueprint-show-services(1) lists the manager (“sysvinit” or “upstart”) and name of each service included in the blueprint on its own line.
manager serviceblueprint-show-sources(1) lists the source installations included in the blueprint. The root directory (for example, “/usr/local”) and tarball filename are printed to standard error and the contents of the tarball are printed to standard output via tar(1)’s tv options.
dirname filename
`tar tv` outputblueprint show-files example
/etc/apt/sources.list
/etc/hosts
/etc/init/example.conf
/etc/mysql/debian.cnf
/etc/nginx/sites-available/example
/etc/nginx/sites-enabled/default
/etc/nginx/sites-enabled/example
/etc/unicorn.conf.rb
blueprint show-packages example
apt binutils 2.21.53.20110810-0ubuntu3
apt blueprint 3.4.0-1py2.7
apt build-essential 11.5ubuntu1
apt ca-certificates 20110502+nmu1ubuntu5
apt cpp 4:4.6.1-2ubuntu5
apt cpp-4.6 4.6.1-9ubuntu3
apt curl 7.21.6-3ubuntu3
apt dpkg-dev 1.16.0.3ubuntu5
apt fakeroot 1.17-1
apt g++ 4:4.6.1-2ubuntu5
apt g++-4.6 4.6.1-9ubuntu3
apt gcc 4:4.6.1-2ubuntu5
apt gcc-4.6 4.6.1-9ubuntu3
apt git 1:1.7.5.4-1
apt git-core 1:1.7.5.4-1
apt git-man 1:1.7.5.4-1
apt libalgorithm-diff-perl 1.19.02-2
apt libalgorithm-diff-xs-perl 0.04-1build1
apt libalgorithm-merge-perl 0.08-2
apt libc-dev-bin 2.13-20ubuntu5
apt libc6-dev 2.13-20ubuntu5
apt libcurl3 7.21.6-3ubuntu3
apt libcurl3-gnutls 7.21.6-3ubuntu3
apt libdbd-mysql-perl 4.019-1
apt libdbi-perl 1.616-1build1
apt libdpkg-perl 1.16.0.3ubuntu5
apt liberror-perl 0.17-1
apt libgmp10 2:5.0.1+dfsg-7ubuntu2
apt libgomp1 4.6.1-9ubuntu3
apt libhtml-template-perl 2.10-1
apt libmpc2 0.9-3
apt libmpfr4 3.0.1-5
apt libmysqlclient16 5.1.58-1ubuntu1
apt libnet-daemon-perl 0.48-1
apt libplrpc-perl 0.2020-2
apt libquadmath0 4.6.1-9ubuntu3
apt libreadline5 5.2-9ubuntu1
apt librtmp0 2.3-2ubuntu1
apt libruby1.8 1.8.7.352-2
apt libstdc++6-4.6-dev 4.6.1-9ubuntu3
apt libtimedate-perl 1.2000-1
apt libwrap0 7.6.q-21
apt linux-libc-dev 3.0.0-12.20
apt manpages-dev 3.27-1ubuntu2
apt mysql-client-5.1 5.1.58-1ubuntu1
apt mysql-client-core-5.1 5.1.58-1ubuntu1
apt mysql-common 5.1.58-1ubuntu1
apt mysql-server 5.1.58-1ubuntu1
apt mysql-server-5.1 5.1.58-1ubuntu1
apt mysql-server-core-5.1 5.1.58-1ubuntu1
apt nginx-common 1.0.5-1
apt nginx-light 1.0.5-1
apt openssh-server 1:5.8p1-7ubuntu1
apt openssl 1.0.0e-2ubuntu4
apt python-pip 1.0-1
apt python-pkg-resources 0.6.16-1
apt python-setuptools 0.6.16-1
apt rsync 3.0.8-1ubuntu1
apt ruby 4.8
apt ruby-dev 4.8
apt ruby1.8 1.8.7.352-2
apt ruby1.8-dev 1.8.7.352-2
apt rubygems 1.7.2-1
apt tmux 1.5-1
python-pip Django 1.3.1
rubygems fpm 0.3.11
rubygems hpricot 0.8.5
rubygems json 1.6.3
rubygems kgio 2.6.0
rubygems mustache 0.99.4
rubygems rack 1.3.5
rubygems rack-protection 1.1.4
rubygems raindrops 0.8.0
rubygems rdiscount 1.6.8
rubygems sinatra 1.3.1
rubygems tilt 1.3.3
rubygems unicorn 4.1.1
blueprint show-services example
sysvinit nginx
sysvinit ssh
upstart example
upstart mysql
blueprint show-sources example
/usr/local 82789250f3ad94d4cfeacc0a8cabae2d26b0270e.tar
drwxr-xr-x root/root 0 2011-11-29 21:42 ./
drwxr-xr-x root/root 0 2011-11-29 20:50 ./share/
drwxr-xr-x root/root 0 2011-11-29 21:55 ./share/rack/
drwxr-xr-x rcrowley/rcrowley 0 2011-12-09 15:12 ./share/rack/example/
-rw-rw-r-- rcrowley/rcrowley 55 2011-11-29 21:02 ./share/rack/example/app.rb
-rw-rw-r-- rcrowley/rcrowley 39 2011-11-29 20:54 ./share/rack/example/config.ru
We’re off to a great start but there are some extraneous files and packages here that need cleaning up.
Rather than requiring you enumerate every detail of your infrastructure in code, Blueprint reverse-engineers most of these details from running systems. There are times, though, when it’s a bit too verbose.
Inspired by similar features in version control software, Blueprint allows you to enumerate files, packages, services, and sources that should be ignored in a file format inspired by gitignore(5). See blueprintignore(5) for details.
Blueprint looks for these rules in /etc/blueprintignore and ~/.blueprintignore. Rules in /etc/blueprintignore will be included in the blueprint itself, thereby propagating to other users of the blueprint. Rules in ~/.blueprintignore will remain local though each revision of a blueprint will store the full set of rules used to create it.
Files may be specified by fully-qualified or relative pathnames, possibly including glob syntax:
/etc/foo
foo/*
*.foo
[abc]/[xyz]
Packages must be specified by their manager and name, prefixed with :package::
:package:apt/build-essential
When a package is ignored, packages on which it depends are also ignored.
Services must be specified by their manager and name, prefixed with :service::
:service:sysvinit/ssh
Sources must be specified by fully-qualified pathnames:
:sources:/usr/local
You can ignore and unignore particular files within a source directory to fine-tune what’s included in the tarball.
Any rule may be negated by prefixing it with a !, which overrides defaults and well as previous matching rules - the last matching rule wins.
/etc/apt/sources.list and /etc/hosts really don’t belong in the blueprint since they’re part of the operating system and we haven’t customized them. Add their pathnames to /etc/blueprintignore:
/etc/apt/sources.list
/etc/hosts
/etc/mysql/debian.cnf is generated by the MySQL server package in its postinst maintainer script. Add its pathname to /etc/blueprintignore:
/etc/mysql/debian.cnf
/etc/nginx/sites-enabled/default is part of the basic Nginx installation and again we don’t particularly care about it. Add its pathname to /etc/blueprintignore:
/etc/nginx/sites-enabled/default
build-essential brought a lot of friends along that are clouding what’s really important in this blueprint. Recall that ignoring a package also ignores its dependencies. The opposite is not true: unignoring a package leaves its dependencies alone. Ignoring and immediately unignoring build-essential, ruby, and ruby-dev will slim down the blueprint without any loss in completeness.
:package:apt/build-essential
!:package:apt/build-essential
:package:apt/ruby
!:package:apt/ruby
:package:apt/ruby-dev
!:package:apt/ruby-dev
Running sudo blueprint create example again will commit a new blueprint that takes these rules into account.
As complexity grows, you’re likely to pass an inflection point when it becomes easier to enumerate the resources you care about, not the resources that should be ignored. When the time comes, Blueprint is ready. The rules syntax used to ignore particular resources can be turned around and used to enumerate the resources to include in the blueprint. See blueprint-rules(5) for details.
The blueprint-rules(1) command reverse-engineers the system just like blueprint-create but limits the resources included in the blueprint to those in the rules file.
blueprint rules pathnameThe rules files are a bit of a hybrid between the traditional Blueprint approach of reverse-engineering and the typical configuration-as-code approach of other configuration management tools.
Add these rules to example.blueprint-rules:
:source:/usr/local
/etc/init/example.conf
/etc/nginx/sites-*/example
:package:apt/libmysqlclient-dev
:package:apt/mysql-client-5.1
:package:apt/nginx-common
:package:apt/nginx-light
:package:apt/ruby-dev
:package:apt/rubygems
:package:rubygems/*
:service:sysvinit/nginx
:service:upstart/example
Now create the blueprint example:
blueprint rules example.blueprint-rules
As usual, the blueprint is stored locally in Git, ready for action.
A good exercise for the reader is creation of a separate blueprint for the MySQL server and its configuration. It’s installed alongside the web stack in development but that’s unlikely to be the case in production, so having two blueprints could be advantageous. The next chapter introduces another way to create two blueprints from one system.
Refactoring is a major part of software development practice and configuration management shouldn’t be left out in the cold. Blueprint provides several tools that can be used to refactor your blueprints into more modular, maintainable, focused artifacts.
blueprint-diff(1) takes direct advantage of the subtraction operator available on Blueprint objects in the underlying library.
blueprint diff minuend subtrahend differenceResources that appear in minuend but not subtrahend will be included in difference and committed to the local Git repository under that name.
blueprint-split(1) and blueprint-prune(1) are interactive refactoring tools.
blueprint-split prints each resource in src and prompts you for a choice of dest-a or dest-b. The resulting dest-a and dest-b blueprints are committed to the local Git repository.
blueprint split src dest-a dest-bblueprint-prune instead prompts you to include or ignore each resource in src in dest.
blueprint prune src destThere are some limitations in both of these tools, however. You can’t currently split the files within a source tarball. The best workaround is to use rules files to create blueprints that contain only the files you want.
blueprint prune example example-nginx
/usr/local 6326b42413443bc2d94e13747d05f650b873a53e.tar
Include in blueprint example-nginx? [y/n] n
/etc/init/example.conf
Include in blueprint example-nginx? [y/n] n
/etc/nginx/sites-available/example
Include in blueprint example-nginx? [y/n] y
/etc/nginx/sites-enabled/example
Include in blueprint example-nginx? [y/n] y
apt nginx-common 1.0.5-1
Include in blueprint example-nginx? [y/n] y
apt nginx-light 1.0.5-1
Include in blueprint example-nginx? [y/n] y
apt ruby-dev 4.8
Include in blueprint example-nginx? [y/n] n
apt rubygems 1.7.2-1
Include in blueprint example-nginx? [y/n] n
rubygems fpm 0.3.11
Include in blueprint example-nginx? [y/n] n
rubygems hpricot 0.8.5
Include in blueprint example-nginx? [y/n] n
rubygems json 1.6.3
Include in blueprint example-nginx? [y/n] n
rubygems kgio 2.6.0
Include in blueprint example-nginx? [y/n] n
rubygems mustache 0.99.4
Include in blueprint example-nginx? [y/n] n
rubygems rack 1.3.5
Include in blueprint example-nginx? [y/n] n
rubygems rack-protection 1.1.4
Include in blueprint example-nginx? [y/n] n
rubygems raindrops 0.8.0
Include in blueprint example-nginx? [y/n] n
rubygems rdiscount 1.6.8
Include in blueprint example-nginx? [y/n] n
rubygems sinatra 1.3.1
Include in blueprint example-nginx? [y/n] n
rubygems tilt 1.3.3
Include in blueprint example-nginx? [y/n] n
rubygems unicorn 4.1.1
Include in blueprint example-nginx? [y/n] n
sysvinit nginx
Include in blueprint example-nginx? [y/n] y
upstart example
Include in blueprint example-nginx? [y/n] n
The same could have been done with blueprint-split to create example-nginx and example-mysql or base and custom — any distinction you desire.
Not all systems are created equal. Certainly your m2.4xlarge AWS EC2 instance packs a bit more CPU than a virtual machine running in a corner of your laptop and your configurations should be able to cope with these operational extremes.
Blueprint allows configuration files (found in /etc) to be specified as templates and (optionally) data rather than static content. These templates are rendered by a special portable dialect of the mustache(5) template language called mustache.sh. See blueprint-template(5) for details.
In their simplest form, these templates allow substitution of system parameters.
{{FOO}}You can also substitute the output of a shell command.
{{`echo foo`}}More complex uses can iterate over the lines of output from a shell command.
{{#`echo foo; echo bar; echo baz`}}
{{_M_LINE}}
{{/`echo foo; echo bar; echo baz`}}Blueprint ships with a small helping of common system parameters:
CORES: the number of CPU cores in the system.MEM: the total amount of memory in the system in bytes.FQDN: the fully-qualified domain name according to hostname(1)’s --fqdn option.PRIVATE_IP: the first private IPv4 address assigned to any interface.PUBLIC_IP: the first public IPv4 address assigned to any interface.There are several more, documented in blueprint-template(7). You can provide your own global data by adding source-able shell scripts with the .sh suffix to /etc/blueprint-template.d.
Any pathname may have an associated template, pathname.blueprint-template.mustache. An additional source-able shell script private to just this template may be placed in pathname.blueprint-template.sh.
To render a template locally (likely during development), use blueprint-template(1).
blueprint template pathnameOur Unicorn configuration statically assumes four workers is the best configuration. In reality, Unicorn workers are a function of the number of CPU cores available.
Configure Unicorn to use one worker per CPU in /etc/unicorn.conf.rb.blueprint-template.mustache:
worker_processes {{CORES}}Configure Unicorn to use four workers per CPU in /etc/unicorn.conf.rb.blueprint-template.mustache:
worker_processes {{`expr 4 \* $CORES`}}Or, you can extract computation out of the template and into /etc/unicorn.conf.rb.blueprint-template.sh:
export WORKER_PROCESSES="$(expr 4 \* $CORES)"
/etc/unicorn.conf.rb.blueprint-template.mustache becomes:
worker_processes {{WORKER_PROCESSES}}Now Blueprint can scale this Unicorn configuration up and down as the system allows or requires.
The last step in applying a blueprint is to restart all the services whose configuration changed. Most configuration management tools require you to connect the dots explicitly but Blueprint finds many of those relationships automatically.
System V init and Upstart services are included in blueprints when their init script or configuration file, or the package that contains the init script or configuration file, are included in the blueprint. From there, Blueprint searches for resources that, when changed, should cause the service to restart.
Other files may be added to the list to watch by naming them in a comment in the service init script or configuration file.
Suppose our example application reads /etc/database.yml for database connection credentials. The Unicorn application server must be restarted to read changes to this file. Mention /etc/database.yml in the Upstart configuration:
description "Example"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
chdir /usr/local/share/rack/example
exec unicorn -c /etc/unicorn.conf.rb
# Dear Blueprint, restart me when /etc/database.yml changes.
Beneath the blueprint-apply(1) command briefly introduced before there is the -S option to blueprint-show(1) which generates a POSIX shell script that can apply a blueprint to any system.
Dependencies are especially painful when bootstrapping new systems so Blueprint takes great pains to generate dependency-free shell scripts. They extract source tarballs, place file contents and adjust owners/groups/modes, install packages, and restart services as necessary according to the algorithm described in blueprint(5). Even template rendering is handled without any dependencies.
The generated shell script for the blueprint name is written to a file in your working directory as name.sh or name/bootstrap.sh if the blueprint contains templates or source tarballs.
blueprint show -S nameThis shell script and its associated files can drive deployment, provisioning, or other development environments without even having to install Blueprint first.
Running blueprint show -S example creates example/bootstrap.sh as follows and bundles mustache.sh and the source tarball containing our example application. example/bootstrap.sh:
#
# Automatically generated by blueprint(7). Edit at your own risk.
#
set -x
cd "$(dirname "$0")"
mkdir -p "/etc/init"
cat >"/etc/init/example.conf" <<EOF
description "Example"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
chdir /usr/local/share/rack/example
exec unicorn -c /etc/unicorn.conf.rb
# Dear Blueprint, restart me when /etc/database.yml changes.
EOF
MD5SUM="$(md5sum "/etc/nginx/sites-available/example" 2>/dev/null)"
mkdir -p "/etc/nginx/sites-available"
cat >"/etc/nginx/sites-available/example" <<EOF
server {
listen 80 default;
location /static { root /usr/local/share/rack/example/static; }
location / { proxy_pass http://127.0.0.1:8080; }
}
EOF
[ "$MD5SUM" != "$(md5sum "/etc/nginx/sites-available/example")" ] && SERVICE_sysvinit_nginx=1
MD5SUM="$(md5sum "/etc/nginx/sites-enabled/example" 2>/dev/null)"
mkdir -p "/etc/nginx/sites-enabled"
ln -s "/etc/nginx/sites-available/example" "/etc/nginx/sites-enabled/example"
[ "$MD5SUM" != "$(md5sum "/etc/nginx/sites-enabled/example")" ] && SERVICE_sysvinit_nginx=1
MD5SUM="$(md5sum "/etc/unicorn.conf.rb" 2>/dev/null)"
mkdir -p "/etc"
(
set +x
. "lib/mustache.sh"
for F in */blueprint-template.d/*.sh
do
. "$F"
done
export WORKER_PROCESSES="$(expr 4 \* $CORES)"
mustache >"/etc/unicorn.conf.rb" <<EOF
worker_processes {{WORKER_PROCESSES}}
EOF
)
[ "$MD5SUM" != "$(md5sum "/etc/unicorn.conf.rb")" ] && SERVICE_upstart_example=1
export APT_LISTBUGS_FRONTEND="none"
export APT_LISTCHANGES_FRONTEND="none"
export DEBIAN_FRONTEND="noninteractive"
apt-get -q update
[ "$(dpkg-query -f'${Version}\n' -W mysql-client-5.1)" = "5.1.58-1ubuntu1" ] || apt-get -y -q -o DPkg::Options::=--force-confold install mysql-client-5.1=5.1.58-1ubuntu1
[ "$(dpkg-query -f'${Version}\n' -W nginx-common)" = "1.0.5-1" ] || { apt-get -y -q -o DPkg::Options::=--force-confold install nginx-common=1.0.5-1; SERVICE_sysvinit_nginx=1; }
[ "$(dpkg-query -f'${Version}\n' -W nginx-light)" = "1.0.5-1" ] || apt-get -y -q -o DPkg::Options::=--force-confold install nginx-light=1.0.5-1
[ "$(dpkg-query -f'${Version}\n' -W ruby-dev)" = "4.8" ] || apt-get -y -q -o DPkg::Options::=--force-confold install ruby-dev=4.8
[ "$(dpkg-query -f'${Version}\n' -W rubygems)" = "1.7.2-1" ] || apt-get -y -q -o DPkg::Options::=--force-confold install rubygems=1.7.2-1
gem -i -v0.3.11 fpm >/dev/null || gem install --no-rdoc --no-ri -v0.3.11 fpm
gem -i -v0.8.5 hpricot >/dev/null || gem install --no-rdoc --no-ri -v0.8.5 hpricot
gem -i -v1.6.3 json >/dev/null || gem install --no-rdoc --no-ri -v1.6.3 json
gem -i -v2.6.0 kgio >/dev/null || gem install --no-rdoc --no-ri -v2.6.0 kgio
gem -i -v0.99.4 mustache >/dev/null || gem install --no-rdoc --no-ri -v0.99.4 mustache
gem -i -v1.3.5 rack >/dev/null || gem install --no-rdoc --no-ri -v1.3.5 rack
gem -i -v1.1.4 rack-protection >/dev/null || gem install --no-rdoc --no-ri -v1.1.4 rack-protection
gem -i -v0.8.0 raindrops >/dev/null || gem install --no-rdoc --no-ri -v0.8.0 raindrops
gem -i -v1.6.8 rdiscount >/dev/null || gem install --no-rdoc --no-ri -v1.6.8 rdiscount
gem -i -v1.3.1 sinatra >/dev/null || gem install --no-rdoc --no-ri -v1.3.1 sinatra
gem -i -v1.3.3 tilt >/dev/null || gem install --no-rdoc --no-ri -v1.3.3 tilt
gem -i -v4.1.1 unicorn >/dev/null || gem install --no-rdoc --no-ri -v4.1.1 unicorn
[ -n "$SERVICE_sysvinit_nginx" ] && /etc/init.d/nginx restart
[ -n "$SERVICE_upstart_example" ] && { restart example || start example; }DevStructure runs a free Blueprint Server at devstructure.com but this service will not be available after June 30th, 2012. Learn more about running your own Blueprint Server
We at DevStructure saw the workflow unfolding around these generated shell scripts and in them an opportunity for a macro don’t-repeat-yourself optimization. Thus were born blueprint-push(1) and blueprint-pull(1).
If you decide to run your own Blueprint Server, you’ll need to configure a server ahead of time in /etc/blueprint.cfg:
[io]
server = serverblueprint-push uploads the JSON document and all source tarballs referenced by a blueprint to a Blueprint Server, which stores them behind a long secret key in AWS S3. The URL that may be used to pull the blueprint later is printed to standard output.
blueprint push nameThe first time you run this command it will prompt you with the contents of /etc/blueprint.cfg which you can optionally put in place. If you do, Blueprint will reuse the same secret the next time blueprint-push is called. Configuring a secret this way allows you to push revisions to your blueprints which can then be pulled from a known location.
blueprint-pull downloads the JSON document and all source tarballs referenced by a blueprint that has been pushed and stores them in the local Git repository. Typically, it accepts the URL printed by blueprint-push but if you configure a default secret in /etc/blueprint.cfg, you can pull blueprints by only their name.
blueprint pull urlblueprint pull nameJust as blueprint-show’s -S option helps bootstrap systems with zero dependencies, devstructure.com can generate shell scripts remotely so Blueprint doesn’t have to be installed ahead of time. You can bootstrap a new system from a pushed blueprint in one command:
curl https://devstructure.com/secret/name/name.sh | shPush the example blueprint to devstructure.com:
blueprint push example
Now configure a production system to apply the latest revision to example every half hour (this behavior should be familiar to Puppet and Chef users). In root’s crontab:
*/30 * * * * curl https://devstructure.com/0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz_-/example/example.sh | sh
Blueprint can also streamline the development workflow in Puppet- or Chef-managed environments by generating complete Puppet modules or Chef cookbooks.
Generate a Puppet module in name/manifests/init.pp:
blueprint show -P nameGenerate a Chef cookbook in name/recipes/default.rb:
blueprint show -C nameThese modules and cookbooks may be included directly in any Puppet or Chef environment or be used as the starting point for further development — the code is formatted according to the style guidelines of the respective communities.
Note, however, that the file templates used by Blueprint are incompatible with Puppet and Chef and so can’t be included in the generated modules and cookbooks.
Running blueprint show -P example creates example/manifests/init.pp as follows and bundles the source tarball containing our example application. example/manifests/init.pp:
#
# Automatically generated by blueprint(7). Edit at your own risk.
#
class example {
Exec {
path => '/home/rcrowley/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games',
}
Class['sources'] -> Class['files'] -> Class['packages']
class files {
file {
'/etc':
ensure => directory;
'/etc/init':
ensure => directory;
'/etc/init/example.conf':
content => template('example/etc/init/example.conf'),
ensure => file,
group => root,
mode => 0644,
owner => root;
'/etc/nginx':
ensure => directory;
'/etc/nginx/sites-available':
ensure => directory;
'/etc/nginx/sites-available/example':
content => template('example/etc/nginx/sites-available/example'),
ensure => file,
group => root,
mode => 0644,
owner => root;
'/etc/nginx/sites-enabled':
ensure => directory;
'/etc/nginx/sites-enabled/example':
ensure => '/etc/nginx/sites-available/example',
group => root,
owner => root;
'/etc/unicorn.conf.rb':
content => template('example/etc/unicorn.conf.rb'),
ensure => file,
group => root,
mode => 0644,
owner => root;
}
}
include files
class packages {
Class['apt'] -> Class['rubygems']
exec { 'apt-get -q update':
before => Class['apt'],
}
class apt {
package {
'mysql-client-5.1':
ensure => '5.1.58-1ubuntu1';
'nginx-common':
ensure => '1.0.5-1';
'nginx-light':
ensure => '1.0.5-1';
'ruby-dev':
ensure => '4.8';
'rubygems':
ensure => '1.7.2-1';
}
}
include apt
class rubygems {
package {
'fpm':
ensure => '0.3.11',
provider => gem;
'hpricot':
ensure => '0.8.5',
provider => gem;
'json':
ensure => '1.6.3',
provider => gem;
'kgio':
ensure => '2.6.0',
provider => gem;
'mustache':
ensure => '0.99.4',
provider => gem;
'rack':
ensure => '1.3.5',
provider => gem;
'rack-protection':
ensure => '1.1.4',
provider => gem;
'raindrops':
ensure => '0.8.0',
provider => gem;
'rdiscount':
ensure => '1.6.8',
provider => gem;
'sinatra':
ensure => '1.3.1',
provider => gem;
'tilt':
ensure => '1.3.3',
provider => gem;
'unicorn':
ensure => '4.1.1',
provider => gem;
}
}
include rubygems
}
include packages
class services {
class sysvinit {
service { 'nginx':
enable => true,
ensure => running,
subscribe => [File['/etc/nginx/sites-available/example'], File['/etc/nginx/sites-enabled/example'], Package['nginx-common'], Exec['82789250f3ad94d4cfeacc0a8cabae2d26b0270e.tar']],
}
}
include sysvinit
class upstart {
service { 'example':
enable => true,
ensure => running,
provider => upstart,
subscribe => [File['/etc/unicorn.conf.rb'], Exec['82789250f3ad94d4cfeacc0a8cabae2d26b0270e.tar']],
}
}
include upstart
}
include services
class sources {
exec { 'tar xf /tmp/82789250f3ad94d4cfeacc0a8cabae2d26b0270e.tar':
alias => '/usr/local',
cwd => '/usr/local',
}
file { '/tmp/82789250f3ad94d4cfeacc0a8cabae2d26b0270e.tar':
before => Exec['/usr/local'],
group => root,
mode => 0644,
owner => root,
source => 'puppet:///modules/example/tmp/82789250f3ad94d4cfeacc0a8cabae2d26b0270e.tar',
}
}
include sources
}
Running blueprint show -C example creates example/recipes/default.rb as follows and bundles the source tarball containing our example application. example/recipes/default.rb:
#
# Automatically generated by blueprint(7). Edit at your own risk.
#
cookbook_file('/tmp/82789250f3ad94d4cfeacc0a8cabae2d26b0270e.tar') do
backup false
group 'root'
mode '0644'
owner 'root'
source 'tmp/82789250f3ad94d4cfeacc0a8cabae2d26b0270e.tar'
end
execute('tar xf "/tmp/82789250f3ad94d4cfeacc0a8cabae2d26b0270e.tar"') { cwd '/usr/local' }
directory('/etc/init') do
group 'root'
mode '0755'
owner 'root'
recursive true
end
cookbook_file('/etc/init/example.conf') do
backup false
group 'root'
mode '0644'
owner 'root'
source 'etc/init/example.conf'
end
directory('/etc/nginx/sites-available') do
group 'root'
mode '0755'
owner 'root'
recursive true
end
cookbook_file('/etc/nginx/sites-available/example') do
backup false
group 'root'
mode '0644'
owner 'root'
source 'etc/nginx/sites-available/example'
end
directory('/etc/nginx/sites-enabled') do
group 'root'
mode '0755'
owner 'root'
recursive true
end
link('/etc/nginx/sites-enabled/example') do
group 'root'
owner 'root'
to '/etc/nginx/sites-available/example'
end
directory('/etc') do
group 'root'
mode '0755'
owner 'root'
recursive true
end
cookbook_file('/etc/unicorn.conf.rb') do
backup false
group 'root'
mode '0644'
owner 'root'
source 'etc/unicorn.conf.rb'
end
execute('apt-get -q update')
package('mysql-client-5.1') { version '5.1.58-1ubuntu1' }
package('nginx-common') { version '1.0.5-1' }
package('nginx-light') { version '1.0.5-1' }
package('ruby-dev') { version '4.8' }
package('rubygems') { version '1.7.2-1' }
gem_package('fpm') { version '0.3.11' }
gem_package('hpricot') { version '0.8.5' }
gem_package('json') { version '1.6.3' }
gem_package('kgio') { version '2.6.0' }
gem_package('mustache') { version '0.99.4' }
gem_package('rack') { version '1.3.5' }
gem_package('rack-protection') { version '1.1.4' }
gem_package('raindrops') { version '0.8.0' }
gem_package('rdiscount') { version '1.6.8' }
gem_package('sinatra') { version '1.3.1' }
gem_package('tilt') { version '1.3.3' }
gem_package('unicorn') { version '4.1.1' }
service('nginx') do
action [:enable, :start]
subscribes :restart, resources('cookbook_file[/etc/nginx/sites-available/example]', 'cookbook_file[/etc/nginx/sites-enabled/example]', 'package[nginx-common]', 'execute[82789250f3ad94d4cfeacc0a8cabae2d26b0270e.tar]')
end
service('example') do
action [:enable, :start]
provider Chef::Provider::Service::Upstart
subscribes :restart, resources('cookbook_file[/etc/unicorn.conf.rb]', 'execute[82789250f3ad94d4cfeacc0a8cabae2d26b0270e.tar]')
end
Production environments are much more than a rack of servers these days and nowhere is that more apparent than AWS. Blueprint integrates with AWS CloudFormation to provision and bootstrap entire infrastructures declaratively.
The --cfn option to blueprint-show(1) generates the skeleton of a CloudFormation template that provisions a single EC2 instance running Amazon Linux (an RPM-based distribution supported by Amazon) which will apply the blueprint during its first boot.
File templates and source tarballs aren’t supported seamlessly, however. File templates can be recreated using CloudFormation’s primitive string functions or an executable user-data. Source tarballs may reference fully-qualified URLs.
Running blueprint show --cfn example creates example.json:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Create an Amazon EC2 instance and bootstrap it as instructed by a blueprint. This is intended to be a starting point for building larger architectures with AWS CloudFormation. **WARNING** This template creates an Amazon EC2 instance. You will be billed for the AWS resources used if you create a stack from this template.",
"Mappings": {
"AWSInstanceType2Arch": {
"c1.medium": {
"Arch": "32"
},
"c1.xlarge": {
"Arch": "64"
},
"cc1.4xlarge": {
"Arch": "64"
},
"m1.large": {
"Arch": "64"
},
"m1.small": {
"Arch": "32"
},
"m1.xlarge": {
"Arch": "64"
},
"m2.2xlarge": {
"Arch": "64"
},
"m2.4xlarge": {
"Arch": "64"
},
"m2.xlarge": {
"Arch": "64"
},
"t1.micro": {
"Arch": "32"
}
},
"AWSRegionArch2AMI": {
"ap-northeast-1": {
"32": "ami-dcfa4edd",
"64": "ami-e8fa4ee9"
},
"ap-southeast-1": {
"32": "ami-74dda626",
"64": "ami-7edda62c"
},
"eu-west-1": {
"32": "ami-24506250",
"64": "ami-20506254"
},
"us-east-1": {
"32": "ami-7f418316",
"64": "ami-7341831a"
},
"us-west-1": {
"32": "ami-951945d0",
"64": "ami-971945d2"
}
}
},
"Outputs": {
"PublicDnsName": {
"Description": "Public DNS name of the EC2 instance.",
"Value": {
"Fn::GetAtt": [
"EC2Instance",
"PublicDnsName"
]
}
}
},
"Parameters": {
"InstanceType": {
"AllowedValues": [
"t1.micro",
"m1.small",
"m1.large",
"m1.xlarge",
"m2.xlarge",
"m2.2xlarge",
"m2.4xlarge",
"c1.medium",
"c1.xlarge",
"cc1.4xlarge"
],
"ConstraintDescription": "must be a valid EC2 instance type.",
"Default": "m1.small",
"Description": "EC2 instance type.",
"Type": "String"
},
"KeyName": {
"Description": "Name of an existing EC2 KeyPair to enable SSH access to the instances",
"Type": "String"
}
},
"Resources": {
"CfnUser": {
"Properties": {
"Path": "/",
"Policies": [
{
"PolicyDocument": {
"Statement": [
{
"Action": "cloudformation:DescribeStackResource",
"Effect": "Allow",
"Resource": "*"
}
]
},
"PolicyName": "root"
}
]
},
"Type": "AWS::IAM::User"
},
"EC2Instance": {
"Metadata": {
"AWS::CloudFormation::Init": {
"config": {
"files": {
"/etc/init/example.conf": {
"content": "description \"Example\"\nstart on runlevel [2345]\nstop on runlevel [!2345]\nrespawn\nchdir /usr/local/share/rack/example\nexec unicorn -c /etc/unicorn.conf.rb\n# Dear Blueprint, restart me when /etc/database.yml changes.\n",
"encoding": "plain",
"group": "root",
"mode": "100644",
"owner": "root"
},
"/etc/nginx/sites-available/example": {
"content": "server {\n\tlisten 80 default;\n\tlocation /static { root /usr/local/share/rack/example/static; }\n\tlocation / { proxy_pass http://127.0.0.1:8080; }\n}\n",
"encoding": "plain",
"group": "root",
"mode": "100644",
"owner": "root"
},
"/etc/nginx/sites-enabled/example": {
"content": "/etc/nginx/sites-available/example",
"encoding": "plain",
"group": "root",
"mode": "120777",
"owner": "root"
},
"/etc/unicorn.conf.rb": {
"content": "worker_processes 4\n",
"encoding": "plain",
"group": "root",
"mode": "100644",
"owner": "root"
}
},
"packages": {
"apt": {
"mysql-client-5.1": [
"5.1.58-1ubuntu1"
],
"nginx-common": [
"1.0.5-1"
],
"nginx-light": [
"1.0.5-1"
],
"ruby-dev": [
"4.8"
],
"rubygems": [
"1.7.2-1"
]
},
"rubygems": {
"fpm": [
"0.3.11"
],
"hpricot": [
"0.8.5"
],
"json": [
"1.6.3"
],
"kgio": [
"2.6.0"
],
"mustache": [
"0.99.4"
],
"rack": [
"1.3.5"
],
"rack-protection": [
"1.1.4"
],
"raindrops": [
"0.8.0"
],
"rdiscount": [
"1.6.8"
],
"sinatra": [
"1.3.1"
],
"tilt": [
"1.3.3"
],
"unicorn": [
"4.1.1"
]
}
},
"services": {
"sysvinit": {
"nginx": {
"enable": true,
"ensureRunning": true,
"files": [
"/etc/nginx/sites-enabled/example",
"/etc/nginx/sites-available/example"
],
"packages": {
"apt": [
"nginx-common"
]
},
"sources": [
"/usr/local"
]
}
},
"upstart": {
"example": {
"enable": true,
"ensureRunning": true,
"files": [
"/etc/unicorn.conf.rb"
],
"sources": [
"/usr/local"
]
}
}
},
"sources": {
"/usr/local": "82789250f3ad94d4cfeacc0a8cabae2d26b0270e.tar"
}
}
}
},
"Properties": {
"ImageId": {
"Fn::FindInMap": [
"AWSRegionArch2AMI",
{
"Ref": "AWS::Region"
},
{
"Fn::FindInMap": [
"AWSInstanceType2Arch",
{
"Ref": "InstanceType"
},
"Arch"
]
}
]
},
"InstanceType": {
"Ref": "InstanceType"
},
"KeyName": {
"Ref": "KeyName"
},
"SecurityGroups": [
{
"Ref": "SecurityGroup"
}
],
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash\n",
"/opt/aws/bin/cfn-init -s ",
{
"Ref": "AWS::StackName"
},
" -r EC2Instance ",
" --access-key ",
{
"Ref": "HostKeys"
},
" --secret-key ",
{
"Fn::GetAtt": [
"HostKeys",
"SecretAccessKey"
]
},
" --region ",
{
"Ref": "AWS::Region"
},
"\n",
"/opt/aws/bin/cfn-signal -e $? '",
{
"Ref": "WaitHandle"
},
"'\n"
]
]
}
}
},
"Type": "AWS::EC2::Instance"
},
"HostKeys": {
"Properties": {
"UserName": {
"Ref": "CfnUser"
}
},
"Type": "AWS::IAM::AccessKey"
},
"SecurityGroup": {
"Properties": {
"GroupDescription": "SSH access only.",
"SecurityGroupIngress": [
{
"CidrIp": "0.0.0.0/0",
"FromPort": "22",
"IpProtocol": "tcp",
"ToPort": "22"
}
]
},
"Type": "AWS::EC2::SecurityGroup"
},
"WaitCondition": {
"DependsOn": "EC2Instance",
"Properties": {
"Handle": {
"Ref": "WaitHandle"
},
"Timeout": "600"
},
"Type": "AWS::CloudFormation::WaitCondition"
},
"WaitHandle": {
"Type": "AWS::CloudFormation::WaitConditionHandle"
}
}
}
The example running throughout this tutorial takes advantage of Blueprint’s default handling of /usr/local to package up an example web application. As an application grows, this can become muddled by other packages you install from source.
Blueprint doesn’t dictate how you deploy your applications but here are a few options play very nicely with Blueprint.
And remember to use templates to render configuration files appropriately in development, staging, and production.
Blueprints are stored in the local Git repository ~/.blueprints.git. Direct access isn’t typically needed but Blueprint comes with the blueprint-git(1) tool that simplifies the parameters to git(1) needed to use this repository.
Clone the entire local Git repository into blueprints in the working directory:
blueprint git clone
Show the diff, from Git’s point-of-view, between the previous two revisions of the example blueprint:
blueprint git show example
The JSON document is pretty-printed for storage so Git’s diffs will actually be meaningful.
DevStructure runs a free Blueprint Server at devstructure.com but this service will not be available after June 30th, 2012.
Running a Blueprint Server opens up the blueprint-push(1) and blueprint-pull(1) commands so you can store your blueprints remotely and share them with your team.
The Blueprint Server protocols and endpoints are documented for adventurous users that want to implement their own client or server.
Prerequisites:
Install the prerequisites from PyPI:
sudo pip install boto flask gunicorn
The rest of Blueprint Server comes bundled with Blueprint itself in the blueprint.io.server module.
Configure your Blueprint Server in /etc/blueprint.cfg:
[s3]
access_key = access_key
secret_key = secret_key
bucket = bucketConfigure your other servers to communicate with the Blueprint Server in /etc/blueprint.cfg:
[io]
server = serverSupervise Blueprint Server with Upstart on Ubuntu or CentOS 6:
description "Blueprint Server"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
env VIA=upstart
pre-start exec mkdir -p /var/log/blueprint-server
exec gunicorn -p/var/run/blueprint-server.pid -b0.0.0.0:5000 -w4 blueprint.io.server:app >>/var/log/blueprint-server/error.log 2>&1
# Mention /etc/blueprint.cfg so Blueprint will connect this service to that file.
Other platforms will have similar production configurations.
Start the server in development mode:
python /usr/lib/python2.7/dist-packages/blueprint/io/server/__init__.py
(Note that the pathname may be different on your system, particularly if you installed Blueprint from source or from PyPI.)
Start the server in production mode running in the foreground:
gunicorn blueprint.io.server:app
Some sort of process supervision, like Upstart as shown above, is recommended for production use.
First, start your Blueprint Server:
start blueprint-server
Push one of your blueprints to the Blueprint Server for safe-keeping or sharing with others:
blueprint push example
Pull a blueprint from the Blueprint Server to configure new development environments or extra production capacity:
blueprint pull example
Prerequisite: obtain a secret key via the GET /secret endpoint.
PUT /secret/name endpoint."sources" object in the blueprint via the PUT /secret/name/sha.tar endpoint.GET /secret/name endpoint."sources" object in the blueprint via the GET /secret/name/sha.tar endpoint.GET /secret/name/user-data.sh endpoint.Production instances can use blueprints without installing Blueprint (and therefore Git and friends) by asking the Blueprint I/O Server to generate POSIX shell code.
GET /secret/name/name.sh endpoint. The resulting program will fetch tarballs as needed via the GET /secret/name/sha.tar endpoint.sh name.sh interactively, at boot, or periodically via cron(8).GET /secretCreate a new secret key known only to the caller. This is the namespace beneath which the caller’s blueprints are stored.
Responses:
PUT /secret/nameStore the JSON representation of the blueprint name. The Content-Type of the body must be application/json.
Parameters:
/ characters.Responses:
GET /secret/nameFetch the JSON representation of the blueprint name.
Parameters:
/ characters.Responses:
PUT /secret/name/sha.tarStore a source tarball referenced by blueprint name. The Content-Type of the body must be application/x-tar.
Parameters:
/ characters.Responses:
GET /secret/name/sha.tarFetch a source tarball referenced by blueprint name.
Parameters:
/ characters.Responses:
application/x-tar content of the source tarball.GET /secret/name/name.shFetch the POSIX shell representation of the blueprint name. The resulting program will fetch tarballs as needed via the GET /secret/name/sha.tar endpoint.
Parameters:
/ characters.Responses:
GET /secret/name/user-data.shFetch POSIX shell commands to download and apply the blueprint name. EC2 instances provisioned from an AMI with cloud-init will execute this program if given as user data.
Parameters:
/ characters.Responses:
Blueprint is open-source, BSD-licensed software. Contributions are welcome via pull requests on GitHub.
irc.freenode.net#devstructureIf Blueprint’s not your cup of tea, check out the following tools. It’s much better to have some form of configuration management than none at all.