We have pretty good systems automation at Ning, but we’d like a better solution for bringing up new instances of our system and keeping various instances in sync. In addition to our live production environment, we have multiple development and testing instances of our system, and frequently need short lived instances for experiments. These system instances tend to get out of sync quickly, and bringing them back into sync, getting changes promoted correctly, and just managing them gets difficult.

To solve this problem we’re building a tool called Atlas which lets us define a template for systems which can be instantiated, torn down, upgraded, and so on. It makes use of our existing tooling, such as Chef and Galaxy, but will also allow for us to bring systems up in different environments, such as one of our datacenters, or a public cloud such as Rackspace or EC2.

In Atlas, a system is a set of servers which interact to provide a coherent set of services. For example, the set of services which, when deployed and running together, provide social network hosting for us is a system. A simpler one might be the combination of a server running WordPress and a server running MySQL in order to host this blog.

An environment is a set of servers, and related operational equipment, in which a system can run, such as EC2 or a datacenter.

A system template is an abstract description of a system, which includes all the logical components independent of the environment in which they might run.

An instance of a system is the realization of a system template in an environment.

The standard way of describing a system in Atlas is in Ruby. Let’s look at the system template for an example system:

system "example" do
 
  system "database" do
 
    server "master", :base => "postgres",
                     :role => "master"
 
    server "slave", :base => "postgres",
                    :role => "slave",
                    :cardinality => 2
  end
 
  system "appserver-cluster", :cardinality => ["clus1", "clus2"]  do
 
    server "appserver", :base => "server",
                        :install => "my-appserver-1.2.3",
                        :cardinality => 3
 
    server "memcached", :base => "server",
                        :install => "memcached-1.4.5",
                        :cardinality => 2
  end
 
  server "static-assets", :base => "server",
                          :install => "httpd-2.2.19",
                          :cardinality => 2
end

This template defines a top level system, example, which contains two subsystems, appserver-cluster and database, and two static-asset servers. The appserver-cluster system actually defines two clusters, clus1 and clus2 (the other higher cardinality systems are unnamed), each of which contains three appserver servers and two memcached instances.

If we look at a server instance, say

  server "static-assets", :base => "server",
                          :install => "httpd-2.2.19",
                          :cardinality => 2

We find a server has a name, and three attributes. The first, base is a label we’ll see more of when we look at an environment definition. The second, install instructs the system as to what services should be installed on the server. Finally we see cardinality which is simply how many of them we need. We defined cardinality as a number in this case, and as an array on the appserver-cluster — using a number means Atlas assigns a name in the derived system tree; using an array allows
you to specify names.

System templates are just that, templates for a system. They can be instantiated in an environment, which will provide concrete meanings to abstract elements in the system template, such as the base element. Environments are described very similarly to to system templates:

environment "ec2" do
 
  provisioner com.ning.atlas.ec2.EC2Provisioner, {
    :access_key => ENV["EC2_ACCESS_KEY"],
    :secret_key => ENV["EC2_SECRET_KEY"],
    :keypair_id => ENV["EC2_KEYPAIR_ID"]
  }
 
  initializer "chef-solo", com.ning.atlas.chef.UbuntuChefSoloInitializer, {
    :ssh_user     => "ubuntu",
    :ssh_key_file => ENV["EC2_PRIVATE_KEY"],
    :recipe_url   => "https://s3.amazonaws.com/waffles/bootstrap.tar.gz"
  }
 
  base "postgres", :ami => "ami-e2af508b",
                   :init => ['chef-solo:role[postgres]']
 
  base "server", :ami => "ami-e2af508b",
                 :init => ['chef-solo:role[server]']
end

An environment defines two things right now: provisioners and initializers. Provisioners know how to obtain bare servers, initializers know how to take bare servers to an initial state. This environment descriptor provisioners servers in EC2, and uses an initializer which installs chef-solo brings nodes up as specified in the base.

In the not distant future, environments will also have installers and investigators (we need a better name for these) which know how to install services (via galaxy or cast, for instance), and how to interrogate a deployed system and match logical existing instances to elements in the deployment tree (required for managing real systems which change outside of Atlas).

Environments can be hierarchical, just like system templates, allowing for things like provisioning a database into Amazon’s RDS instead of bringing up a server. This would be done by creating a sub-environment with a different provisioner and initializer.

The details for how systems are instantiated into environments, upgraded, and so on will need to come in a later post. This is all very young software — a couple of us have been thinking about it, and have done a bit of work, but it is far from usable yet for non-trivial examples.

We decided to do something a bit different in building this particular project: to actually build it in the open, rather than open sourcing it when it hit some completion milestone. The code is all available right now, under the Apache License, 2.0, at http://github.com/ning/atlas and we have a (brand new) mailing list for discussing how things should work. Our efforts are completely focused on meeting our internal needs for this system right now, but the eventual goal is to support arbitrary systems and environments.