ShutIt

Automation framework for programmers

Description

ShutIt is an automation tool that models a user's actions on a terminal.

It can automate any process that can be run by a human on the command line with little effort.

It was originally written to manage complex Docker builds, but is a now general-purpose automation tool that supports bash, Docker, Vagrant, ssh and arbitrary build contexts.

If you want to know more about Docker, see the official site or take a look at the book by the creators of ShutIt - Docker in Practice.

ShutIt is also an educational tool, as it can produce videos of demos, capture reproducible steps required to set environments up, and even challenge you to get the right output (see grep-scales).

Code Quality Rank: L2

Programming language: Python

License: MIT License

Tags: Build Tools     DevOps Tools    

Latest version: v1.0.162

README

ShutIt

A versatile automation framework.

ShutIt is an automation tool that models a user's actions on a terminal.

It can automate any process that can be run by a human on the command line with little effort.

It was originally written to manage complex Docker builds, but is a now general-purpose automation tool that supports bash, Docker, Vagrant, ssh and arbitrary build contexts.

ShutIt can also be used as an educational tool, as it can produce videos of demos, capture reproducible steps required to set environments up, and even challenge you to get the right output (see grep-scales).

If you want to know more about Docker, see the official site or take a look at the book by the creators of ShutIt - Docker in Practice.

Really Quick Overview

Some use cases:

• You like bash, want to automate tasks, have structure and support, but don't want to learn a configuration management framework that takes you away from the command line you know and love.

• Want to create complex Vagrant environments to model clusters of machines.

• Want to create instructive walkthroughs:

• Are interested in "phoenix deployment".

• Want to take your scripts and turn them into stateless containers quickly, without needing to maintain (or learn) a configuration management solution designed for moving-target systems.

• You're programmer who wants highly configurable stateless containers development, testing, and production.

• Want to build everything from source in a way that's comprehensible and auditable.

What Does it Do (bash Builds)?

ShutIt acts as a modular and easy to use wrapper around pexpect.

Here is a simple example of a script that creates a file and a directory if they are not there already:

What Does it Do (Tutorials)?

This builds on the docker features (see below), but allows you to interrupt the run at points of your choosing with 'challenges' for the user to overcome.

Two types of 'challenge' exist in ShutIt:

• scales
• free form

Scales tell you to run a specific command before continuing. This is useful when you want to get certain commands or flags 'under your fingers', which does not happen without dedicated and direct practice.

Free form exercises give you a task to perform, and free access to the shell. This is to give the user a realistic environment in which to hone their skills. You can check man pages, look around the directories, search for useful utils (even install new ones!). When you are finished, a pre-specified command is run to check the system is in an appropriate state. Here's an example for the [basics of git](ianmiell.github.io/git-101-tutorial/):

If you use a Docker-based tutorial and you mess the environment up, the state can be restored to a known one by hitting CTRL-G.

What Does it Do (Vagrant)?

Uses a bash build to set up n vagrant machines, and uses Landrush to give them useful hostnames accessible from the hosts and in the guest VMs.

It supports both Virtualbox and Libvirt providers.

This allows another kind of contained environment for more infrastructural projects than Docker allows for.

This example demonstrates a reproducible build that sets up Docker on an Ubuntu VM (on a Linux host), then runs a CentOS image within Docker within the Ubuntu VM.

It deposits the user into a shell mid-build to interrogate the environment, after which the user re-runs the build to add a directive to ensure ps is installed in the image.

What Does it Do (Docker Builds)?

We start with a "ShutIt Module", similar to a shell script, or a Dockerfile (see bash builds above).

In the image above there are five of these. At a high level they each have the following attributes:

• a list of zero or more dependencies on other modules
• a unique number that represents its ordering within the available modules
• a set of steps (bash commands) for building the module

In the image we imagine a scenario where we want to build our blog into a docker image, with all its attendant content and config.

We instruct ShutIt to build the MyBlog module, and it runs the build as per the image on the right.

The container environment is set up, the modules are ordered, and the build steps are run. Finally, the image is committed, tagged and pushed as configured.

This is a core function of ShutIt - to manage dependencies and image building for complex image setups.

But it doesn't just run build steps, it also manages The ShutIt Lifecycle to make the build more robust and flexible.

The ShutIt Lifecycle

• gathers all the modules it can find in its path and determines their ordering
• for all modules, it gathers any build-specific config (e.g. passwords etc.)
• it checks dependencies and conflicts across all modules and figures out which modules need to be built
• for all modules, it checks whether the module is already installed
• for all modules, if it needs building, it runs the build
• for all modules, run a test cycle to ensure everything is as we expect
• for all modules, run a finalize function to clean up the container
• do any configured committing, tagging and pushing of the image

These correspond to the various functions that can be implemented in the ShutIt module file.

Auto-Generate Modules

ShutIt provides a means for auto-generation of modules (either bare ones, or from existing Dockerfiles) with its skeleton command. See here for an example.

Docker Image

An image with ShutIt in it is available here as docker pull ianmiell/shutit

Really Quick Start

Full User Guide

API

Installation

Contributing

We always need help, and with a potentially infinite number of libraries required, it's likely you will be able to contribute. Just mail [email protected] if you want to be assigned a mentor. He won't bite

Tests

Mailing List

https://groups.google.com/forum/#!forum/shutit-users [email protected]

Known Issues

Since a core technology used in this application is pexpect - and a typical usage pattern is to expect the prompt to return. Unusual shell prompts and escape sequences have been known to cause problems. Use the shutit.setup_prompt() function to help manage this by setting up a more sane prompt. Use of COMMAND_PROMPT with echo -ne has been seen to cause problems with overwriting of shells and pexpect patterns.

Licence

The MIT License (MIT)

Copyright (C) 2014 OpenBet Limited

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.