Scripting your system administration

Scripting your system administration

As a systems administrator you’re often called on to make changes in a consistent way across a network for machines. While Linux can be updated by various command line commands, scripting the changes you make is useful for two reasons:

  1. If you write your planned changes into a script, you are at least minimally documenting the changes you’re implemented.

  2. Scripted changes are reproducible.

Ideally, however, scripts used for systems administration should be idempotent, that is running the script twice should not run in an error. For example:

yum install -y expect

results in the tool expect being installed. Running this command twice will cause no harm, since the final state (expect is installed) will be the same. By contrast:

echo myhostname.domain >>/etc/ssh/shosts.equiv

is not idempotent, as each time that it runs it adds a hostname to the shosts.equiv file. To stop this from happening you could add a test to your script:

HOSTNAME=myhostname.domain
SHOSTS_FILE=/etc/ssh/shosts.equiv
if [ ! -e $SHOSTS_FILE ] ||  grep $HOSTNAME $SHOSTS_FILE >/dev/null ; then
    echo $HOSTNAME >>$SHOSTS_FILE
fi

Automating interaction with expect

While shell scripts are invaluable for automating systems administration tasks they cannot interact with tools that expect an interactive session. For example, if you want to automate resetting the password of a list of users, you could supply the new passwords in a text file, but the passwd expects a user interacting on a terminal. Tools such as the passwd, scp and sudo commands also pose a challenge for automation. To automate interactive tasks you can use expect. This is not installed by default on Scientific Linux so install it using:

yum install -y expect

Expect is an extension of the TCL language, so expect scripts are special-purpose TCL programs. Here is an example of an expect script that, when run as root, will change a user’s password:

#!/usr/bin/expect

if { $argc != 1 } {
  puts "Usage: $argv0 <username>"
  exit 1
}

set password ThisIsAStr1ngePassword
set username [lindex $argv 0]

spawn passwd $username
expect "password: "
send "$password\r"
expect "new password: "
send "$password\r"
interact

The key expect commands are:

  • spawn: this starts a command
  • expect: this waits for a pattern to appear in the command’s output
  • send: this sends a string to the command
  • interact: this switches to interactive mode and is normally used at the end of a script when all scripted interaction has finished and you just want the spawned command to run to completion.

For more information on expect you can read about Using Expect Scripts to Automate Tasks or consult the Tcl wiki or the mini-reference manual.

In the above example, the password was included in the file. This obviously isn’t appropriate for real world use. The password could have been passed in as a command line argument, like the username was, but that would mean that it would be (momentarily) visible to anyone that ran ps on the machine where this script ran.

Alternatively the password could be set in an environment variable e.g. using export NEWPASSWORD=AStrangPassw8rd in the shell where the script is being run. This is safe because it doesn’t show up in ps. Then in the expect script you can retrieve the value:

if { [ info exists env(NEWPASSWORD) ] } {
  set password $::env(NEWPASSWORD)
} else {
  puts "Please set the new password in the NEWPASSWORD environment variable"
  exit 1
}

Finally you can read data from a file, as this (rather more complex) example illustrates:

#!/usr/bin/expect

proc setpassword {username password} {
    catch {
      spawn passwd $username
      expect "password: "
      send "$password\r"
      expect "new password: "
      send "$password\r"
      interact
   }
}

if { $argc != 1 } {
  puts "Usage: $argv0 <user/password list>"
  exit 1
}

set fp [open [lindex $argv 0] r]
set file_data [read $fp]
close $fp

set lines [split $file_data "\n"]
foreach line $lines {
  set fields [split $line]
  if { [llength $fields] == 2 } {
    set username [lindex $fields 0]
    set password [lindex $fields 1]
    setpassword $username $password
  }
}

Of course as an alternative to this scripting you can use chpasswd to set passwords from a file. There are many cases (e.g. interacting with a remote network device via ssh) where expect scripts are useful, however.

Scripts can be run on remote servers using ssh, and if public key authentication is used, this can be done without any password prompts. If, however, you want to use sudo to execute some commands as the root user, expect comes in very handy.

The following script copies a tar archive of the ABySS assembler to a remote server and unpacks it as root:

#!/bin/bash

# exit on error
set -e

function cleanup() {
    if [ -e tmpscript ] ; then
      rm tmpscript
    fi
}

USER=pvh
ARCHIVE=abyss-1.5.2-x86_64.tar.bz2
HOST=sl1.sanbi.ac.za
PASSWORD=AVeryFinePassword
TMPDIR=/var/tmp
SCRIPT_LOC=$TMPDIR/install.expect

SCRIPT="spawn sudo tar -C /opt -xf $TMPDIR/$ARCHIVE\nexpect \"* password for $USER:\"\nsend \"$PASSWORD\\\r\"\ninteract"

trap cleanup EXIT
umask 077
echo -e $SCRIPT >tmpscript
scp -p tmpscript $HOST:$SCRIPT_LOC
scp $ARCHIVE $HOST:$TMPDIR
ssh $HOST expect -f $SCRIPT_LOC
ssh $HOST rm -f $SCRIPT_LOC

Note that the expect script is set up using a string and then written to a local temporary location. The use of umask and trap ensures that this sensitive script is not readable to other users and is removed on exit. Also note the use of \r: this is the carriage return character that is required to actually submit the password to sudo.

Scripting sysadmin tasks with ansible

Ansible is another tool for automating systems administration tasks, written on a higher level than expect. Whereas expect allows you to script the interaction with commands in great detail, ansible provides modules (written in Python) to perform common tasks such as installing packages, running commands and moving files. Ansible also allows scripting playbooks to specify complex configurations, but we won’t be covering that.

If you have EPEL installed on your machine you can install ansible using:

sudo yum install -y ansible

Before ansible can work on a server, it needs to know of its existence, so the hostname of the machien must be added to an ansible inventory. The system inventory is /etc/ansible/hosts and follows an INI-file type format. For example:

server1.example.com
server2.example.com

[compute]
worker1.example.com
worker2.example.com

Bare lines (such as server1 in the above example) describe individual machines, and lines in [] describe groups. You can also provide your own inventory file using the -i command line flag.

Here is an example ansible command that reboots all the machines in group compute:

ansible -k -K -s -m command -a reboot compute

The meaning of the commandline flags is:

  • -k – ask for ssh password
  • -K – ask for sudo password
  • -s – run command using sudo
  • -m – specify the module to run
  • -a – specify arguments for the module
  • -u – remote username to log in as (if it is different to the username you are logged in as locally)

When the command is executed, ansible connects to the machines in group compute using ssh and runs sudo reboot. Note that commands run by ansible are non-interactive. So yum upgrade -y is correct, yum upgrade is not.

Ansible provides a wide variety of modules, consult the module index for a list. Some of the more useful are:

  • command – executes commands on the remote host
  • yum – installs or removes packages using yum
  • copy – copies files to remote locations
  • file – sets permissions and other file attributes
  • git – deploys files from git checkouts
  • group – adds or removes groups
  • lineinfile – adds or removes lines in files
  • pip – installs or removes python packages using pip
  • shell – runs a command on the remote host with shell expansions
  • script – runs a local script on the remote host
  • user – adds and removes users

Here is the script, listed above, that installs ABySS from a tarball, but now rewritten to use ansible.

#!/bin/sh

ARCHIVE=abyss-1.5.2-x86_64.tar.bz2
HOST=sl1.sanbi.ac.za
TMPDIR=/var/tmp

ansible -m copy -a "src=$ARCHIVE dest=$TMPDIR" $HOST
ansible -s -K -m command -a "tar -C /opt -xf $TMPDIR/$ARCHIVE" $HOST

Note that this script will ask for the sudo password for the remote host, so it can’t be used non-interactively. You can specify the sudo password in the inventory file as a variable, however, although this carries with it all the security risks of saving passwords in plaintext:

[sl.sanbi.ac.za]
sl1.sanbi.ac.za ansible_sudo_pass=AnAmazingPassword
sl2.sanbi.ac.za

If you do want to use this feature, it is best to do so in a custom inventory file that you can set to permissions 0600. Then the above script would be changed to:

#!/bin/sh

ARCHIVE=abyss-1.5.2-x86_64.tar.bz2
HOST=sl1.sanbi.ac.za
TMPDIR=/var/tmp

ansible -i myinventory.txt -m copy -a "src=$ARCHIVE dest=$TMPDIR" $HOST
ansible -i myinventory.txt -s -m command -a "tar -C /opt -xf $TMPDIR/$ARCHIVE" $HOST

You can of course modify this script to apply to a group instead of an individual host, in which case you could use a group variable.

For more about ansible, get in touch with the Ansible community.

Leave a Reply

Your email address will not be published. Required fields are marked *