dtrace PID provider vs. libraries loaded by python

As I was doing some work on beadm, I really wanted to know who was mucking with properties on datasets. It looked as though I could do something like:

# dtrace -qn 'pid$target:libzfs.so.1:zfs_prop_set:entry / copyinstr(arg0 + 8) == $1 /
{
    printf("%s(%s,%s,%s)\n", probefunc, copyinstr(arg0 + 8), copyinstr(arg1), copyinstr(arg2));
    ustack();
}' -c 'beadm create test' $dataset

But that doesn't work:

... probe description pid12942:libzfs.so.1:zfs_prop_set:entry does not match any probes

My guess is because that this is because the python executable is not linked against libzfs - a very good thing. To get around this, we need to preload libzfs. For example:

# LD_PRELOAD_32=libzfs.so.1 dtrace -qn 'pid$target:libzfs.so.1:zfs_prop_set:entry / copyinstr(arg0 + 8) == $1 /
{
    printf("%s(%s,%s,%s)\n", probefunc, copyinstr(arg0 + 8), copyinstr(arg1), copyinstr(arg2));
    ustack();
}' -c 'beadm create test' rpool/zones/z1/rpool/ROOT/zbe-1
zfs_prop_set(rpool/zones/z1/rpool/ROOT/zbe-1,mountpoint,legacy)

              libzfs.so.1`zfs_prop_set
              libbe.so.1`be_unmount_zone_root+0xa9
              libbe.so.1`be_update_zone_vfstab+0x106
              libbe.so.1`be_copy_zones+0x9d9
              libbe.so.1`be_copy+0x8f2
              libbe_py.so`beCopy+0x245
...

The only change here was the addition of LD_PRELOAD_32=libzfs.so.1 at the beginning of the command. LD_PRELOAD would have worked as well, but it wouldn't work if a different library that does not have a 64-bit variant was being traced. That is because dtrace would fail to start because the 64-bit library was not able to be found. I found this when I needed to poke around at calls within libbe.so.1, which is 32-bit only.

ksh93 backtraces

It is often times handy to get a backtrace to help you understand how a program got to an error condition. Unfortunately, shell scripting languages tend to not provide an easy mechanism to get a backtrace. The following examines how it can be accomplished with ksh93.

#! /bin/ksh

function backtrace {
        typeset -a stack 
        # Use "set -u" and an undefined variable access in a subshell
        # to figure out how we got here.  Each token of the result is
        # stored as an element in an indexed array named "stack".
        set -A stack $(exec 2>&1; set -u; unset __unset__; echo $__unset__)

        # Trim the last entries in stack array until we find the one that
        # matches the name of this function.
        typeset i=0
        for (( i = ${#stack[@]} - 1; i >= 0; i-- )); do
                [[ "${stack[i]}" == "${.sh.fun}:" ]] && break
        done

        # Print the name of the function that called this one, stripping off
        # the [lineno] and appending any arguments provided to this function.
        print -u2 "${stack[i-1]/\[[0-9]*\]} $*"
        # Print the backtrace.
        for (( i--; i >= 0; i-- )); do
                print -u2 "\t${stack[i]%:}"
        done
}

# A couple functions to illustrate the output
function a {
        b "$@"
}

function b {
        c "$@"
}

function c {
        # Trigger a backtrace and exit if no arguments were passed
        (( $# == 0 )) && backtrace "missing arg" && exit 1
        print -- "$@"
}

a "$@"

A couple example runs:

$ ./backtrace.ksh hello world!
hello world!
$ ./backtrace.ksh
c: missing arg
        c[37]
        b[32]
        a[28]
        ./backtrace.ksh[41]

Connecting to twist server

Once your development environment is set up, it is useful to know what is required to connect to the twist server.

#! /opt/opsware/smopython2/python2

import sys
sys.path.append("/opt/opsware/smopylibs2")
from pytwist import *

# Establish an unauthenticated connection to twist server.  Note that the
# hostname "twist" must resolve either through /etc/hosts or DNS.
ts = twistserver.TwistServer()

In the event the hostname twist does not resolve or points to the wrong twist server, you can do...

ts = twistserver.TwistServer("twist.mycompany.com")

See the TwistServer Method Syntax section on page 75 of the Developer's Guide for other options that may be important to you.

It is also quite likely that you will need to do an authenticated session. This is covered in the Error Handling section on page 76 of the Developer's Guide. In order to prompt for a username and password, I have found something like the following works well...

def authenticate(server):
        for tries in range(1,3):
                try:
                        sys.stdout.write("HPSA login: ")
                        user = sys.stdin.readline().strip()
                        pw = getpass.getpass("HPSA password: ")
                        server.authenticate(user, pw)
                        return True
                except:
                        sys.stderr.write("Authentication failed.\n")
        return False

...
ts = twistserver.TwistServer()
authenticate(ts)

Opsware/HPSA API: Getting Started

The HP Server Automation Platform Developer's Guide (for HPSA 7.50, September, 2008) and any other resources I've been able to find are far from complete and sometimes misleading with respect to using the HPSA API. As I've tried to clear up my confusion with the API, google has helped me exactly 0% of the time. Maybe my rambling on the subject will lead to someone leading me to a better approach...

In this first post on the subject, let's start out with enough to get a working Python environment. Chapter 3 of the developer's guide indicates that Pytwist relies on Python 1.5.2. Ugh. However, pytwist can be used with Python 2.4. Rather than following the advice to install the /Opsware/Tools/Opsware API Access software policy, instead install the following policies:

• /Opsware/Tools/Python 2/Python 2 for Server Modules
• /Opsware/Tools/Python 2 Opsware API Access for Server Modules/Python 2 Opsware API Access for Server Modules

The permissions are likely such that you need to be root to execute. Fix this with:

# chmod -R a+r /opt/opsware/smopython2
# exit
$ /opt/opsware/smopython2/python -V
Python 2.4.4

Ahhh, much better.

Notice the path to python above. It is actually a shell script that sets PYTHON_HOME then tries to execute python. Unfortunately, Solaris doesn't like a shebang line in one script referring to another script. The resulting error looks like:

$ ./twister1 
import: Unable to connect to X server ().
./twister1: line 4: syntax error near unexpected token `"/opt/opsware/smopylibs2"'
./twister1: line 4: `sys.path.append("/opt/opsware/smopylibs2")'

In other words, the shebang (#!) line python scripts will be useless as the software policy installs it.

Workaround 1:

$ export PYTHONHOME=/opt/opsware/smopython2/34c.0.0.5.31-1

Then use a shebang line like:

#! /opt/opsware/smopython2/34c.0.0.5.31-1/bin/python2

Workaround 2: Compile this program and put it at /opt/opsware/smoptyhon2/python2

#include <stdlib.h>
#include <unistd.h>

#define PYTHONHOME "/opt/opsware/smopython2/34c.0.0.5.31-1"

int main(int argc, char **argv) {
        char *python = PYTHONHOME "/bin/python2";

        argv[0] = python;
        setenv("PYTHONHOME", PYTHONHOME, 1);

        return execv(python, argv);
}

Then use a shebang line like:

#! /opt/opsware/smopython2/python2

In any case (even if you used Python 1.5 like the developer's guide suggests) the path to python does not match reality. Adjust your scripts accordingly.

Shell Programming and PATH

As most readers of this blog will already know, the PATH environment variable is used to locate commands that are executed. Key things to remember as you read this post are:

• Environment variables (including PATH) are inherited by child processes
• Child processes are unaffected by the parent process subsequently changing PATH to something else

So what's the big deal? Suppose you have a shell script that calls ps -fe. It works great for you because you have /usr/bin first in your PATH. However, the guy down the hall that cut his teeth on a BSD system has /usr/ucb first. If your shell script does not set PATH=/usr/bin:... prior to calling ps, your shell script will work for you but give strange errors for the guy down the hall. Of course, your shell script could just specify /usr/bin/ps -fe...

This brings up four different styles that are seen...

Style 1: Just hope for the best

#! /usr/bin/ksh

count=$(ps -fe | wc -l)
echo "There are $count processes running"

Style 2: Specify full path whenever calling a program

#! /usr/bin/ksh

count=$(/usr/bin/ps -fe | wc -l)
/usr/bin/echo "There are $count processes running"

Style 3: Create variables to store full path to all programs

#! /usr/bin/ksh

PS=/usr/bin/ps
WC=/usr/bin/wc
ECHO=/usr/bin/echo

count=$($PS -fe | $WC -l)
$ECHO "There are $count processes running"

Style 4: Set PATH to use the commands you want to use

#! /usr/bin/ksh

export PATH=/usr/bin
count=$(ps -fe | wc -l)
echo "There are $count processes running"

With Style 1, the script is only reliable for the subset of users that have the right version of ps first in their PATH.

A workaround for this is shown in Style 2. However, this example has an intentional problem that is somewhat common when this approach is used. Notice that wc is not specified by its full path. This will work fine until someone with a really messed up (or unset) PATH tries to execute the script.

Style 3 fixes the ps and wc problems, but introduces another small problem: it forces a fork() and exec*() to run something that could be more efficiently done via a built-in. I'll talk more about this in a future post.

Style 4 keeps the simplicity of Style 1, but ensures that each user will get the same version of the commands. The author of the script can tailor PATH to contain the minimum set to find the required commands and test the script to gain a high degree of confidence that the script will work for others.

I have a strong preference for Style 4. Performing shell programming retains the feel of using a shell interactively, keeps the code understandable, and performs reliably. But this doesn't mean that it is always the right thing to do. Consider the batch command. It doesn't set PATH and it is very correct in not doing so. That is, if the following:

exec /usr/bin/at -qb $*

were replaced with

PATH=/usr/bin; export PATH
...
at -qb $*

This would change the environment that at(1) attaches to the job - potentially breaking it.

Shell Programming and temporary files

I've been generally impressed with the coding standards that are enforced on C code in OpenSolaris, as checked by cstyle. However, similar automated checks don't exist for shell scripts. This, combined with a history of "/bin/sh is the one true shell", has led to inefficient scripts that very commonly have security vulnerabilities.

I'm hoping to breathe some life into this blog with a series of posts that describe some of the common problems and some potential solutions. I'll start with problems related to security problems with temporary files.

Consider the following bit of code taken from /usr/lib/lp/bin/lpadmin.

   293  # Do the LP configuration for a local printer served by lpsched
   294  if [[ -x ${LPADMIN} && -n "${local}" ]] ; then
   295          # enumerate LP configured printers before modification
   296          PRE=/tmp/lpadmin-pre.$$
   297          (/bin/ls /etc/lp/printers 2>/dev/null ; /bin/ls /etc/lp/classes \
   298                  2>/dev/null) >${PRE}

There are two problems here:

1. A symbolic link vulnerability exists.
2. The TMPDIR environment variable is not respected. If a user has a reason to want temporary files stored in a particular file system, utilities should respect the user's wishes.

The solution

For quite some time, Solaris has provided the mktemp command has existed to facilitate the secure creation of files. In the fix for this problem, I used mktemp in a way that both creates the file securely and respects TMPDIR.

     41 MKTEMP="/usr/bin/mktemp -t"
      ...
    300  PRE=$(${MKTEMP} lpadmin-pre.XXXXXX)
    301  if [[ -z "${PRE}" ]] ; then
    302   gettext "lpadmin: System error; cannot create temporary file\n" 1>&2
    303   exit 2
    304  fi
    305 
    306  (/bin/ls /etc/lp/printers 2>/dev/null ; /bin/ls /etc/lp/classes \
    307   2>/dev/null) >${PRE}

Installation of Studio 11 on Nevada

The OpenSolaris Sun Studio installation page forgets to mention that the default version of Java is incompatible with the Studio installer. The following command works for me:

# export JAVA_HOME=/usr/jdk/instances/jdk1.5.0
# export PATH=/usr/jdk/instances/jdk1.5.0/bin:$PATH
# ./installer