Mike Gerdts

Oracle Solaris 11 Summit Day at USENIX LISA 2011

2011-11-17T23:08:00.001-06:00

LISA '11 is just around the corner and once again includes an Oracle Solaris Summit the day before the main conference. Please come to the summit and to as my esteemed colleagues and I introduce many of the great improvements found in Solaris 11. I'll be giving a talk on Zones.

Even with a full day to talk about Solaris 11, we'll certainly be unable to get into the depth in the areas that concern you the most. To get some face time with Oracle engineers, stop by the Oracle demo booth - I'll be there Wednesday from 2:00 - 4:00.

If I have any updates, I'll be posting them to The Zones Zone.

The Zones Zone

2011-11-09T21:55:00.000-06:00

New posts related to Solaris can be found at The Zones Zone

dtrace PID provider vs. libraries loaded by python

2011-01-03T17:07:00.000-06:00

As I was doing some work on beadm, I really wanted to know who was mucking with properties on datasets. It looked as though I could do something like:

# dtrace -qn 'pid$target:libzfs.so.1:zfs_prop_set:entry / copyinstr(arg0 + 8) == $1 /
{
    printf("%s(%s,%s,%s)\n", probefunc, copyinstr(arg0 + 8), copyinstr(arg1), copyinstr(arg2));
    ustack();
}' -c 'beadm create test' $dataset

But that doesn't work:

... probe description pid12942:libzfs.so.1:zfs_prop_set:entry does not match any probes

My guess is because that this is because the python executable is not linked against libzfs - a very good thing. To get around this, we need to preload libzfs. For example:

# LD_PRELOAD_32=libzfs.so.1 dtrace -qn 'pid$target:libzfs.so.1:zfs_prop_set:entry / copyinstr(arg0 + 8) == $1 /
{
    printf("%s(%s,%s,%s)\n", probefunc, copyinstr(arg0 + 8), copyinstr(arg1), copyinstr(arg2));
    ustack();
}' -c 'beadm create test' rpool/zones/z1/rpool/ROOT/zbe-1
zfs_prop_set(rpool/zones/z1/rpool/ROOT/zbe-1,mountpoint,legacy)

              libzfs.so.1`zfs_prop_set
              libbe.so.1`be_unmount_zone_root+0xa9
              libbe.so.1`be_update_zone_vfstab+0x106
              libbe.so.1`be_copy_zones+0x9d9
              libbe.so.1`be_copy+0x8f2
              libbe_py.so`beCopy+0x245
...

The only change here was the addition of LD_PRELOAD_32=libzfs.so.1 at the beginning of the command. LD_PRELOAD would have worked as well, but it wouldn't work if a different library that does not have a 64-bit variant was being traced. That is because dtrace would fail to start because the 64-bit library was not able to be found. I found this when I needed to poke around at calls within libbe.so.1, which is 32-bit only.

ksh93 backtraces

2011-01-02T21:15:00.004-06:00

It is often times handy to get a backtrace to help you understand how a program got to an error condition. Unfortunately, shell scripting languages tend to not provide an easy mechanism to get a backtrace. The following examines how it can be accomplished with ksh93.

#! /bin/ksh

function backtrace {
        typeset -a stack 
        # Use "set -u" and an undefined variable access in a subshell
        # to figure out how we got here.  Each token of the result is
        # stored as an element in an indexed array named "stack".
        set -A stack $(exec 2>&1; set -u; unset __unset__; echo $__unset__)

        # Trim the last entries in stack array until we find the one that
        # matches the name of this function.
        typeset i=0
        for (( i = ${#stack[@]} - 1; i >= 0; i-- )); do
                [[ "${stack[i]}" == "${.sh.fun}:" ]] && break
        done

        # Print the name of the function that called this one, stripping off
        # the [lineno] and appending any arguments provided to this function.
        print -u2 "${stack[i-1]/\[[0-9]*\]} $*"
        # Print the backtrace.
        for (( i--; i >= 0; i-- )); do
                print -u2 "\t${stack[i]%:}"
        done
}

# A couple functions to illustrate the output
function a {
        b "$@"
}

function b {
        c "$@"
}

function c {
        # Trigger a backtrace and exit if no arguments were passed
        (( $# == 0 )) && backtrace "missing arg" && exit 1
        print -- "$@"
}

a "$@"

A couple example runs:

$ ./backtrace.ksh hello world!
hello world!
$ ./backtrace.ksh
c: missing arg
        c[37]
        b[32]
        a[28]
        ./backtrace.ksh[41]

Connecting to twist server

2010-04-03T21:50:00.004-05:00

Once your development environment is set up, it is useful to know what is required to connect to the twist server.

#! /opt/opsware/smopython2/python2

import sys
sys.path.append("/opt/opsware/smopylibs2")
from pytwist import *

# Establish an unauthenticated connection to twist server.  Note that the
# hostname "twist" must resolve either through /etc/hosts or DNS.
ts = twistserver.TwistServer()

In the event the hostname twist does not resolve or points to the wrong twist server, you can do...

ts = twistserver.TwistServer("twist.mycompany.com")

See the TwistServer Method Syntax section on page 75 of the Developer's Guide for other options that may be important to you.

It is also quite likely that you will need to do an authenticated session. This is covered in the Error Handling section on page 76 of the Developer's Guide. In order to prompt for a username and password, I have found something like the following works well...

def authenticate(server):
        for tries in range(1,3):
                try:
                        sys.stdout.write("HPSA login: ")
                        user = sys.stdin.readline().strip()
                        pw = getpass.getpass("HPSA password: ")
                        server.authenticate(user, pw)
                        return True
                except:
                        sys.stderr.write("Authentication failed.\n")
        return False

...
ts = twistserver.TwistServer()
authenticate(ts)

Opsware/HPSA API: Getting Started

2010-04-03T20:55:00.006-05:00

The HP Server Automation Platform Developer's Guide (for HPSA 7.50, September, 2008) and any other resources I've been able to find are far from complete and sometimes misleading with respect to using the HPSA API. As I've tried to clear up my confusion with the API, google has helped me exactly 0% of the time. Maybe my rambling on the subject will lead to someone leading me to a better approach...

In this first post on the subject, let's start out with enough to get a working Python environment. Chapter 3 of the developer's guide indicates that Pytwist relies on Python 1.5.2. Ugh. However, pytwist can be used with Python 2.4. Rather than following the advice to install the /Opsware/Tools/Opsware API Access software policy, instead install the following policies:

/Opsware/Tools/Python 2/Python 2 for Server Modules
/Opsware/Tools/Python 2 Opsware API Access for Server Modules/Python 2 Opsware API Access for Server Modules

The permissions are likely such that you need to be root to execute. Fix this with:

# chmod -R a+r /opt/opsware/smopython2
# exit
$ /opt/opsware/smopython2/python -V
Python 2.4.4

Ahhh, much better.

Notice the path to python above. It is actually a shell script that sets PYTHON_HOME then tries to execute python. Unfortunately, Solaris doesn't like a shebang line in one script referring to another script. The resulting error looks like:

$ ./twister1 
import: Unable to connect to X server ().
./twister1: line 4: syntax error near unexpected token `"/opt/opsware/smopylibs2"'
./twister1: line 4: `sys.path.append("/opt/opsware/smopylibs2")'

In other words, the shebang (#!) line python scripts will be useless as the software policy installs it.

Workaround 1:

$ export PYTHONHOME=/opt/opsware/smopython2/34c.0.0.5.31-1

Then use a shebang line like:

#! /opt/opsware/smopython2/34c.0.0.5.31-1/bin/python2

Workaround 2: Compile this program and put it at /opt/opsware/smoptyhon2/python2

#include <stdlib.h>
#include <unistd.h>

#define PYTHONHOME "/opt/opsware/smopython2/34c.0.0.5.31-1"

int main(int argc, char **argv) {
        char *python = PYTHONHOME "/bin/python2";

        argv[0] = python;
        setenv("PYTHONHOME", PYTHONHOME, 1);

        return execv(python, argv);
}

Then use a shebang line like:

#! /opt/opsware/smopython2/python2

In any case (even if you used Python 1.5 like the developer's guide suggests) the path to python does not match reality. Adjust your scripts accordingly.

Shell Programming and PATH

2009-06-19T21:59:00.002-05:00

As most readers of this blog will already know, the PATH environment variable is used to locate commands that are executed. Key things to remember as you read this post are:

Environment variables (including PATH) are inherited by child processes
Child processes are unaffected by the parent process subsequently changing PATH to something else

So what's the big deal? Suppose you have a shell script that calls ps -fe. It works great for you because you have /usr/bin first in your PATH. However, the guy down the hall that cut his teeth on a BSD system has /usr/ucb first. If your shell script does not set PATH=/usr/bin:... prior to calling ps, your shell script will work for you but give strange errors for the guy down the hall. Of course, your shell script could just specify /usr/bin/ps -fe...

This brings up four different styles that are seen...

Style 1: Just hope for the best

#! /usr/bin/ksh

count=$(ps -fe | wc -l)
echo "There are $count processes running"

Style 2: Specify full path whenever calling a program

#! /usr/bin/ksh

count=$(/usr/bin/ps -fe | wc -l)
/usr/bin/echo "There are $count processes running"

Style 3: Create variables to store full path to all programs

#! /usr/bin/ksh

PS=/usr/bin/ps
WC=/usr/bin/wc
ECHO=/usr/bin/echo

count=$($PS -fe | $WC -l)
$ECHO "There are $count processes running"

Style 4: Set PATH to use the commands you want to use

#! /usr/bin/ksh

export PATH=/usr/bin
count=$(ps -fe | wc -l)
echo "There are $count processes running"

With Style 1, the script is only reliable for the subset of users that have the right version of ps first in their PATH.

A workaround for this is shown in Style 2. However, this example has an intentional problem that is somewhat common when this approach is used. Notice that wc is not specified by its full path. This will work fine until someone with a really messed up (or unset) PATH tries to execute the script.

Style 3 fixes the ps and wc problems, but introduces another small problem: it forces a fork() and exec*() to run something that could be more efficiently done via a built-in. I'll talk more about this in a future post.

Style 4 keeps the simplicity of Style 1, but ensures that each user will get the same version of the commands. The author of the script can tailor PATH to contain the minimum set to find the required commands and test the script to gain a high degree of confidence that the script will work for others.

I have a strong preference for Style 4. Performing shell programming retains the feel of using a shell interactively, keeps the code understandable, and performs reliably. But this doesn't mean that it is always the right thing to do. Consider the batch command. It doesn't set PATH and it is very correct in not doing so. That is, if the following:

exec /usr/bin/at -qb $*

were replaced with

PATH=/usr/bin; export PATH
...
at -qb $*

This would change the environment that at(1) attaches to the job - potentially breaking it.

Shell Programming and temporary files

2009-06-19T21:06:00.005-05:00

I've been generally impressed with the coding standards that are enforced on C code in OpenSolaris, as checked by cstyle. However, similar automated checks don't exist for shell scripts. This, combined with a history of "/bin/sh is the one true shell", has led to inefficient scripts that very commonly have security vulnerabilities.

I'm hoping to breathe some life into this blog with a series of posts that describe some of the common problems and some potential solutions. I'll start with problems related to security problems with temporary files.

Consider the following bit of code taken from /usr/lib/lp/bin/lpadmin.

   293  # Do the LP configuration for a local printer served by lpsched
   294  if [[ -x ${LPADMIN} && -n "${local}" ]] ; then
   295          # enumerate LP configured printers before modification
   296          PRE=/tmp/lpadmin-pre.$$
   297          (/bin/ls /etc/lp/printers 2>/dev/null ; /bin/ls /etc/lp/classes \
   298                  2>/dev/null) >${PRE}

There are two problems here:

A symbolic link vulnerability exists.
The TMPDIR environment variable is not respected. If a user has a reason to want temporary files stored in a particular file system, utilities should respect the user's wishes.

The solution

For quite some time, Solaris has provided the mktemp command has existed to facilitate the secure creation of files. In the fix for this problem, I used mktemp in a way that both creates the file securely and respects TMPDIR.

     41 MKTEMP="/usr/bin/mktemp -t"
      ...
    300  PRE=$(${MKTEMP} lpadmin-pre.XXXXXX)
    301  if [[ -z "${PRE}" ]] ; then
    302   gettext "lpadmin: System error; cannot create temporary file\n" 1>&2
    303   exit 2
    304  fi
    305 
    306  (/bin/ls /etc/lp/printers 2>/dev/null ; /bin/ls /etc/lp/classes \
    307   2>/dev/null) >${PRE}

Installation of Studio 11 on Nevada

2008-07-23T19:10:00.002-05:00

The OpenSolaris Sun Studio installation page forgets to mention that the default version of Java is incompatible with the Studio installer. The following command works for me:

# export JAVA_HOME=/usr/jdk/instances/jdk1.5.0
# export PATH=/usr/jdk/instances/jdk1.5.0/bin:$PATH
# ./installer

OpenSolaris build service proposal

2008-06-11T21:15:00.003-05:00

With most open source projects the various pieces are relatively small and as a result, the build process for any component is measured in seconds or minutes. Solaris has historically been based upon consolidations - relatively large sets of code. The most common wrapper around the build process is called nightly for good reason - it takes hours to days to perform a full build. Once a full build is complete incrementals can take a lot less time, but if you infrequently build the software, you are pretty much forced into a full build every time. I just stumbled across this post that shows that others have noticed this problem too.

What can be done?

I've been thinking that most people really don't need a full clobber build all the time. If there is a 2 line fix somewhere, just verifying that the code builds and the resulting binaries work is most important - and most encouraging for the casual bug fixer. If there's a big re-work of Makefiles or changes are made to the linking of a program a full build may be more important. A service that can do incremental builds would be a big help.

So here's (very roughly) how I envision it.

Each time a release is tagged by a participating consolidation...

A job is kicked off to build it on each supported architecture.
A zfs snapshot is created of the workspace with the completed build.
The completed build is packaged into an IPS repository

When a developer wants to fix a bug...

Populate a workspace with the original source code
Hack
Create a webrev
Submit the webrev to the build system, saying which consolidation and build to build against and which architectures to build for

The build system then...

Schedules the job via Hudson or Grid Engine.
The job starts by cloning the pre-built release of the consolidation. Quite likely, it also clones then boots the proper boot environment.
The patch included in the webrev is applied to the source
An incremental build is done
The pre-built IPS repository is cloned to a per-developer repository
Packages associated with the changed files are populated into the per-developer IPS repository. The changed packages associated with this build are added as dependencies to a special package (e.g. build-20080611.0)
Results are sent to the developer.

In most cases, the execution time of the above could be just a few minutes. If the change triggers lots of dependencies, it could be many hours. To test the changes, the developer can use...

beadm create testbe
beadm mount testbe /testbe
pkg -R /testbe set-authority -O http://pkg.opensolaris.org/mysername myusername.opensolaris.org
pkg -R /testbe install build-20080611.0
beadm unmount testbe
beadm activate testbe
init 6

There are certainly plenty of details to work out. This is meant to be the start of a discussion to determine wether it is something worth pursuing and if so what those missing details are.

Solaris Wish List: Make it Open Source

2008-03-21T19:14:00.007-05:00

I frequently see references from Sun and those that quote some Sun people as saying that Solaris is Open Source.

What is Solaris?

There are a few different flavors of things that people may refer to as Solaris:

Solaris 10 and earlier versions. This is what most references to the word Solaris seem to be referring to. Each release of Solaris is supported for a number of years through commercial and no-cost channels, to varying degrees.
Solaris Express. This is a collection of software that is viewed as a fairly stable distribution based upon the development branch of Solaris. Very limited ("installation and configuration as well as for developer assistance") is available, but the support period seems to be limited to around 3 months per release.
OpenSolaris. This is best summarized as "The main difference between the OpenSolaris project and the Solaris Operating System is that the OpenSolaris project does not provide an end-user product or complete distribution. Instead it is an open source code base, build tools necessary for developing with the code, and an infrastructure for communicating and sharing related information. Support for the code will be provided by the community; Sun offers no formal support for the OpenSolaris product in either source or binary form."

What is Open Source?

The annotated Open Source Definition covers this quite well.

Is Solaris Open Source?

Let's see if the license used by Solaris aligns with the Open Source definition.

Free redistribution	No. The Software License Agreement states: You may make a single archival copy of Software, but otherwise may not copy, modify, or distribute Software.
Source Code	No. The source code for Solaris is not available. Note that while a bunch of code is available at src.opensolaris.org, this is not the same source code that is used for building Solaris. If I want or need to modify the behavior of Solaris, there is no straight-forward way to do so.
Derived Works	No. Since I may not copy, modify, or distribute Solaris, this point is moot.
Integrity of Author's Source Code	No. Since the source code is not available, this point is also moot.
No Discrimination Against Persons or Groups	I think so. See section 11 of the license agreement for export restrictions.
No Discrimination Against Fields of Endeavor	I think so. While Sun's lawyers don't want to suggest that running your nuclear power plant with Solaris, they don't say that you can't.
Distribution of license	No. Since redistribution is not allowed, this point is moot.
...	...

Since one or more of the requirements to be called Open Source are not met by the license under which Solaris is distributed, Solaris is not open source.

People that need support don't want source code.

That is great in theory, but in practice it falls down a bit. Let's pretend that you need to write a dtrace script to dig into a thorny performance problem. If the stable dtrace providers don't provide probes at the right spot, you need to fall back on fbt or pid probes. The only way to understand what these are tracing is to read the source. Since the OpenSolaris and Solaris branch point is now about 3 years old, this is becoming extremely difficult to do reliably. The code that you get by browsing the latest OpenSolaris code sometimes does not match up with the fbt probes that are available in any release of Solaris. You may have more luck looking at historical versions of the source code, but that is a guessing game at best.

There are also times when a customer's needs just do not align with what Sun is willing to offer. Suppose you need different functionality from a device driver. It is possible that it is a trivial change from the customer's standpoint, so long as they have current source code. However, if the source code is not available, the best the customer can do is grab the same driver from somewhere else (e.g. opensolaris.org) and try to to maintain a special version of the driver and provide custom ports of all the bug fixes that they would otherwise get from from the Solaris source.

Suppose that hypothetical fix that the customer needed was something that Sun agreed was needed but they did not have the time to develop the fix. If the Solaris source code were as open as the OpenSolaris source code, the customer could work with the OpenSolaris community to get the fix integrated into OpenSolaris, then provide the backport to Solaris. If the customer could do this, the code would see the appropriate code reviews, development branch burn-in, etc. with minimal additional workload on Sun.

My Wish

I know that Sun went through tremendous work to make OpenSolaris happen. They should be commended for that and the other tens (hundreds?) of millions of lines of code they have opened up or written from scratch in the open. This gives them tremendous opportunity with Solaris 11 (assuming that 10+1 = 11). Keep the code that is open, well, open. When patches or other updates are released, be sure that it is clear in the open source code repository which files are used in building that update. To get the full benefit of this, it should be possible for the Solaris customer to set up a build environment to build this source.

It is OK if Sun doesn't want to support the code I modify. However, I would expect that they would support the unmodified parts of Solaris much the same as they do if I install a third-party package that adds some device drivers and mucks with some files in /etc.

OpenSolaris Wish List: Feature-based meta packages

2008-03-15T22:07:00.006-05:00

The problem

There are many occasions when an administrator needs to install a particular feature on a system but tracking down the packages, patches, or revisions thereof is next to impossible. Consider the following scenarios:

A sysadmin is trying to build minimized systems that have a firewall, bash, python, and can support zones using Jumpstart. What needs to appear in the Jumpstart profile?
That same sysadmin then decides that a particular installed system needs to be augmented to support the OSPF routing protocol.
Some time passes and an updated feature of is released. The sysadmin needs to be able to update systems to support that feature without guessing which packages or patches are required. Note that the a subset of the required packages may not have been installed initially because they did not exist initially or they weren't required for to meet the dependencies of the earlier feature set.
A sysadmin notices a nifty feature mentioned in the "What's New" documentation and wants to determine if a recently patched system system supports the feature.

Dealing with any of these situations is very difficult today.

Most commonly, the best advice that is offered when trying to indicate what is required to have a particular feature is to say that the feature is available in Solaris X update Y. While that is true to a certain extent, that is not the complete story. It assumes that the system in question is in no way minimized and ignores the fact that the same patches that are integrated in update Y are also available for installation on releases before update Y.

Ongoing Work

The Image Packaging System looks to be outstanding work in this direction. The one-pager is likely the best place to look for a quick overview. While this will provide a good foundation, there are numerous posts to mailing lists of the form "I'm trying to install X, but pkg can't find it."

Suggested Approach

Some geeks (power users, whatever you would like to call them) will know the exact name of the packages that they want or will otherwise be able to search for them rather adeptly. Having the ability to say "I want bash" rather than "I want the same shell used by most Linux distros" is important.

I suspect that most people will be quite happy looking at their requirements and matching them up to advertised features. I suggest that the feature/* package namespace be reserved for meta-packages that represent features. A feature meta-package should not deliver any files, rather it just defines dependencies for the minimal set of packages required to support the feature. See the Package FMRIs and Versions section of pkg(5) for details on package naming.

For instance, if I needed the zones feature I should be able to say something along the lines of "pkg install feature/zones". This would likely correspond to an FMRI of pkg://opensolaris.org/feature/zones@5.11,5.11-7:20080326T164523Z. When new features are updated, the build number (7 in the example FMRI) should be updated.

While searching the published packages would yield a pretty good list of packages required for supporting zones, someone needing the OSPF routing protocol may have a harder time with a search. However, installing feature/network/routing/ospf would likely know that the zebra or quagga package is needed (varying by Solaris release).

Integration with Documentation

With such an approach, the Solaris What's New documentation could be very helpful to sysadmins needing to update existing systems, installation profiles, etc. Consider the following, based upon a particular entry from the Solaris 10 What's New documentation.

IPsec Tunnel Reform
Solaris now implements IPsec Tunnel Mode per RFC 2401. Inner-packet selectors can be specified on a per-tunnel-interface basis using the new "tunnel" keyword of ipsecconf(1M). IKE and PF_KEY handle Tunnel Mode identities for Phase 2/Quick Mode. Interoperability with other IPsec implementations is greatly increased.
For more information, see Transport and Tunnel Modes in IPsec in System Administration Guide: IP Services. This feature is available via the feature/network/protocol/rfc2401software collection.

To help someone with this feature find documentation of this type in the future, a documentation system (installed on system or internet accessible) should be able to point a person interested in documentation related to a feature to the appropriate man page(s), online books, and other documentation.

Similarly, if the feature is just an update of an existing feature, having the documentation refer to a particular build (release, whatever works) of the feature package would be must useful.

Future of OpenSolaris Boot Environment management

2008-03-11T18:36:00.004-05:00

I was quite happy to see this recent post from Ethan Quach proposing an efficient method for sharing the variable parts of /var. It bears a striking resemblance to something that I suggested and and clarified in the past.

Correction June 6, 2009: Links to mail archives at opensolaris.org seem not to be stable. The same messages are available at the following: My initial suggestion and clarification.

But why does this matter? When you are making significant changes to the system, such as during a periodic patch cycle or upgrade, it is generally desirable to...

be able to do so without taking the system down for the duration of the process
be able to abort the operation if you have a change of heart
be able to fail back if you realize that newer isn't better

Consider what is in /var:

Mail boxes If the machine is a mail server (using sendmail et. al.) there is a pretty good chance that users have their active mail boxes at /var/mail.
In flight mail messages Most machines process some email. For example, if a cron job generates output it is sent to the user via email. Many non-web mail clients invoke /bin/mail or /usr/lib/sendmail to cause mail to be sent. Each message spends somewhere between a few milliseconds and a few days in /var/spool/mqueue or /var/spool/clientmqueue.
Print jobs If the machine acts as a print server (even for a directly attached printer) each print job spends a bit of time in /var/spool/lp.
Logs When something goes wrong, it is often times useful to look in log messages to figure out why it went wrong. Those are often found under /var/adm.
Temporary files that may not be It is rather common for people to stick stuff in /var/tmp and expect to be able to find it sometime in the future.
DHCP If a machine is a dhcp server, it will store configuration and/or state information in /var/dhcp.
...

All of those things should be of a stable file format and usable before and after you patch/upgrade/whatever. If you take the traditional Live Upgrade approach, you can patch or upgrade to an alternate boot environment. As part of activating the new environment, a bunch of files are copied between boot environments. According to this page the following things are synchronized:

/var/mail                    OVERWRITE
/var/spool/mqueue            OVERWRITE
/var/spool/cron/crontabs     OVERWRITE
/var/dhcp                    OVERWRITE
/etc/passwd                  OVERWRITE
/etc/shadow                  OVERWRITE
/etc/opasswd                 OVERWRITE
/etc/oshadow                 OVERWRITE
/etc/group                   OVERWRITE
/etc/pwhist                  OVERWRITE
/etc/default/passwd          OVERWRITE
/etc/dfs                     OVERWRITE
/var/log/syslog              APPEND
/var/adm/messages            APPEND

Notice that the default configuration loses your in flight print jobs because /var/spool/lp is not copied. Suppose you have a mail server with a few gigs of mail at /var/mail. Is it a good use of time or disk space to copy /var/mail between boot environments?

A much better solution seems to be to make those directories shared between the boot environments. The way to do this in Live Upgrade and presumably in the future is to remove (or not add) them to /etc/lu/synclist and allocate separate file systems. However, do you really want a file system for /, /var/mail, /var/spool/mqueue, /var/spool/clientmqueue, /var/spool/lp, /var/adm, /var/tmp, /var/dhcp, ...? What if you had someone tell you that you had to monitor every file system on every machine for being out of space? How big would you make all of those file systems so that your monitoring didn't wake you up in the middle of the night?

In the future, it looks as though OpenSolaris will use ZFS to store each boot environment. Among the features of ZFS that make this desirable are snapshots, clones, and rethinking the boundary between disk slices (or volumes) and file systems. If the organization of /var is changed just a bit...

/var/adm -> share/adm
/var/dhcp -> share/dhcp
/var/mail -> share/mail
/var/spool -> share/spool
/var/tmp -> share/tmp
/var/share/adm
/var/share/dhcp
/var/share/mail
/var/share/spool
/var/share/tmp

Then you can get by with having two zfs file systems: / and /var/share. The Snap Upgrade process would then likely do the following:

Take a snapshot of /, clone it, then mount it somewhere usable in subsequent steps (e.g. /mnt/abe)
Do whatever is needed on the alternate boot environment mounted at /mnt/abe.
Unmount the alternate boot environment

When it comes time to activate the new boot environment, there are some files that are likely need to be synchronized using the traditional mechanism. For instance, if someone tried to get into a system by guessing a user's password, there is a reasonable chance that the account was locked via a modification to /etc/shadow. Presumably you don't want to give the bad guy another chance when you activate the new boot environment. Note, however that the files that may need to be synchronized in /etc are nearly always small files and there would not be very many of them. The files in /var/shared would not need to be synchronized. However, just in case the new version of sendmail decides to eat mailboxes, it would be very nice to be able to recover.

This means that activating a boot environment would look like:

Bring the system into single-user mode
Mount the alternate boot environment
Synchronize those files that need to be synchronized
Take a snapshot of /var/shared
Set the boot loader to boot from the new boot environment and offer a failback option to the old boot environment
Reboot

The items in italics are special to boot environment activation. Each one should take a couple seconds or less - adding far less than thirty seconds to the normal reboot process to activate the new boot environment. Failback would be similarly quick.

Now suppose this system is a bit more complicated and has 20 zones on it. Have you ever patched a system with 20 zones on it? Did you start and Friday and finish on Monday? How happy were the users with the "must install in single-user mode" requirement? This same technique should allow you to have two file systems per non-global zone - one for the zone root and one for /var/shared in the zone. Supposing that the reboot processing takes 5 seconds per zone you are looking at an extra minute to reboot rather than a weekend of down time.

Without Live Upgrade or Snap Upgrade, what would backout look like? After you had the system down for patching for a couple days, you could take it down again for a couple days to back the patches out. Or you could go to tape. Neither is an attractive option. With Snap Upgrade you should be able to fail back with your normal reboot time plus a minute.

Today's dtrace one-liners

2007-04-20T18:34:00.001-05:00

Well, I've been spending a bit more time with dtrace lately. I'm now starting to do interesting things with dtrace without looking at the manual or snagging other scripts. Today I found some processes that looked like they weren't doing much of use. In particular, I saw the following in "prstat -mL"

   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID 
 15274 oracle    46  53 0.0 0.0 0.0 0.0 0.0 0.0   0  67 .22   0 racgmain/1
 15295 oracle    47  53 0.0 0.0 0.0 0.0 0.0 0.0   0  74 .23   0 racgmain/1
 16810 oracle    47  53 0.0 0.0 0.0 0.0 0.0 0.0   0  66 .23   0 racgmain/1
 25029 oracle    47  53 0.0 0.0 0.0 0.0 0.0 0.0   0  70 .23   0 racgmain/1

First, it is confusing that it says that these processes are doing less than one system call per second (.22 or .23), but spending more than 50% in system calls. Looks like a bug in prstat. To see what system call was being used...

# dtrace -n 'syscall:::entry/execname == "racgmain"/ { @[probefunc] = count() }
dtrace: description 'syscall:::entry' matched 232 probes
^C

  read                                                         269039

Ahh, so it is only reads that are causing all of this. Let's see which file descriptors are involved.

# dtrace -n 'syscall::read:entry/execname == "racgmain"/{ @[pid, probefunc,arg0] = count() }'
^C
    22469  read      9                7
    22469  read      4                7
    22469  read      3                9
    22469  read      8                9
    22469  read     10               10
    22469  read      7               21
    22469  read      6               24
    15274  read     10          1026462
    15295  read     10          1044932
    25029  read     10          1051450
    16810  read     11          1060720

It seemed rather odd that there would be so much reading on a single file descriptor per file (10 or 11, depending on the process). If this were really the case, I would expect to see some processes showing up in prstat that were high in %sys doing writes. Using pfiles on each of the pids showed that the file descriptor in question was a FIFO, but gave no indication what (if anything) was on the other end of the FIFO.

To help answer the question of what might be on the other end of the FIFO, I tried this (ok, not a one liner...)

#!/usr/sbin/dtrace -qs 

syscall::read:return
/execname == "racgmain"/
{
        @c[pid, arg0] = count()
}

tick-1s
{
        printf("%6s %6s %4s\n", " pid  ", " size ", "count");
        printa("%6d %6d %@4d\n", @c);
        clear(@c);
}

Which gave the following output...

 pid    size  count
 15295      0 117524
 15274      0 121065
 25029      0 121308
 16810      0 125358
 pid    size  count
 15274      0 108177
 25029      0 109639
 15295      0 115735
 16810      0 117175
 pid    size  count
 15295      0 119411
 15274      0 119471
 25029      0 120273
 16810      0 120650

Imagine that... these processes are each doing 100,000+ zero-byte reads per second. Time to tell the DBA's that they have some real problems with one of their daemons.

reproducible hang with ldom preview

2007-04-11T16:59:00.000-05:00

This blog entry is because formatting is horribly broken at http://forum.java.sun.com/thread.jspa?threadID=5159568 where I originally posted it.

I configured a T2000 as described in the beginner's guide (http://www.sun.com/blueprints/0207/820-0832.pdf) with the exception of the device allocated for the root disk. For that I came up with my own variant of http://unixconsole.blogspot.com/2007/04/time-to-build-guest-domain.html.

My variant of using a file involved creating the file with mkfile on a zfs file system. That is...

zpool create zfs mirror c1t0d0s4 c1t1d0s4
zfs create zfs/ldoms
zfs set compress=on zfs/ldoms
mkfile 32G /zfs/ldoms/root.img

As I install Solaris in the ldom, the server (control domain) dies after extracting a few hundred megabytes of a flash archive. I have traced this down to it running out of memory. Here's "vmstat 4" output on the control domain console:

...
0 0 0 8636536 24776  1  70  0  0  0  0  0  0  0  0  0 1954  311 2244  0 23 76
0 0 0 8645096 22280  1  36  0  0  0  0  0  0  0  0  0 5957  370 9827  0 19 80
0 0 0 8649008 17720  1  40  0 296 313 0 44 0  0  0  0 9975  361 15877 0 24 75
0 0 0 8651104 15944  1  52  0 807 1671 0 700 0 0 0  0 10725 347 17545 0 26 74
0 0 0 8650800 18376  0  60  0 88 239 0 127 0  0  0  0 9816  391 15545 1 33 67
0 0 0 8640432 15936  0  76  0 497 3025 0 3874 0 0 0 0 11367 432 17975 0 35 65
0 0 0 8642968 17032  1  59  0 452 2028 0 842 0 0 0  0 10266 363 16127 0 27 73
kthr      memory            page            disk          faults      cpu
r b w   swap  free  re  mf pi po fr de sr m0 m1 m2 m1   in   sy   cs us sy id
0 0 0 8644768 15744  0  56  0 387 1298 0 126 0 0 0  0 10170 330 16355 0 24 75
0 0 0 8652504 18368  1 113  0 372 2462 0 273 0 0 0  0 11171 321 18613 0 35 65
0 0 0 8652832 15720  1 134  0 411 6081 0 738 0 0 0  0 11541 332 18979 0 34 66
0 0 0 8652232 14312  1  94  0 413 1806 0 7775 0 0 0 0 10718 358 18271 0 38 62
0 0 0 8647360 12592 18 133  9 555 5176 0 17490 1 0 1 0 10394 320 16970 1 37 63
0 0 0 8645248 14408  2  73 22 486 5039 0 3111 2 1 1 0 11749 383 18336 0 40 59
2 0 43 8641800 2784  1 148 99 1070 1517 0 53982 19 9 9 5 8316 356 14226 0 43 57
0 0 116 8647032 800  1  42 127 134 312 3688 76207 14 7 7 1 2153 114 3726 0 29 71

At this point the server froze. Note that 116 processes were swapped and the "de" column is 3688. Very bad news.

My initial thoughts were that I was running into some of the low-memory problems known to happen with the ZFS arc. This does not seem to be the case. According to mdb, the arc size is around 60 MB:

# mdb unix.3 vmcore.3
...
> arc::print -td size
uint64_t size = 0t61455360

The control domain is S10 11/06 + 118833-36 + those required for ldoms + many others. The ldom is in the process of being installed is booted from a S10 11/06 netinstall image (118833-33).

Random password one-liner

2006-08-31T21:26:00.000-05:00

I recently came up with this method for generating reasonable random 8-character passwords:

$ dd if=/dev/random bs=6 count=1 2>/dev/null | openssl base64
LCia46S4

If 8 characters is not long enough, increase the number after bs= to 75% of the number of characters you would like in the password.

Install Solaris from DVD image on disk

2006-08-26T17:26:00.000-05:00

My personal SPARC machine is pathetic by today's standards - An Ultra II with a pair of 300 MHz processors, 768 MB RAM, and a very slow CDROM drive. This is pretty much the slowest machine that is supported by Solaris 10. That, and today I decided it was time to get a fresh installation of Solaris Express (build 46) on it.

I first tried the live upgrade route. However, that didn't work out too well because I had previously used bfu to get some newer OpenSolaris bits on the machine. I really did not want to repeat the download process for all the CD ISO's (already had downloaded the DVD ISO). Now, if you think that downloading and burning is slow - you should see the speed of the installation on this CDROM drive. It was probably OK in the days when Solaris fit on one CD, but not today with 5(?) CD's to complete the installation.

The disk layout of the machine was as follows:

c0t0d0 32 GB disk
- c0t0d0s0 - 4.5 GB available for new /
- c0t0d0s1 - ~500 MB swap
- c0t0d0s7 - remainder as zfs pool "pool0"
c0t1d0 4 GB disk
- c0t1d0s0 - Root with build 36 (?) + random BFU bits

I had the DVD image in a subdirectory of my home directory that was in the pool0/home file system in the zfs pool. To make use of that DVD image without buying a SCSI DVD drive, I did the following:

Burn build 46 CD0 to a CD-R
Boot from the CD-R
Go clean up the shop from the woodworking I was doing earlier
Do some laundry
Return to the Ultra II to find that it was just about to ask me which language I speak. Really, it was still working on it. Now do you know why I didn't want to feed it 5 CD's?
Answer sysidcfg questions
Exit the installer
zpool import pool0. After the import was complete but before mounting file systems, zpool crashed with a segv. Later I saved that core file to /a for later analysis
zfs set mountpoint=/tmp/home pool0/home
zfs mount pool0/home
lofiadm -a /tmp/home/build46.iso
umount /cdrom
mount -F hsfs -o ro /dev/lofi/1 /cdrom
install-solaris
Go blog about a cool hack. :)

The installation is now about 40% done. Looks like the hack is working just fine. I wonder if I could bundle this all up in a begin script (especially the laundry) to automate the installation from an ISO image after booting from local media.

Update on zoneadm create with zfs

2006-02-28T17:20:00.000-06:00

After reaching out to Sun to work on getting my work integrated into OpenSolaris, I found that Sun was already working on this feature. Subsequently, they indicated that the code made it into some internal source code tree. As such, I am holding off on future development until I can get at that code.

However, if you are wanting to try it out, I have posted the code for others to play with. If you have a working OpenSolaris build environment, you should be able to drop in my modified zoneadm.c, run dmake all, then use the resulting zoneadm command. Alternatively, the sparc version of the zoneadm binary is also available.

Enjoy!

T: OpenSolaris Zones ZFS

Zone created in 0.922 seconds

2006-02-19T19:45:00.000-06:00

I noticed today that in the latest OpenSolaris code that "zoneadm clone" exists. Unfortunately, cloning a zone only offered the copy mechanism that was essentially "find | cpio". A bit of hacking later and we have this:

# time ksh -x /var/tmp/clone
+ newzone=fast
+ template=template
+ zoneadm=/ws/usr/src/cmd/zoneadm/zoneadm
+ PATH=/usr/bin:/usr/sbin
+ zonecfg -z fast create -t template
+ zonecfg -z fast set zonepath=/zones/fast
+ /ws/usr/src/cmd/zoneadm/zoneadm -z fast clone -m zfsclone template
Cloning zonepath /zones/template...

real    0m0.922s
user    0m0.128s
sys     0m0.171s

This comes is achieved using zfs to create a snapshot of the template zone, then clone the snapshot to create the zonepath of the new zone. A bit of cleanup is needed, but goodness is on the way.

T: OpenSolaris Zones ZFS

hunting bugs in filebench

2006-02-15T19:32:00.000-06:00

I've been using filebench a bit at work and decided that I would like to try a few things out at home. My home machine is not quite as beefy as the V40z's that I have been testing on at work. Getting filebench to compile in the first place is a bit of work. Probably works really well on someone else's system, but mine is obviously different. That's another story though. After compiling filebench, I ran it for the first time and saw this:

$ /opt/filebench/bin/filebench
Segmentation fault (core dumped)

Bummer. Well, let's see where that is at:

$ gdb /opt/filebench/bin/filebench core
GNU gdb 6.4-debian

. . .

(gdb) where
#0  0x37dd84aa in memset () from /lib/tls/i686/cmov/libc.so.6
#1  0x0807b01e in ?? ()
#2  0x080522da in ipc_init () at ipc.c:264
#3  0x08058bc1 in main (argc=1, argv=0x3f8fdcf4) at parser_gram.y:1140

OK, so let's go with the assumption that the bug is in the code listed as alpha on the web site, and not libc. So we go up the stack a couple levels.

(gdb) up 2
#2  0x080522da in ipc_init () at ipc.c:264
264             memset(filebench_shm, 0, c2 - c1);
(gdb) print filebench_shm
$1 = (filebench_shm_t *) 0xffffffff

Hmmm... 0x with a bunch of f's looks like -1. Perhaps some system call on Solaris (presumably where filebench started) returns NULL on error and on Linux it returns -1. Let's go looking for that system call.

(gdb) list
259     #endif /* USE_PROCESS_MODEL */
260
261             c1 = (caddr_t)filebench_shm;
262             c2 = (caddr_t)&filebench_shm->marker;
263
264             memset(filebench_shm, 0, c2 - c1);
265             filebench_shm->epoch = gethrtime();
266             filebench_shm->debug_level = 2;
267             filebench_shm->string_ptr = &filebench_shm->strings[0];
268             filebench_shm->shm_ptr = (char *)filebench_shm->shm_addr;

Nope, not there. Maybe a bit further up.

(gdb) list 250
245     #endif
246
247             if ((filebench_shm = (filebench_shm_t *)mmap(0, sizeof(filebench_shm_t),
248                     PROT_READ | PROT_WRITE,
249                     MAP_SHARED, shmfd, 0)) == NULL) {
250                     filebench_log(LOG_FATAL, "Cannot mmap shm");
251                     exit(1);
252             }
253
254     #else

It looks like mmap may be the culprit. I first asked man, but this is Linux, not Solaris. No man page for mmap! Next try google. Google comes up with this page that looks a lot like a man page. Why isn't that found on my system? Another thing for another day. Anyway, it says:

RETURN VALUE
On success, mmap returns a pointer to the mapped area. On error, the value MAP_FAILED (that is, (void *) -1) is returned, and errno is set appropriately. On success, munmap returns 0, on failure -1, and errno is set (probably to EINVAL).

Ok, so it is returning -1 because it doesn't like something. Let's see what it is trying to mmap:

(gdb) print sizeof(filebench_shm_t)
$2 = 907368000
(gdb) print sizeof(filebench_shm_t) / 1024 / 1024
$3 = 865
(gdb)

That 'splains it. It looks like it is trying to set up a shared memory segment that is 865 MB. My poor little system only has 512. FWIW, I have created a patch that addresses this one problem but I haven't had a chance to test it on Solaris yet. Unfortunately, with the patch, it just tells me that the mmap failed. It doesn't address the fact that it is trying to allocate a shared memory segment larger than the size of RAM on my system.

Update 1:

I have posted several patches to the bug tracking system at sourceforge.net. This particular one is 1432638. It turns out that mmap on Solaris also returns MAP_FAILED so the patch is simpler than I originally expected.

T: filebench Debugging

Download and gunzip in one step

2006-01-31T20:23:00.000-06:00

I was feeling the need to take a look at Nexenta and decided that I wasn't terribly interested in waiting for a download, then waiting for a gunzip. Why not do them both at the same time?

$ wget -O /dev/stdout  http://www.gnusolaris.org/gsmirror/genunix.org/elatte_installcd_alpha2_i386.iso.gz  | gunzip > elatte_installcd_alpha2_i386.iso         => `/dev/stdout'
--20:21:47--  http://www.gnusolaris.org/gsmirror/genunix.org/elatte_installcd_alpha2_i386.iso.gz
Resolving www.gnusolaris.org... 216.129.112.21
Connecting to www.gnusolaris.org|216.129.112.21|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://www.genunix.org/distributions/gnusolaris/elatte_installcd_alpha2_i386.iso.gz [following]
--20:21:48--  http://www.genunix.org/distributions/gnusolaris/elatte_installcd_alpha2_i386.iso.gz
Resolving www.genunix.org... 204.152.191.100
Connecting to www.genunix.org|204.152.191.100|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 567,433,011 (541M) [text/plain]

13% [====>                                ] 77,025,880   359.25K/s    ETA 22:27

Just 22 minutes to go. I guess at this rate I could have piped it through cdrecord with "speed=2".

T: Nexenta OpenSolaris

patch_order made easy

2006-01-13T21:22:00.000-06:00

Some of my most tedious times as a Solaris administrator have been when I needed to create a patch_order file for a custom patch cluster. For a long time I have intended to just write a script...

But now, I don't have to do that any longer! Today I discovered that smpatch(1M) now has an order subcommand. This makes it really quite simple for me to create a patch_order file for a very long list of patches. In this example, I create the patch_order file for the patches in the Solaris 10 Update 1 UpgradePatches directory:

# cd /mnt/Solaris_10/UpgradePatches
# ls > /tmp/patches
# smpatch order -d `pwd` -x idlist=/tmp/patches > /tmp/patch_order

Now, if you want to go the full length and create a patch cluster for it:

# mkdir /tmp/10U1_UpgradePatches
# cd /tmp/10U1_UpgradePatches
# mv /tmp/patch_order .
# ln -s /mnt/Solaris_10/UpgradePatches/* .
# cp /somewhere/10_Recommended/install_cluster .

Modify the SUPPLEMENT_NAME="..." line in install_cluster to be more descriptive for this patch cluster. Be sure to not use characters like /, \, |, etc.

# cd /tmp
# zip -rq 10U1_UpgradePatches.zip 10U1_UpgradePatches

At this point, you can copy the 10U1_UpgradePatches around to your various machines and use it just like you would a 10_Recommended bundle. Enjoy!

2005-05-11T20:54:00.000-05:00

I am sad to report that the world is running short of geeks. I think it was the 1200 baud modem in my 386sx system that I smoked will building Linux 0.99pl14 that solidified my position in the 90's.

My first SUPerG

2005-04-22T18:26:00.000-05:00

I had the privilige of attending SUPerG this week. Some of the highlights of the presentations come in the form of short quotes from some of the speakers.

Volume managers are shims that were introduced to work around bugs that should have been fixed in the file system. - Richard McDougall

And finally, someone of authority that has said what I have been trying to get across for a couple years:

IO Wait == Idle - Glenn Fawcett

Overall I was surprised by the amount of emphasis that Sun has put on the future of storage. With the combination of 10 Gigabit Ethernet, FireEngine, and NFSv4, Sun very clearly stated that the future of storage is in NAS. The story is pretty well outlined by Richard McDougall in this blog entry. I was happy to see that the first Niagara systems seem to be ahead of schedule. Within the next 8 months I should be able to get a single processor SPARC box that is more powerful than a 12-processor 4800 or any commodity box sporting chips from Intel or AMD. Way cool. The thing with silicon that really blew me away was Rock's hardware scout. This chip is able to look a thousand or more instructions ahead to optimize cache loads. The net effect is that you can get by with a lot less cache, thereby making huge caches unnecessary. And for those of us that are a couple years into the life of 15k's, the good news keeps coming. First, the upgrade to UltraSPARC IV boards brings about 90% performance improvement, the UltraSPARC IV+ will add another 70%. With an expected July release of the US IV+, this means that through periodic system board (and carrier plate...) upgrades, the 15k (and 4800, 6800, 12k, etc.) are keeping pace with Moore's Law without a forklift upgrade. Very impressive. There was also a lot of talk about various Solaris 10 features... however, things like zones, dtrace, zfs, etc. have been covered consistently for a long time in the press and in many data centers that there was not a whole lot new to say. Putting all of the improvements in Solaris together with advances in other technologies (10 gigabit ethernet, multiple vendors with NFSv4, CMT, ...) really shows that there is a bright future for Solaris.