Shell Programming and temporary files

I've been generally impressed with the coding standards that are enforced on C code in OpenSolaris, as checked by cstyle. However, similar automated checks don't exist for shell scripts. This, combined with a history of "/bin/sh is the one true shell", has led to inefficient scripts that very commonly have security vulnerabilities.

I'm hoping to breathe some life into this blog with a series of posts that describe some of the common problems and some potential solutions. I'll start with problems related to security problems with temporary files.

Consider the following bit of code taken from /usr/lib/lp/bin/lpadmin.

   293  # Do the LP configuration for a local printer served by lpsched
   294  if [[ -x ${LPADMIN} && -n "${local}" ]] ; then
   295          # enumerate LP configured printers before modification
   296          PRE=/tmp/lpadmin-pre.$$
   297          (/bin/ls /etc/lp/printers 2>/dev/null ; /bin/ls /etc/lp/classes \
   298                  2>/dev/null) >${PRE}

There are two problems here:

1. A symbolic link vulnerability exists.
2. The TMPDIR environment variable is not respected. If a user has a reason to want temporary files stored in a particular file system, utilities should respect the user's wishes.

The solution

For quite some time, Solaris has provided the mktemp command has existed to facilitate the secure creation of files. In the fix for this problem, I used mktemp in a way that both creates the file securely and respects TMPDIR.

     41 MKTEMP="/usr/bin/mktemp -t"
      ...
    300  PRE=$(${MKTEMP} lpadmin-pre.XXXXXX)
    301  if [[ -z "${PRE}" ]] ; then
    302   gettext "lpadmin: System error; cannot create temporary file\n" 1>&2
    303   exit 2
    304  fi
    305 
    306  (/bin/ls /etc/lp/printers 2>/dev/null ; /bin/ls /etc/lp/classes \
    307   2>/dev/null) >${PRE}

Solaris Wish List: Make it Open Source

I frequently see references from Sun and those that quote some Sun people as saying that Solaris is Open Source.

What is Solaris?

There are a few different flavors of things that people may refer to as Solaris:

• Solaris 10 and earlier versions. This is what most references to the word Solaris seem to be referring to. Each release of Solaris is supported for a number of years through commercial and no-cost channels, to varying degrees.
• Solaris Express. This is a collection of software that is viewed as a fairly stable distribution based upon the development branch of Solaris. Very limited ("installation and configuration as well as for developer assistance") is available, but the support period seems to be limited to around 3 months per release.
• OpenSolaris. This is best summarized as "The main difference between the OpenSolaris project and the Solaris Operating System is that the OpenSolaris project does not provide an end-user product or complete distribution. Instead it is an open source code base, build tools necessary for developing with the code, and an infrastructure for communicating and sharing related information. Support for the code will be provided by the community; Sun offers no formal support for the OpenSolaris product in either source or binary form."

What is Open Source?

The annotated Open Source Definition covers this quite well.

Is Solaris Open Source?

Let's see if the license used by Solaris aligns with the Open Source definition.

Free redistribution	No. The Software License Agreement states: You may make a single archival copy of Software, but otherwise may not copy, modify, or distribute Software.
Source Code	No. The source code for Solaris is not available. Note that while a bunch of code is available at src.opensolaris.org, this is not the same source code that is used for building Solaris. If I want or need to modify the behavior of Solaris, there is no straight-forward way to do so.
Derived Works	No. Since I may not copy, modify, or distribute Solaris, this point is moot.
Integrity of Author's Source Code	No. Since the source code is not available, this point is also moot.
No Discrimination Against Persons or Groups	I think so. See section 11 of the license agreement for export restrictions.
No Discrimination Against Fields of Endeavor	I think so. While Sun's lawyers don't want to suggest that running your nuclear power plant with Solaris, they don't say that you can't.
Distribution of license	No. Since redistribution is not allowed, this point is moot.
...	...

Since one or more of the requirements to be called Open Source are not met by the license under which Solaris is distributed, Solaris is not open source.

People that need support don't want source code.

That is great in theory, but in practice it falls down a bit. Let's pretend that you need to write a dtrace script to dig into a thorny performance problem. If the stable dtrace providers don't provide probes at the right spot, you need to fall back on fbt or pid probes. The only way to understand what these are tracing is to read the source. Since the OpenSolaris and Solaris branch point is now about 3 years old, this is becoming extremely difficult to do reliably. The code that you get by browsing the latest OpenSolaris code sometimes does not match up with the fbt probes that are available in any release of Solaris. You may have more luck looking at historical versions of the source code, but that is a guessing game at best.

There are also times when a customer's needs just do not align with what Sun is willing to offer. Suppose you need different functionality from a device driver. It is possible that it is a trivial change from the customer's standpoint, so long as they have current source code. However, if the source code is not available, the best the customer can do is grab the same driver from somewhere else (e.g. opensolaris.org) and try to to maintain a special version of the driver and provide custom ports of all the bug fixes that they would otherwise get from from the Solaris source.

Suppose that hypothetical fix that the customer needed was something that Sun agreed was needed but they did not have the time to develop the fix. If the Solaris source code were as open as the OpenSolaris source code, the customer could work with the OpenSolaris community to get the fix integrated into OpenSolaris, then provide the backport to Solaris. If the customer could do this, the code would see the appropriate code reviews, development branch burn-in, etc. with minimal additional workload on Sun.

My Wish

I know that Sun went through tremendous work to make OpenSolaris happen. They should be commended for that and the other tens (hundreds?) of millions of lines of code they have opened up or written from scratch in the open. This gives them tremendous opportunity with Solaris 11 (assuming that 10+1 = 11). Keep the code that is open, well, open. When patches or other updates are released, be sure that it is clear in the open source code repository which files are used in building that update. To get the full benefit of this, it should be possible for the Solaris customer to set up a build environment to build this source.

It is OK if Sun doesn't want to support the code I modify. However, I would expect that they would support the unmodified parts of Solaris much the same as they do if I install a third-party package that adds some device drivers and mucks with some files in /etc.

Today's dtrace one-liners

Well, I've been spending a bit more time with dtrace lately. I'm now starting to do interesting things with dtrace without looking at the manual or snagging other scripts. Today I found some processes that looked like they weren't doing much of use. In particular, I saw the following in "prstat -mL"

   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID 
 15274 oracle    46  53 0.0 0.0 0.0 0.0 0.0 0.0   0  67 .22   0 racgmain/1
 15295 oracle    47  53 0.0 0.0 0.0 0.0 0.0 0.0   0  74 .23   0 racgmain/1
 16810 oracle    47  53 0.0 0.0 0.0 0.0 0.0 0.0   0  66 .23   0 racgmain/1
 25029 oracle    47  53 0.0 0.0 0.0 0.0 0.0 0.0   0  70 .23   0 racgmain/1

First, it is confusing that it says that these processes are doing less than one system call per second (.22 or .23), but spending more than 50% in system calls. Looks like a bug in prstat. To see what system call was being used...

# dtrace -n 'syscall:::entry/execname == "racgmain"/ { @[probefunc] = count() }
dtrace: description 'syscall:::entry' matched 232 probes
^C

  read                                                         269039

Ahh, so it is only reads that are causing all of this. Let's see which file descriptors are involved.

# dtrace -n 'syscall::read:entry/execname == "racgmain"/{ @[pid, probefunc,arg0] = count() }'
^C
    22469  read      9                7
    22469  read      4                7
    22469  read      3                9
    22469  read      8                9
    22469  read     10               10
    22469  read      7               21
    22469  read      6               24
    15274  read     10          1026462
    15295  read     10          1044932
    25029  read     10          1051450
    16810  read     11          1060720

It seemed rather odd that there would be so much reading on a single file descriptor per file (10 or 11, depending on the process). If this were really the case, I would expect to see some processes showing up in prstat that were high in %sys doing writes. Using pfiles on each of the pids showed that the file descriptor in question was a FIFO, but gave no indication what (if anything) was on the other end of the FIFO.

To help answer the question of what might be on the other end of the FIFO, I tried this (ok, not a one liner...)

#!/usr/sbin/dtrace -qs 

syscall::read:return
/execname == "racgmain"/
{
        @c[pid, arg0] = count()
}

tick-1s
{
        printf("%6s %6s %4s\n", " pid  ", " size ", "count");
        printa("%6d %6d %@4d\n", @c);
        clear(@c);
}

Which gave the following output...

 pid    size  count
 15295      0 117524
 15274      0 121065
 25029      0 121308
 16810      0 125358
 pid    size  count
 15274      0 108177
 25029      0 109639
 15295      0 115735
 16810      0 117175
 pid    size  count
 15295      0 119411
 15274      0 119471
 25029      0 120273
 16810      0 120650

Imagine that... these processes are each doing 100,000+ zero-byte reads per second. Time to tell the DBA's that they have some real problems with one of their daemons.