|
|
Last update:
Mon Oct 3 16:47:36 2005
Wed Oct 5 18:49:21 2005
Sat Oct 14 17:17:33 2006
Thu Mar 29 06:18:56 2007
Mon Mar 9 14:51:46 2009
Sat Jul 19 10:16:54 2014
Mon Nov 9 19:13:51 2015
Thu Mar 23 14:06:30 2017
Where is software stored on Unix systems?
Traditional Unix systems stored executable programs in /bin and /usr/bin, load libraries in /lib and /usr/lib, and manual pages in directories under /usr/man. Later systems added /sbin and /usr/sbin for software intended only for system administration.
The separation into the pairs /bin and /usr/bin kept directories small and speeded file lookups, with programs likely used by everyone held in the first, and more specialized software needed by only a subset of users relegated to the second. Another reason might be that /bin contained critical filesystem recovery tools, and thus, could be stored on a separate device so that it could be used for filesystem repair. Some modern Unix systems dispense with that separation, and simply make one of them a symbolic link to the other, and do that for library directories as well. More efficient filesystem design, and faster hardware, have largely removed the need for keeping directories small, and as a result, /bin has grown from holding perhaps 50 programs in the 1970s, to up to 4000 in the 2010s.
Before the Internet, operating systems were installed once, and then kept stable for months or years, until a major system upgrade installed new versions of most software.
The Internet radically changed that, and users of desktop computers with almost all operating systems are now accustomed to frequent software upgrades. Most Unix vendors provide some sort of package management system that allows managers to selectively upgrade software components, or add new ones, with as many as 40,000 packages available for some Unix distributions. Any single system is likely have only a few hundred of those packages installed.
With incremental updates, the problem is then: where should new software packages be installed? In the 1990s, the practice was that vendor-provided software was put in the usual system directories, and many sites therefore adopted a separate directory tree, called /usr/local, into which site-specific software could be installed. That made system upgrades easy: just save a copy of the /usr/local tree and all user home directories, reinstall the operating system and its packages (which usually erases the disk, and then writes the new contents), and then restore the saved directories. The GNU Project in particular adopted that model, and GNU software packages almost always represent the name of the local tree with the prefix variable, whose default value is always /usr/local.
Unfortunately, some Unix package systems, notably on some flavors of BSD Unix, choose to install added packages in /usr/local, causing havoc if site-specific software is also installed in that tree. Some compilers, notably the GNU ones, automatically search for C/C++ header files under /usr/local/include, before looking in the conventional /usr/include directory. That regrettable practice makes it hard to control which header files should be used for a particular compilation.
At the University of Utah Mathematics Department, we have long followed the GNU Project recommendations for filesystem layout. However, in 2014, we decided to abandon that practice, and on all new operating systems, site-specific software is installed in directories under /usr/uumath. The change will take several years, because there is a huge amount of software in the old /usr/local tree on most of our systems, so both trees will co-exist for some time. However, the PATH variable will normally be set with /usr/uumath/bin ahead of /usr/local/bin, so that newer software versions are preferred. Just type the command echo $PATH to see what the program-search order is for your login session.
In the rest of this FAQ, wherever you see /usr/local, just remember that it might now be /usr/uumath on the system that you are currently using.
Our facility is more than 25 years old, and has a huge amount of installed software. Depending on the platform, the default system search path on our systems contains between 2000 and 15,000 programs, which you can readily test on any of them with this simple command: pathfind -a PATH '*' | wc -l
In addition to Unix-vendor-provided software, our systems have all of the important GNU utilities from the Free Software Foundation, the Portland Group and Intel compilers (Fortran 77/90/95, C, C++) for IA-32, IA-64, and AMD64, the PathScale compilers (Fortran 77/90/95, C, C++) for AMD64, the Princeton/AT&T lcc C compiler, the LLVM + Clang C/C++ compilers, and on some systems, the commercial Magma, Matlab, Maple, Mathematica, and S-Plus systems for numerical and symbolic mathematics, and statistics. In addition to S-Plus, have the R program for statistics, which was originally developed to be compatible with S-Plus, and now has thousands of contributed packages. We also have the GNU GSL scientific libraries that can be used from Fortran, C, and C++ programs. Further information about those, and many more, packages can be found at our software Web site: http://www.math.utah.edu/software/.
Where are software source distributions stored?
There are five main directory trees for software source archives:
| Directory tree | Contents |
| /usr/local/src | public non-GNU freeware |
| /usr/local/sys/src | commercial licensed software |
| /usr/local/gnu/src | GNU freeware |
| /usr/local/math | mathematical freeware |
| /u/ftp/pub/ | freeware packages in our FTP tree |
Each of those has subdirectories named by software package. For the first two, each subdirectory generally contains a WHERE-FROM file that records the origin elsewhere on the Internet of the package. The GNU packages generally come from their official source site, ftp://ftp.gnu.org/gnu/, but some also have WHERE-FROM files to record developer sites that sometimes have newer versions than the official GNU ones.
Within each package directory are distribution files, typically with names ending in .tar.gz or .tar.bz, indicating Unix tar archives compressed with gzip or bzip2. There may also be README.* files that record build problems.
Each package directory contains a logs subdirectory, in which subdirectories named by particular package versions contain build logs. Many of them are generated by the automated software-build system, build-all, described in Chapter 8 of the book Classic Shell Scripting.
What packages can I copy to my personal computer?
You are welcome to copy anything from the three freeware directories listed above. Under no circumstances may the commercial distributions be copied to other systems. If in doubt, please ask our systems staff.
Whenever you copy a software distribution from those directories, or any other Internet sites, please check for accompanying detached digital-signature files with extension .asc or .sig. Copy those files as well, and then verify the digital signatures before attempting to build software.
By doing so, you can avoid nasty surprises if the software package has been maliciously modified by an attacker, something that has already happened at major software distribution sites.
A tutorial on digital signatures is available here.
How do I run Unix programs on my Microsoft Windows computer?
There are several ways to do this:
More information on this topic is available here.
How do I run Microsoft Windows applications on my Unix system?
Fewer alternatives are available than for the reverse environment, but you might find a workable solution by following this link.
I use program x on Microsoft Windows. What is its analogue on Unix?
Some software packages run in both Microsoft Windows and Unix environments under the same, or similar, names. In general, however, one needs a Rosetta Stone to translate from one package name to another. One such source is a Web page that lists Linux software equivalent to Windows software.
Where are binary programs, header files, and load libraries installed?
Following widespread and long-standing Unix practice, and also GNU conventions, software installations for free packages are done with a path prefix of /usr/local, so binary programs are installed in /usr/local/bin, C/C++ header files in /usr/local/include, and load libraries in /usr/local/lib. [However, see the beginning of this document for news of the migration to the prefix /usr/uumath,]
Some platforms support two or more memory models and/or load library formats, so to distinguish between identically named libraries, they must be stored in different directories. In the following table, the default library directories are highlighted:
| Vendor/CPU and O/S | 32-bit libraries | 64-bit libraries |
| Apple PowerPC Mac OS X | /usr/local/lib | /usr/local/lib64 |
| AMD64 GNU/Linux | /usr/local/lib | /usr/local/lib64 |
| HP/Compaq/DEC Alpha OSF/1 | /usr/local/lib32 | /usr/local/lib |
| IA-64 GNU/Linux | /usr/local/lib32 | /usr/local/lib |
| SGI MIPS IRIX | /usr/local/lib, /usr/local/lib32 | /usr/local/lib64 |
| Sun SPARC Solaris | /usr/local/lib | /usr/local/lib64, /usr/local/lib/64 |
To link C and Fortran programs with one of those libraries, use something like this:
% cc -I/usr/local/include myprog.c -o myprog -L/usr/local/lib -lgsl % f77 myprog.f -o myprog -L/usr/local/lib -llapack
The compilers and linkers on some operating systems, and the GNU compilers, automatically search /usr/local/include for header files and /usr/local/lib for libraries, so you may not require the -Ipath and -Lpath options.
With shared libraries, it is usually desirable to record in the executable program where the libraries were found at link time, to avoid the need to set the LD_LIBRARY_PATH environment variable at run time. Regrettably, practices differ across Unix systems, but commands like one of these should work, after adjusting the library pathname appropriately:
% cc -I/usr/local/include myprog.c -o myprog -L/usr/local/lib -lgmp -Wl,-rpath,/usr/local/lib % cc -I/usr/local/include myprog.c -o myprog -L/usr/local/lib -lgmp -R/usr/local/lib
There are often multiple versions of installed libraries, and those in the /usr/local tree are almost always the newest. Use the ldd command, available on most systems, to find which libraries are used. Here is an example from a Sun Solaris SPARC system:
% ldd /usr/local/bin/ndiff libgmp.so.3 => /usr/local/lib/libgmp.so.3 libCrun.so.1 => /usr/lib/libCrun.so.1 libm.so.1 => /usr/lib/libm.so.1 libw.so.1 => /usr/lib/libw.so.1 libc.so.1 => /usr/lib/libc.so.1 libm.so.2 => /lib/libm.so.2 /platform/SUNW,Sun-Blade-1000/lib/libc_psr.so.1
Where are programs and library software documented?
Unix manual pages, accessed with the man command, are the most common, and briefest, form of software documentation:
% man -a sqrt
The -a option ensures that all manual pages of the given topic name are displayed, because there may be more than one.
Manual pages are traditionally divided into several sections: 1 for commands, 2 for system calls, 3 for library calls, and so on, corresponding to subdirectories man1, man2, man3, .... You can request the manual page from a specific section like this:
% man -s 3m sqrt
Many of the manual-page files are also available in HTML form, and can be visited by browsers at the local URL file:///usr/local/man/html/ . Those files are not visible from Web browsers on remote computers.
Typeset versions of some manual-page files, suitable for printing and viewing, are available in PDF (Portable Document Format) and PostScript form, at file:///usr/local/man/ps/ and file:///usr/local/man/pdf/ .
The locally written man2ps program can be used to generate typeset manual pages in PostScript:
% man2ps /usr/local/man/man1/gcc.1 > /tmp/gcc.ps
You can convert PostScript to PDF with at least four different programs, as shown by these examples:
% distill /tmp/gcc.ps % pstill /tmp/gcc.ps % ps2pdf /tmp/gcc.ps % convert /tmp/gcc.ps /tmp/gcc.pdf
In each case, the output PDF file is /tmp/gcc.pdf.
Most GNU software, and many other packages that we have developed or installed, are documented in the emacs info system. You can access it inside the editor by typing C-h i, and externally, with the standalone info and xinfo programs. The info system is an easy-to-use hypertext system that predates HTML and the World-Wide Web by 15 years. Once inside, type ? for brief help, h for a tutorial, and d to return to the top-level directory. Type q to quit.
In emacs, a subsequent entry to the info system puts you back where you last left it, so often, no further navigation is needed. Also, your navigation history is preserved, as long as you do not exit the editor.
The info system provides detailed documentation, and for many programs and packages, their info nodes are on-line indexed books that, if printed, would be hundreds of pages long.
Other locations where you can find on-line documentation include the /usr/local/doc tree, and for classroom software, the /u/cl/doc tree. For example, under the first of those, the c, cs, f77, fortran, and pascal directories contain online books and drafts of international standards for the C, C++, C#, Fortran, and Pascal programming languages.
Commercial and other software usually has its own documentation, described in the frequently asked questions (FAQ) pages for Magma, Maple, Maxima, Mathematica, Matlab and Octave, MuPAD, SAS, S-Plus and R, TeX and Metafont, and others listed in the top-level computing facilities FAQ directory.
What versions of software are available?
Many modern software programs provide a command-line option to report version information. The option is variously called -V, -v, -version, or best, the GNU standard --version. Programs that provide the latter normally also offer a --help option that provides a brief reminder of common options, sometimes removing the need to consult manual pages or info documentation. Here are some examples:
% cc -V cc: Sun C 5.11 SunOS_i386 145355-01 2010/10/11 usage: cc [ options ] files. Use 'cc -flags' for details % gcc --version gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-16)
Our practice has long been to preserve older versions of software, to minimize impact on users when newer versions behave slightly differently. Here is how to find what other versions there might be beyond the default:
### Where is the default GNU C compiler, gcc, found?
% which gcc
/usr/local/bin/gcc
### Where else are there gcc compilers?
### [The leading backslash forces use of a utility program,
### rather than the shell built-in that reports only one answer.]
% \which -a gcc
/usr/local/bin/gcc
/usr/bin/gcc
/usr/local/gnat/bin/gcc
### What other versions of gcc are available?
% ls /usr/local/bin/gcc-*
/usr/local/bin/gcc-2.95.3 /usr/local/bin/gcc-4.4
/usr/local/bin/gcc-3.3.1 /usr/local/bin/gcc-4.4-20081017
/usr/local/bin/gcc-3.3.3 /usr/local/bin/gcc-4.4-20081128
... many lines omitted ..
### What minor versions of gcc are available?
% ls /usr/local/bin/gcc-4.{?,??}
/usr/local/bin/gcc-4.1 /usr/local/bin/gcc-4.4 /usr/local/bin/gcc-4.8
/usr/local/bin/gcc-4.10 /usr/local/bin/gcc-4.5 /usr/local/bin/gcc-4.9
/usr/local/bin/gcc-4.2 /usr/local/bin/gcc-4.6
/usr/local/bin/gcc-4.3 /usr/local/bin/gcc-4.7
### What subminor versions are there?
% ls /usr/local/bin/gcc-4.9*
/usr/local/bin/gcc-4.9 /usr/local/bin/gcc-4.9-20140702
/usr/local/bin/gcc-4.9-20130414 /usr/local/bin/gcc-4.9.1
### Which subminor version is the default?
% file /usr/local/bin/gcc-4.9
/usr/local/bin/gcc-4.9: symbolic link to `gcc-4.9-20140702'
While a command-line request works for many programs, how can we find the version of a library? That issue was long ignored in the Unix world, so there are a few other ways to discover such information.
Some more recent libraries provide an associated program, usually implemented as a short shell script, that supplies version information:
% libpng-config --version 1.5.12
Such wrapper programs are commonly named with the suffix -config on the basename of the library file. Here, that library would be something like libpng.a, libpng.so, or libpng.dylib, depending on the operating-system conventions and practices.
If the library developers failed to supply a suitable wrapper program, there is still likely to some information inside the library that reveals its version. Load libraries are binary files that are rarely easy to examine with a text editor, but there is a easy solution: use the strings utility to extract and report apparent text strings:
### How many `text' strings are there in the C runtime library?
% strings -a /lib/libc-2.5.so | wc -l
13055
### That is far too many to search by eye, so let's just count the
### ones that contain the word `version':
% strings -a /lib/libc-2.5.so | grep -c version
34
### That is a reasonable number for a human to scan, so now display
### them, sorting the output so that we can discard duplicates.
% strings -a /lib/libc-2.5.so | grep -i version | sort -u
The C stubs add-on version 2.1.2.
crypt add-on version 2.1 by Michael Glad and others
...
Compiled by GNU CC version 4.1.2 20080704 (Red Hat 4.1.2-54).
GNU C Library stable release version 2.5, by Roland McGrath et al.
...
versionsort64@GLIBC_2.1
The three standard Unix tools used in the pipeline in the last example are handy ways of reducing voluminous data to permit examination by an intelligent human: the computer does the drudge work of discarding lots of uninteresting material. Our probe not only reveals the C library version, 2.5, it also tells us which compiler was used to compile the library code, and that, as is expected in Unix, the library is written largely in the C language. There could be occasional snippets of machine-specific assembly code embedded in __asm__(...) directives in the C code, but they cannot be discovered by examination of the library text strings.
GNU/Linux distributions that supply the Red Hat Package Manager, rpm, permit one to ask to which package a particular file belongs:
% rpm -q -f /lib/libc-2.5.so glibc-2.5-118.el5_10.2
Here we discover that the C library on that system is the GNU C Library, version 2.5, patch level 118, for operating system release Enterprise Linux version 5.10.2.
For GNU/Linux distributions that have the pacman utility, we can use a query-ownership request, like this:
% pacman -Q -o /usr/lib/libc.so.6 /usr/lib/libc.so.6 is owned by glibc 2.19-3
For GNU/Linux distributions that provide the apt-get utility, use a command like this:
% dlocate /lib/x86_64-linux-gnu/libc.so.6 libc6:amd64: /lib/x86_64-linux-gnu/libc.so.6