Talk:Unix filesystem

/sbin's name

Pretty sure that sbin intially stood for static linked binaries. — Preceding unsigned comment added by 173.55.202.3 (talk) 08:16, 3 May 2012 (UTC)

Given that /usr/sbin also exists, and doesn't contain statically-linked binaries, I'm not sure about that. Guy Harris (talk) 08:35, 19 November 2013 (UTC)

I always understood it to mean "system administration binaries", but I can't remember ever having an explicit explanation of the name. QVVERTYVS (hm?) 23:16, 19 November 2013 (UTC)

I have the impression that ("system administration", or something such as that) was what the "s" meant. Guy Harris (talk) 00:09, 20 November 2013 (UTC)

/usr/libexec is in FHS

Early versions of the FHS did not include /usr/libexec; this has since been rectified: http://www.linuxbase.org/betaspecs/fhs/fhs/ch04s07.html

That URL has "betaspecs" in it; the /usr section of the 2.3 version of the FHS doesn't mention /usr/libexec. Is there a later official version of the FHS that includes it, or is this something that will appear in a future release, such as 3.0? Guy Harris (talk) 01:33, 17 November 2013 (UTC)

So where did /sbin, /usr/sbin, and /var come from?

Actually, unless my memories are too faded, I know where they came from - some people at Sun, around the time SunOS 4.0 was being developed, were making some changes to the directory layout, oriented towards NFS-only diskless workstations (when Sun were killing of the ND remote-disk-access protocol), and one of the changes was the introduction of /var (to separate read-only stuff in /usr, with diskless workstations mounting the same export on /usr, from read-write stuff in /var, with each diskless workstation mounting its own private directory tree on /var), and another was splitting (/usr)/bin from (/usr)/sbin (not related to diskless workstations, but it was a cleanup done while they were at it, so that users who didn't need the administrative tools didn't have to have them in their $PATH).

I seem to remember a document written describing this, and possibly even being circulated outside Sun (possibly amongst licensees of NFS and/or the BSD folk). However, I can't seem to find any trace of that document online. Anybody have less-faded memories than mine? Guy Harris (talk) 01:26, 17 November 2013 (UTC)

I don't have the document you want, but wouldn't /sbin have been created for the same reason? In 4.3BSD/early System V, /etc contained (writable) configuration files as well as system programs, e.g. /etc/init (now /sbin/init). QVVERTYVS (hm?) 09:58, 18 November 2013 (UTC)

(/etc/init dates back even earlier, at least to V6.)

Yeah, I think that was another reason for creating /sbin.

I think the split between /sbin and /usr/sbin, and between /bin and /usr/bin, was between "stuff we don't want to require shared libraries" and "stuff that can use shared libraries"; "stuff we don't want to require shared libraries" included "stuff that had to run before we had the file system containing the shared libraries mounted", as well as "stuff we'd like to be able to run without shared libraries so that somebody can install a test build of a shared library after saving the old version, and then back out the change in case the new shared library doesn't work", so SunOS 4.0 had either /bin/mv or /bin/cp for putting the old shared library back.

But this is all fading memories from the mid '80's, which is why I wish the stuff the Sun folks had written about this was available somewhere. Guy Harris (talk) 20:16, 18 November 2013 (UTC)

The split between /bin and /usr/bin was because they (Kernigan, Ritchies, Thompson, et al) ran out of disk space on the pack (UNIX was written on a PDP!) that had /bin, so they made /usr/bin. See http://lists.busybox.net/pipermail/busybox/2010-December/074114.html which was referenced from http://www.osnews.com/story/25556/Understanding_the_bin_sbin_usr_bin_usr_sbin_Split/ 174.46.232.2 (talk) 18:19, 3 April 2014 (UTC)

That was the original split. The directory layout was changed in SunOS 4.0, with /sbin and /usr/sbin and /var being introduced, and with, as I remember, a bunch of stuff moved from /bin to /usr/bin. Diskless workstations had /bin on a per-machine root file system and /usr/bin on a shared read-only /usr file system, so they moved as many programs as possible to the shared /usr/bin and moved all writable files from the read-only /usr to a per-machine writable /var. Guy Harris (talk) 19:01, 3 April 2014 (UTC)

Other Unices, other filesystems

Would it be useful to describe -- or even mention -- variant file systems from other UNIX variants? Specifically, I was thinking of the scheme in SCO Unix which was... er, different. (Probably because it was derived from Xenix, which was its own unique thread of the UNIX split. All I know is encountering SCO 10 years ago, scratching my head over the file system, & bewilderedly muttering, "Okay....") -- llywrch (talk) 20:37, 28 April 2014 (UTC)

The article discusses variants in filesystems in several places, although so far I've tried to keep it a general overview and I would like to keep it that way rather than discussing every minute detail of every single Unix variant and Linux distro that ever existed. What filesystem differences do you have in mind? The location of standard files? Different file types? The implementation? QVVERTYVS (hm?) 21:35, 28 April 2014 (UTC)

In the case of SCO Unix -- & I'm going on memory here -- it originally had filesystems with names starting with two digits with a string indicating the purpose of the files in the directory. Then, when Santa Cruz decided to adopt Sys V standards, all of these directories were symlinked to the familiar directories (e.g. /bin, /sbin/, /var, etc.). Every other version of UNIX I've worked with -- Solaris, SunOS, FreeBSD, Linux -- has kept pretty close to the standard, although with the occasional quirk, such as /opt. It was strange enough for the general sense to stick in my mind, despite the fact I've since tried to forget as much about SCO Unix as I could. Had this peculiarity been documented under either Xenix or SCO Unix, I wouldn't think of mentioning it, & I'm not going to push for it even now. However, I think an example or two are worth mentioning in the article as a reason why a filesystem standard was considered a good thing, & likely arose somewhat later in UNIX history. -- llywrch (talk) 16:15, 29 April 2014 (UTC)

/var/tmp

Schily, why is the description of /var/tmp "superfluous text"? It's in several of the hier(7) manpages cited. QVVERTYVS (hm?) 14:40, 15 February 2015 (UTC)

If this was for /var/tmp/ than I would expect a clear explanation that describes the difference. Note that /tmp is usually (since 1988) tmpfa mounted while /var/tmp is not. Schily (talk) 15:05, 15 February 2015 (UTC)

/tmp is described as: "A place for temporary files. Many systems clear this directory upon startup; it might have tmpfs mounted atop it, in which case its contents do not survive a reboot, or it might be explicitly cleared by a startup script at boot time." The contrast with /var/tmp should be clear from the description. If it's not, then why remove it rather than improve it? Never mind, I see the issue now. QVVERTYVS (hm?) 15:08, 15 February 2015 (UTC)

Tmpfs limitations

Of all of the tmpfs implementations, the Linux one appears to be the only one that imposes a limit based solely on the size of RAM. Solaris's doesn't, and, according to the NetBSD man page, the FreeBSD man page, and the OpenBSD man page, the file system size limit is based, by default, on the sum of the RAM size and the swap size. says the same thing, as does. Guy Harris (talk) 18:43, 27 March 2015 (UTC)

We also already have an article about tmpfs, and I think any discussion of details of it should be relegated there. This article is an overview of the Unix filesystem and its conventions. QVVERTYVS (hm?) 20:15, 27 March 2015 (UTC)

I.e., for /tmp, just say something such as "A place for temporary files that do not have to survive a reboot.", with the possible addition of a brief mention of the possibility of it being a tmpfs file system, with a link to tmpfs, and nothing about the characteristics of tmpfs? Sounds good to me. Guy Harris (talk) 20:28, 27 March 2015 (UTC)

Sorry, have seen your post here a little late. It would seem that we are approximately in agreement that tmpfs is somehow limited by (size of RAM + size of swap) and although the details of this restriction are different on every platform and can be usually tweaked by the sysadmin it may be a significant enough restriction that it could be at least mentioned here. Richiez (talk) 12:44, 28 March 2015 (UTC)

Yes, like all file systems, tmpfs is going to be limited by the amount of backing store it has. I've used systems where a disk-based /tmp had less space than /{usr,var}/tmp, with /tmp being on a small root partition and /{usr,var}/tmp being on a larger /usr partition, /var partition, or a partition of its own, so about all I'd say about /tmp, tmpfs or no, is that it might be significantly limited in the total amount of available space, with no indication of whether that's the typical case or not (the machine on which I'm typing this has only one partition, a root partition, for all data, with both /tmp and /var/tmp on the root partition, and it runs a very common desktop UN*X). Guy Harris (talk) 18:45, 28 March 2015 (UTC)

Stating that /tmp might be significantly limited in amount of available space looks good to me. Also I think many systems run for months without reboot so somehow it should also say that the files are expected to be cleaned regularly, not only on reboot. Richiez (talk) 11:57, 29 March 2015 (UTC)

I wouldn't go so far as to say "expected", but I would say that old files might be removed without a reboot. Guy Harris (talk) 18:27, 29 March 2015 (UTC)

It is of course subject to configuration but users/software should expect that files in /tmp may be cleaned up regularly. Afaik 1-7 days based on atime are common? Richiez (talk) 11:32, 30 March 2015 (UTC)

Stating that tmpfs is a small filesystem with limited space is definitely a false claim as the limitation for tmpfs is sizeof RAM + sizeof swap and this typically is much more than /usr/tmp. I am disappinted to see that you repeatedly introduce this false claim. Schily (talk) 14:15, 30 March 2015 (UTC)

Stating broadly, for all platforms, that tmpfs is, or isn't, a small file system with limited space is definitely a false claim, as:

• on Linux, the documentation says

   tmpfs has three mount options for sizing:
   
   size:      The limit of allocated bytes for this tmpfs instance. The 
              default is half of your physical RAM without swap. If you
              oversize your tmpfs instances the machine will deadlock
              since the OOM handler will not be able to free that memory.

• on Solaris, the size limit is based on the size of RAM plus swap, and I can personally attest to sticking large files on tmpfs on Solaris.

So it would be a mistake to say that /tmp is a small file system or a file system that will be cleared or reboot or that will have files not recently referred to removed. We should, at most, note that it might be small (it is quite literally no smaller than /var/tmp on my machine, as they're on the same partition, and HFS+ doesn't have per-directory size limits) and that files might be removed on reboot or might be removed after some period of time, depending on whether the OS you're using happens to use tmpfs or clean /tmp on reboot (that's not a requirement, and, in fact, FreeBSD 10.1 didn't remove a file in /tmp on reboot when I tried it just now) or whether /tmp happens to be on a small root partition or whether it happens to be a tmpfs file system that imposes a RAM-only-based size limitation. Guy Harris (talk) 17:22, 30 March 2015 (UTC)

Tmpfs on Solaris supports large files if the kernel is running in 64 bit mode. This is because a 32 bit kernel cannot deal with an address space > 4 GB (which still is larger than the official large file limit of 2**31-2 ;-) Schily (talk) 13:09, 31 March 2015 (UTC)

the thing with filesystems, people know how big their hard drives are. Compared to that RAM+swap is usually ridiculously small, sometimes less then the memory of a mediocre phone and imho this is worth noting.Richiez (talk) 22:29, 30 March 2015 (UTC)

"Compared to that RAM+swap is usually ridiculously small" "Usually" is different from "always". This isn't a howto, so the purpose here isn't to give advice to personal workstation users as to how big their /tmp directory is and how large a file they should expect to be able to put there, it's to give a picture that includes everything from a smartphone to a huge server.

On a machine running Solaris 10, on which I have an account (it's not mine, so I don't know what it's physical configuration is), /tmp has a 44GB /tmp and a 161 GB /var/tmp, so that's a factor of 3.65 difference in size. (As for phones, that's about halfway between, for example, the smallest Samsung Galaxy S6 and the next storage size up and halfway between twice the smallest iPhone 6 and the next storage size up.)

(/tmp and /var/tmp are, as noted, exactly the same size on my machine, which is running the UN*X that, as far as I know, has the largest desktop/notebook market share. It doesn't have tmpfs, however, and it has multiple temporary-file directories, including per-user ones, so using tmpfs in that context would be a bit of work; it also doesn't have a swap partition, it swaps to files in the file system, and adds new swap files as necessary and removes swap files when possible, so if there were an OS X tmpfs, it might well let you fill up the entire disk, just as you can with the regular /tmp and /var/tmp that default to both being on the root partition. It might still be faster, e.g. because it would only push tmpfs pages to the backing store if the VM system needed a free page, it wouldn't have to do so on a sync() or fsync() call, and thus might also be a bit gentler on SSDs.)

So I'd be willing to have the article note that /tmp might be limited in the amount of space available on it, but not willing to have it make a bold statement that it's only for "small" files, unless you'd consider, say, a 32 GB file, which would fit quite nicely on the tmpfs-based /tmp on the Solaris box I referred to earlier, a "small" file. Guy Harris (talk) 23:23, 30 March 2015 (UTC)

@ Schilly: I am wondering how much RAM your system has and if you designate most of your hard disk space to swap? Because on a typical workstation the RAM+swap is 100-1000x smaller than the available hard disk space. So on a typical workstation any tmpfs based system would be "small". Also, why are you talking about /usr/tmp all the time? It is not even mentioned in the article page as far as I can see. Richiez (talk) 22:22, 30 March 2015 (UTC)

Is what matters the available hard disk space or the available space in temporary file directories?

(And perhaps he's talking about /usr/tmp because he, like me, used and developed on and for UN*X systems since before /var even existed; I tend to speak of /var/tmp, which the article most definitely does discuss, rather than /usr/tmp now that all the UN*Xes out there have adopted the SunOS 4.x-derived directory layout with /var, but perhaps Schily's a bit more old-fashioned than I am. :-)) Guy Harris (talk) 23:30, 30 March 2015 (UTC)

I believe it was you whom I met in December 1987 in the ballroom of the Fairmount hotel in San Francisco at the Auspex booth on the SUG meeting...or was it December 1988 at the Fauntainebleau Hilton in Miamy Beach? I was working for H.Berthold AG at that time, the company that sold 1/4 of all Sun's made in the 1980s and we ordered the first Sun to Europe in January 1985.

I believe that we are of comparable age (you might be a bit older). At that time, tmpfs has been invented by Sun and typical values for workstations at that time have been 4MB RAM, 20MB swap, 10MB root partition with aprox. 1MB free space, 60 MB /usr with aprox. 10 MB free space. So before using tmpfs, /tmp indeed was small compared to /usr/tmp (OK, /var/tmp was introduced with SunOS-3.x in 1986 to support diskless clients and to allow to mount /usr read only).

With tmpfs, things changed dramatically, as there suddenly have been 20 MB free space in /tmp while there still only have been 10 MB in /var/tmp. As a result, the Sun C-compiler put tmp files to /tmp instead of /var/tmp and gcc frequently failed to compile larger projects because gcc stayed in the past and believed /var/tmp/ was still a good choice.

11 years ago, I build my first Opteron based PC and there are 4 GB RAM, 10 GB swap, 10 GB root fs with 1 GB free space and /var/tmp is now on that space, so I have 10 GB of free space in /tmp and 1 GB of free space on /var/tmp. The ratio did not really change since 1988. But even on a newer server at my university, /var/tmp/ is not larger than /tmp. So the text introduced by User:Richiez is at least missleading and if we rate tmpfs, we should look at the original implementation instead of a Linux clone. Schily (talk) 12:07, 31 March 2015 (UTC)

Add the /run directory

The run directory is a new addition to Linux that contains files that programs need through reboots, because /tmp is vulnerable to getting wiped. Is this a useful addition to the article? I'm asking this because this is article for the industry standard Unix filesystem, and not anything added by an OS. — Preceding unsigned comment added by One Blue Hat (talk • contribs) 20:51, 16 January 2020 (UTC)

One Blue Hat, I'd say probably not, because, as you said, this is based on the standard UNIX filesystem. MoonyTheDwarf (Braden N.) (talk) 20:55, 16 January 2020 (UTC)

Yes - /run is already covered in Filesystem Hierarchy Standard, but not all Unix-like systems follow the FHS in its entirety. (macOS, for example, currently has /var/run but not /run.) Guy Harris (talk) 21:17, 16 January 2020 (UTC)

Merge with Filesystem Hierarchy Standard?

The Conventional directory layout section contains information also in Filesystem Hierarchy Standard. Should the two be merged? — Preceding unsigned comment added by 151.53.72.73 (talk) 23:22, 9 September 2020 (UTC)

They both contain that information because the FHS is based on conventional UN*X usage, as per the quote from that section:

The details of the directory layout have varied over time. Although the file system layout is not part of the Single UNIX Specification, several attempts exist to standardize (parts of) it, such as the System V Application Binary Interface, the Intel Binary Compatibility Standard, the Common Operating System Environment, and Linux Foundation's Filesystem Hierarchy Standard (FHS).

I wish I could find a reference for this, but at least some of the layout comes from a proposal that some people at Sun made in the 1980's containing, among other things, the /var directory; at least part of the goal was better support for diskless workstations, with per-machine and shareable information divided up so that they could be on different remote-mounted file systems.

Not every system that uses some form of the conventional layout follows the FHS. Guy Harris (talk) 00:23, 10 September 2020 (UTC)