Lesson 1.4: OverlayFS

OverlayFS is a union filesystem that merges multiple directories (layers) into a single unified view. It's widely used in Docker/LXC for container filesystems.

OverlayFS Structure

Lower layers (l1, l2, l3): Read-only (like Docker image layers).
Upper layer (l4): Writable (like container’s ephemeral layer).
Workdir: Temporary directory for atomic operations.
Mount point (mount): Unified view of all layers.

[root@hssl overlay-example]# mkdir -p mount
[root@hssl overlay-example]# mkdir -p /root/overlay-example/{l1,l2,l3,l4,workdir}
[root@hssl overlay-example]# ls
l1  l2  l3  l4  mount  workdir
[root@hssl overlay-example]# mount -t overlay overlay-test -o lowerdir=/root/overlay-example/l1:/root/overlay-example/l2:/root/overlay-example/l3,upperdir=/root/overlay-example/l4,workdir=/root/overlay-example/workdir /root/overlay-example/mount

.
├── l1/       # Lowerdir 1 (base)
├── l2/       # Lowerdir 2 
├── l3/       # Lowerdir 3 (topmost read-only)
├── l4/       # Upperdir (writable)
├── workdir/  # Workdir (must be empty)
└── mount/    # Merged view

How Files Are Merged

File Lookup Order
- When you read /mount/l1.txt, OverlayFS checks layers from top to bottom:
  - Upperdir (l4) → If file exists, return it.
  - Lowerdirs (l3 → l2 → l1) → Return the first found.
Example
- l1.txt exists in l1/ → Appears in mount/.
- l4.txt exists only in l4/ → Also appears in mount/.

[root@hssl overlay-example]# echo "hello from l1" > /root/overlay-example/l1/l1.txt
[root@hssl overlay-example]# echo "hello from l2" > /root/overlay-example/l2/l2.txt
[root@hssl overlay-example]# echo "hello from l3" > /root/overlay-example/l3/l3.txt
 
# Viewing the created files 
[root@hssl overlay-example]# cat /root/overlay-example/l1/l1.txt 
hello from l1
[root@hssl overlay-example]# cat /root/overlay-example/l2/l2.txt 
hello from l2
[root@hssl overlay-example]# cat /root/overlay-example/l3/l3.txt 
hello from l3
 
# Viewing the mount files
[root@hssl overlay-example]# cat /root/overlay-example/mount/l1.txt 
hello from l1
[root@hssl overlay-example]# cat /root/overlay-example/mount/l2.txt 
hello from l2
[root@hssl overlay-example]# cat /root/overlay-example/mount/l3.txt 
hello from l3
 
[root@hssl overlay-example]# tree
.
├── l1
│   └── l1.txt
├── l2
│   └── l2.txt
├── l3
│   └── l3.txt
├── l4
├── mount
│   ├── l1.txt
│   ├── l2.txt
│   └── l3.txt
└── workdir
    └── work
 
# Creating file l4
[root@hssl overlay-example]# echo "hello from l4" > /root/overlay-example/l4/l4.txt
[root@hssl overlay-example]# cat /root/overlay-example/l4/l4.txt 
hello from l4

Copy-on-Write (CoW) Mechanism

[root@hssl overlay-example]# echo "Copy Up Test 1" >> /root/overlay-example/mount/l3.txt 
[root@hssl overlay-example]# cat /root/overlay-example/mount/l3.txt 
hello from l3
Copy Up Test 1
[root@hssl overlay-example]# cat /root/overlay-example/l3/l3.txt 
hello from l3
 
[root@hssl overlay-example]# tree
.
├── l1
│   └── l1.txt
├── l2
│   └── l2.txt
├── l3
│   └── l3.txt
├── l4
│   ├── l3.txt
│   └── l4.txt
├── mount
│   ├── l1.txt
│   ├── l2.txt
│   ├── l3.txt
│   └── l4.txt
└── workdir
    └── work
 
[root@hssl overlay-example]# cat /root/overlay-example/l4/l3.txt 
hello from l3
Copy Up Test 1

OverlayFS checks l4/ (upperdir) for l3.txt → Not found.
Copies l3.txt from l3/ (lowerdir) to l4/ ("copy-up").
Appends the new line to l4/l3.txt.
Original l3/l3.txt remains unchanged (hello from l3).
Modified version is in l4/l3.txt (hello from l3\nCopy Up Test 1).
Why CoW?
- Preserves immutability of lower layers (critical for Docker images).
- Ensures isolation between containers sharing the same image.

File Deletion & Whiteout Files

[root@hssl overlay-example]# rm -rf /root/overlay-example/mount/l2.txt 
[root@hssl overlay-example]# ls /root/overlay-example/mount/
l1.txt  l3.txt  l4.txt
 
# removed l2.txt from mount, but exists in l2/l2.txt
[root@hssl overlay-example]# tree
.
├── l1
│   └── l1.txt
├── l2
│   └── l2.txt
├── l3
│   └── l3.txt
├── l4
│   ├── l2.txt
│   ├── l3.txt
│   └── l4.txt
├── mount
│   ├── l1.txt
│   ├── l3.txt
│   └── l4.txt
└── workdir
    └── work
        └── #20
 
# Creates a caracter type file called white out file in the editable layer
[root@hssl overlay-example]# cd l4 
[root@hssl l4]# ls
l2.txt  l3.txt  l4.txt
[root@hssl l4]# stat l2.txt 
  File: l2.txt
  Size: 0         	Blocks: 0          IO Block: 4096   character special file
Device: fd00h/64768d	Inode: 17798677    Links: 2     Device type: 0,0
Access: (0000/c---------)  Uid: (    0/    root)   Gid: (    0/    root)
Context: unconfined_u:object_r:admin_home_t:s0
Access: 2025-05-05 07:47:22.509096314 +0545
Modify: 2025-05-05 07:47:22.509096314 +0545
Change: 2025-05-05 07:47:22.509096314 +0545
 Birth: 2025-05-05 07:47:22.509096314 +0545

OverlayFS cannot delete from read-only l2/.
Instead, it creates a "whiteout" file in l4/:
A character device file with c-------- permissions.
Tells OverlayFS: "Hide l2.txt from the merged view."
Effect: l2.txt disappears from mount/ but still exists in l2/.

OverlayFS storage driver (Docker)

The image layers

[root@hssl overlay2]# pwd
/var/lib/docker/overlay2
[root@hssl overlay2]# ls
backingFsBlockDev  l
 
[root@hssl overlay2]# docker pull ubuntu
Using default tag: latest
latest: Pulling from library/ubuntu
49b96e96358d: Pull complete 
Digest: sha256:1e622c5f073b4f6bfad6632f2616c7f59ef256e96fe78bf6a595d1dc4376ac02
Status: Downloaded newer image for ubuntu:latest
docker.io/library/ubuntu:latest

The new l (lowercase L) directory contains shortened layer identifiers as symbolic links. These identifiers are used to avoid hitting the page size limitation on arguments to the mount command.

[root@hssl overlay2]# ls
083680cd4e72ec9f8e99678636ff627e755efaef2b7522b1e9dd3179ef01fbfb  backingFsBlockDev  l
[root@hssl overlay2]# ls l -la 
total 0
drwx------. 2 root root  40 May  5 08:42 .
drwx--x---. 4 root root 112 May  5 08:42 ..
lrwxrwxrwx. 1 root root  72 May  5 08:42 QZSIFLQ6QJ3CNT3QKUZC4J5LEL -> ../083680cd4e72ec9f8e99678636ff627e755efaef2b7522b1e9dd3179ef01fbfb/diff

The lowest layer contains a file called link, which contains the name of the shortened identifier, and a directory called diff which contains the layer's contents.

[root@hssl 083680cd4e72ec9f8e99678636ff627e755efaef2b7522b1e9dd3179ef01fbfb]# ls
diff  link
 
[root@hssl 083680cd4e72ec9f8e99678636ff627e755efaef2b7522b1e9dd3179ef01fbfb]# cat link 
QZSIFLQ6QJ3CNT3QKUZC4J5LEL
 
[root@hssl 083680cd4e72ec9f8e99678636ff627e755efaef2b7522b1e9dd3179ef01fbfb]# ls diff/
bin  boot  dev  etc  home  lib  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

The container layer

Pulling nginx image

[root@hssl ~]# docker pull nginx 
Using default tag: latest
latest: Pulling from library/nginx
943331d8a9a9: Pull complete 
67ef22056282: Pull complete 
844fa86a5e03: Pull complete 
4e82158dafdd: Pull complete 
e1242a59b7fa: Pull complete 
ff2745aabaf7: Pull complete 
a53cddf3d9ee: Pull complete 
Digest: sha256:c15da6c91de8d2f436196f3a768483ad32c258ed4e1beb3d367a27ed67253e66
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest
 
[root@hssl ~]# ls /var/lib/docker/overlay2/
0da2fc0095629184be052cbc80cb86d243af7db0e1e5c1e3e89699e72be442fd
52f39241e38ece0f937a8aa4216f5667558452db12df5b7b3343848a9c415496
7e871264a989d3029291709e86c96199809b75b9096ac8e70a299a59d6b68c27
80b70c70766169d1418944a3f49c1d41c5d29cd44842dd64e97fcfcc4dbe3986
b32c618928a8dde026a52d66ffbc75b9b089867374462dd4143b93e9f9a6ac23
backingFsBlockDev
dc534da2dceecc39e51c25caa938944e6b42fdc12441a8b13f2ff28d7a78bb58
fd393d1ae31848448b07effa776d05126bf5e36b7a72fb64c0809543a42d3dc2
l

Running the nginx container

[root@hssl overlay2]# docker run --name my-nginx -d -p 8080:80 nginx
6e4cbb90b6e4fcc307d2d72a030c4f99f30f1b5526f7e51ce6c0989284de8595
 
[root@hssl overlay2]# ls
0da2fc0095629184be052cbc80cb86d243af7db0e1e5c1e3e89699e72be442fd
52f39241e38ece0f937a8aa4216f5667558452db12df5b7b3343848a9c415496
7e871264a989d3029291709e86c96199809b75b9096ac8e70a299a59d6b68c27
80b70c70766169d1418944a3f49c1d41c5d29cd44842dd64e97fcfcc4dbe3986
824d4a24788688d18df6f803985102a3c6c683e04aa3383a3c7661e25cfe6806            # New layer added
824d4a24788688d18df6f803985102a3c6c683e04aa3383a3c7661e25cfe6806-init       # New layer added
b32c618928a8dde026a52d66ffbc75b9b089867374462dd4143b93e9f9a6ac23
backingFsBlockDev
dc534da2dceecc39e51c25caa938944e6b42fdc12441a8b13f2ff28d7a78bb58
fd393d1ae31848448b07effa776d05126bf5e36b7a72fb64c0809543a42d3dc2
l

The lower-id file contains the ID of the top layer of the image the container is based on, which is the OverlayFS lowerdir.
The upper directory contains the contents of the container's read-write layer, which corresponds to the OverlayFS upperdir.
The merged directory is the union mount of the lowerdir and upperdirs, which comprises the view of the filesystem from within the running container.
The work directory is internal to OverlayFS.
To view the mounts which exist when you use the overlay2 storage driver with Docker, use the mount command. The following output is truncated for readability.

[root@hssl overlay2]# cd 824d4a24788688d18df6f803985102a3c6c683e04aa3383a3c7661e25cfe6806
[root@hssl 824d4a24788688d18df6f803985102a3c6c683e04aa3383a3c7661e25cfe6806]# ls
diff  link  lower  merged  work

[root@hssl 824d4a24788688d18df6f803985102a3c6c683e04aa3383a3c7661e25cfe6806]# mount | grep overlay
overlay on /var/lib/docker/overlay2/824d4a24788688d18df6f803985102a3c6c683e04aa3383a3c7661e25cfe6806/merged type overlay (rw,relatime,seclabel,lowerdir=/var/lib/docker/overlay2/l/E6MUG2IDKI4UA72CKKUZ4Z24UV:/var/lib/docker/overlay2/l/OJKUKBTLGBJPHNUCZK2JLBQYMP:/var/lib/docker/overlay2/l/GKJQW7WMLGN2YFSKLUVYAIOHOE:/var/lib/docker/overlay2/l/HS6YETZ7QPJHKOWFOARXNWSKE5:/var/lib/docker/overlay2/l/A4J65LGJ374VD7C3VZRJLPMVHZ:/var/lib/docker/overlay2/l/XJCJB64MQEY4PES2UUQDDBPHCZ:/var/lib/docker/overlay2/l/4BBXTUQD3E34JDD6433GHWHPOH:/var/lib/docker/overlay2/l/TBUD63BG2NK4LYNT4ZEVM6S4MA,upperdir=/var/lib/docker/overlay2/824d4a24788688d18df6f803985102a3c6c683e04aa3383a3c7661e25cfe6806/diff,workdir=/var/lib/docker/overlay2/824d4a24788688d18df6f803985102a3c6c683e04aa3383a3c7661e25cfe6806/work)