On chroots breaking host systems

Background

Once in a while, I find myself exploring something new and dangeous. One day it might be changing the root filesystem, another day it may be replacing swap with something bigger, yet another it might be upgrade to EFI. All of those have met with various amounts of pain and sometimes were outright failures.

Yesterday, I was evaluating applications containers because they hold the promise of making services deployment easier. Every system upgrade brings me pain with incompatible configuration changes, so wouldn't it be nice if I could decouple system upgrades and app upgrades?

Usually, when I do these experiments, I set up a dedicated VM to prevent my main system from being contaminated or disabled. This time, I decided that chroot is simple enough to do it without protection. Famous last words?

I set up a minimal chroot jail. I mounted --rbind the necessary directories and bootstrapped it successfully.

The problems started when I wantedd to stop it. umount said the mount point was busy. I thought "no worries", decided umount -l will fix the problem, and proceeded as usual.

Disaster

After resuming my system from hibernation the next day... I could no longer start new terminals - Konsole windows showed up empty. While I could use su and sudo, new terminals couldn't be opened.

$ xterm
xterm: Error 32, errno 2: No such file or directory
Reason: get_pty: not enough ptys
$ screen
Must be connected to a terminal.
$ rxvt
rxvt: can't load font "-misc-fixed-medium-r-normal--14-130-75-75-c-70-iso8859-1"
rxvt: can't open pseudo-tty
rxvt: aborting

Luckily, I have another system with a similar setup, so I was able to compare to something working.

Via painstaking strace and gdb tracing, I figured out that:

1. /dev/pts became unmounted
2. /dev/pts MUST be mounted with -o gid=5 (5 stands for tty).

After mounting /dev/pts:

mount -t devpts devpts /dev/pts -o mode=620 -o gid=5

I can open terminal windows again.