Graham King

Solvitas perambulum

Online upgrades in Go

software go tcp unix
Summary
To achieve a zero-downtime upgrade for a server with long-running connections, you can either inherit file descriptors or send them over a UNIX domain socket. The latter method is more versatile as it can handle additional state and communication between processes. The key functions involved are `syscall.UnixRights` to send file descriptors and `syscall.ParseUnixRights` to receive them. An example demonstrates this process: the parent creates a socket pair for communication, passes necessary file descriptors and identification to the child, and the child signals when ready to take over. This allows the new process to seamlessly continue handling connections from where the old process left off.

tl;dr Send your socket fd over a UNIX domain socket: syscall/passfd_test.go.

When your server holds long running connections (WebSocket, long-running HTTP, IRC, XMPP, etc) you often want to be able to upgrade the server without dropping the connections (zero downtime upgrade). In UNIX there are at least two ways to do this:

  1. Inherit the file descriptor
  2. Send the file descriptor over a domain socket

The first one is straightforward, because a UNIX process automatically inherits the file descriptors of it’s parent, except if they have the close-on-exec flag set. Go complicates things a bit by always setting that flag on it’s sockets (in net/sock_cloexec.go). For a child process to inherit it’s parent’s file descriptors, you have to manually add them to ExtraFiles in os/exec/Cmd. There’s an example in TestExtraFiles in os/exec/exec_test.go.

Usually you need to send more that just the connections to the child process. There will be some state, and probably a communication where the child tells the parents it’s ready to take over (after priming it’s cache, for example). Hence the second approach, unix domain sockets, is more interesting.

Here’s a full working example, and how to test it:

  • Build it and run it. It should display it’s pid.
  • Telnet to localhost 1122 from two separate terminals. You should see lots of [v1] Message 1 in your terminals.
  • Now edit the file to change the VERSION number.
  • Rebuild it.
  • Signal it to upgrade with kill -USR2 <pid-displayed-earlier>.
  • The numbers in the telnet sessions should keep increasing, the connections continue, but the version number has changed.

Ta da - an online upgrade!

When reading the code below, skip straight to about line 112 (// We're the parent), as that’s where things start. We’ll go into detail after the code.

The two key calls, for passing the sockets to the child, are:

  1. syscall.UnixRights, which puts socket file descriptors into the ancillay data (also called socket control) part of a message.
  2. syscall.ParseUnixRights which gets socket file descriptors out of the ancillary data part of a message.

We’re not really passing a file descriptor at all, just a reference. The fd on the child process will most likely have a different number than in the parent, but they both point to the same entry in the kernel’s file descriptor table.

Around line 187, when the parent is killed (SIGTERM) it sends it’s connections to the child before exiting. The idea is that the child will start, do any initialization it needs, such as priming it’s cache, and then SIGTERM the parent to signal it’s willingness to take over connections. The SIGTERM is sent by the child line 70.

The domain socket that parent and child use to communicate is created line 148 (syscall.Socketpair). We need to get the read end of that pipe to the child. We use the ExtraFiles method, line 293, to pass it.

We also need the child to know that it’s the child. Line 294 we pass it an environment variables called PARENT_PID. We don’t ever use the value of that environment variable – a --child command line argument would have done just as well.

Finally, and not in the example, the child process may want to wait until the parent is confirmed dead. To check if a process is alive you can send it signal 0:

parent, _ := os.FindProcess(ppid)
err = parent.Signal(syscall.Signal(0))
for err == nil {
    time.Sleep(10 * time.Millisecond)
    // TODO: Don't wait forever
    err = parent.Signal(syscall.Signal(0))
}

If err is nil the signal succeeded, meaning the parent process is still running. See man 2 kill.

Happy restarting!

References