path: root/posts/packaging-nebula-for-debian/main.md

Packaging Nebula for Debian
===========================

I am close to concluding a multi-week endevor to package [Nebula](https://github.com/slackhq/nebula), a VPN-style network mesh networking overlay. If all goes well, it will be uploaded to `debian/experimental` within the next few days. This would also mean the package would be pulled into Ubuntu during the next merge window.

### Timeline
 
Unfortunately, Debian does not adhere to a constant release cycle. This means the timeline is uncertain. It will likely be uploaded to `experimental` within a few days. [See the new queue.](https://ftp-master.debian.org/new.html) It will stay in experimental for the next three months or so until the next release occurs. (It is incompatible with the version of protobuf in unstable. This prevents it from moving into unstable until the next version release.)
```
{upload queue} -> [experimental] -> [unstable (sid)] -> [testing] -> [next release]
```

Preemptively, I'm going to write up a set of install instructions specific to debian derivatives and briefly a few of the decisions made during the packaging process.

Installation
------------

**Step one will currently fail. See [installing from experimental](#installing-from-experimental)**

For the sake of simplicity, I'm going to assume that you're setting up a network with two nodes -- one lighthouse node and a node on your laptop. Once you understand the process, it easily scales to as many nodes as you wish. Pick your favorite virtualization provider in order to set up the lighthouse. The lighthouse requires minimal resources because it functions as a mutually-reachable node which synchronizes the address mappings. You could use a home server provided that you have a static ip (unlikely) or setup dynamic DNS. The latter may introduce some instability. I'm also assuming both clients are debian derivatives and have access to `apt`.
 
If this is not the case, please consult the [upstream instructions](https://github.com/slackhq/nebula#user-content-getting-started-quickly) which will guide you through the processing of installing the binaries directly.
 
#### 1. Install Nebula through Aptitude

You'll need to install Nebula on both endpoints.

```bash
sudo apt install nebula
```

#### 2. Creating a certificate authority

The certificate authority is to "root of trust" for a Nebula network. Compromising the certificate authority's key file would compromise the integrity and security of the entire network. The upstream instructions recommend that you store the key file in a location with strong encryption [^1].

You can generate a `ca.key` and `ca.cert` file with the following command:
```bash
nebula-cert ca -name "Myorganization, Inc"
```

You will copy the `ca.crt` file to all the hosts. The `ca.key` file should remain secret.

#### 3. Nebula host keys and certificates generated from that certificate authority
 
With your `ca.key` file in hand, generate keys for each node.
 
```bash
nebula-cert sign -name "lighthouse" -ip "192.168.100.1/24"
nebula-cert sign -name "laptop" -ip "192.168.100.2/24"
```
 
Repeate this process for each node. It is important that each is issued a unique internal ip. The IPs are specified in CIDR notation [^2]. This internal ip will be used to configure Nebula later.
 
#### 4. Copy the configuration files to each host
 
Each host requires the `host.key`, `host.crt`, and `ca.crt` files to be present on the system. By convention, these are located in the `/etc/nebula` directory. Make sure to copy them into this directory.
 
For example, to copy the credentials to a lighthouse with ip `203.0.113.11` as `user` you may use sftp and ssh as follows:
 
```bash
sftp user@203.0.113.11 <<EOF
put lighthouse.key
put lighthouse.crt
put ca.crt
EOF
```
 
```bash
ssh user@203.0.113.11
sudo install -m 600 -o root lighthouse.{key,crt} /etc/nebula
sudo install -m 600 -o root ca.crt /etc/nebula
rm ca.crt lighthouse.{key,crt}
```
 
#### 5. Configure your network

The upstream recommends that you start from an example configuration file:

```
cp /usr/share/doc/nebula/examples/config.yml /etc/nebula/my_network.yml
```

- On your lighthouse, you'll want to change the `cert` and `key` sections to the paths `/etc/nebula/lighthouse.crt` and `/etc/nebula/ligthouse.key`. Change `am_lighthoue: true`. Remove the lighthouse ip from the `hosts` section under `lighthouse`.

- On the host, change the `cert` and `key` sections to the paths `/etc/nebula/laptop.crt` and `/etc/nebula/laptop.key`. Ensure the lighthouse is added to the `static_host_map` and the `hosts` section.

Once you're done, you can test whether your configuration is valid with `nebula-service -test -config /etc/nebula/my_network.yml`.

#### 6. Bringing up the tunnel

To start the tunnel, you can use the templated `systemd` service packaged alongside Nebula [^3][^4].
 
```bash
sudo systemctl start nebula@my_network
```

There is also a means by which a Nebula lighthouse can be run by a unprivileged user but further configuration is required [^5].

Once both ends of the tunnel have been started, you should be able to ping the lighthouse from the laptop node and vice versa.
```bash
ping 192.168.100.1
PING 192.168.100.1 (192.168.100.1) 56(84) bytes of data.
64 bytes from 192.168.100.1: icmp_seq=1 ttl=64 time=5.67 ms
```

#### 7. Additional Configuration

Nebula has built-in default deny firewall. The default configuration file allows network traffic `outbound`. (That is, any node is permitted to initiate a connection.) In order for a node to provide services, the port mapping needs to be added to the `inbound` section. For instance, to permit ssh to the lighthouse:

```yaml
inbound:
   - port: 22
     proto: tcp
     host: lighthouse
```

Now, an ssh connection should be able to be initiated via Nebula's internal ip:
```bash
ssh 192.168.100.1
```

Once you're happy with the setup, you can automatically start Nebula when the laptop / server boots:

```bash
sudo systemctl enable nebula@my_network
```

For more information on usage and configuration, you may want to take a look at `nebula.yml(5)`, `nebula(1)`, and `nebula-cert(1)`.

Enjoy.
```
  *  . . *    * .          . *    .      *   .   .   * .
. * .     .  *      .   *    .     *   .      *    .    *
.    .   *   .    *   .      *    .     *    .      *   .
. *   *   .   *   .   *   .    .   *   .     .  *    *   .
  .     .  *   .         *    .      *   .  *    *  .   .
 *   . *      .   *   .  *   . *    .    *   .   .    *  
```

### Installation Footnotes

[^1] If you're in need of a technology to provide strong encryption, [LUKS](https://guardianproject.info/archive/luks/) is a popular choice on Linux. [Veracrypt](https://www.veracrypt.fr) is a venerable cross-platform encryption application. Some password managers, like KeePassXC, also allow you to attach files.

[^2] Effectively, this means all nodes will receive an ip in the form "192.168.y.x". The y part is a value in the range [0, 255] and is specific to the network. (Thus, all nodes should have the same "y" value.) "x" should be a unique ip for each node and be in the range [0, 244].

[^3] The launcher I wrote will detect the `my_network.yml` and `my_network.yaml` files. Do not specify the extension when launching the service. The launcher has no way to discriminate between `my_network.yml` and `my_network.yaml` extension so pick a distinct name for each network.

[^4] If Nebula is misconfigured, the service will fail without warning. You can check the status of the unit with `systemctl status nebula@my_network`. Nebula can also be started within the terminal using `nebula -conifg /etc/nebula/my_network.yml`.

[^5] The systemd unit that is packaged with Nebula runs the interface as root. This is what I expect most users will want. If the lighthouse doesn't need to be connected to the network, you can `sudo systemd edit nebula@.serivce` and simply change the `User` section to the user you wish to use to launch Nebula. The user will also need read access to the configuration file, key, and cert files.

Packaging Notes
---------------

I am going to create a brief summary of the changes made while packaging. I suspect other distros might benefit from some of the work done to package Debian [^6].

The Debian package differs from the packaging done on [Arch](https://archlinux.org/packages/community/x86_64/nebula/). There's also a package created for [NixOS](https://github.com/NixOS/nixpkgs/blob/8284fc30c84ea47e63209d1a892aca1dfcd6bdf3/nixos/tests/nebula.nix) but Nix is its own beast.

### Templating the Unit File

If we were to use the unit file provided by the upstream project, it would fail without warning until the user fully setup the service because (1) the path of to Nebula configuration was hard coded as `/etc/nebula/config.yml` and (2) the user needed to change the configuration file in order for Nebula to function.

To make the relationship between the user configuration and the `systemd` unit clear, the `systemd` unit was templated. This also means that there is a clear and simple way to connect one machine to multiple Nebula networks. To accomplish this and support both `.yml` and `.yaml` extension, the systemd file executes a shell script under `/usr/lib/nebula/bin/nebula-systemd-launcher` passing the "instance variable" as the first argument. This script then identifies the proper configuration and launches Nebula with this configuration. The script was installed user `/usr/lib` so that it would not autocomplete in the user's shell.

### Doc and examples

- Man pages were generated from the nebula help flag to create `nebula(1)` and `nebula-cert(1)`.
- `nebula.yml(5)` man page was created to describe the configuration process. It was derived from the comments in the example configuration.
- The `config.yml` example configuration was installed under `/usr/share/doc/nebula/examples/` so users could copy it into `/etc/nebula` if they wished to use it as a starting point.

### Patching for Go 1.13

Debian packages all go dependencies to maintain tight control over the versions used while building go binaries. It also packages `go` itself. Currently, `go 1.16` is not in the debian repos [^7]. The following patch was applied since `net.ErrClosed` is not available in older versions of go.

```go
--- nebula.orig/sshd/server.go
+++ nebula/sshd/server.go
@@ -1,7 +1,7 @@
package sshd
import (
-   "errors"
+   "strings"
   "fmt"
   "net"
   "sync"
@@ -116,7 +116,8 @@ func (s *SSHServer) run() {
   for {
       c, err := s.listener.Accept()
       if err != nil {
-           if !errors.Is(err, net.ErrClosed) {
+           str := err.Error()
+           if !strings.Contains(str, "use of closed network connection"){
               s.l.WithError(err).Warn("Error in listener, shutting down")
           }
           return
```
 
### Dependencies
 
- I packaged `golang-github-nbrownus-go-metrics-prometheus` since changes were made to [go-metrics-prometheus](https://github.com/deathowl/go-metrics-prometheus) that were not backwards-compatible
- I also packaged `golang-github-flynn-noise` since it was not in the debian repositories
 
### Packaging footnotes
[^6] All files used to create the package are located in [Salsa (Debian's VCS)](https://salsa.debian.org/go-team/packages/nebula). All external configuration and build rules are located in the `debian` directory.
 
[^7] Actually, it is packaged individually but not under the `golang-go` moniker. I initially compiled it by preloading the `PATH` with `go 1.16` to force `dh-golang` to use those build tools. Thus caused `dh-golang` to misbehave and not harden or strip the binaries. Since the changes required to adapt Nebula to `go 1.13` were minimal, I opted to create a patch.

### Installing from experimental
 
This is a temporary aside. As mentioned above, the package is currently bouncing around Debian's packaging infrastructure. I'm assuming at the time of reading that it is in experimental. This is an internal Debian repository which allows maintainers, developers, or the curious to test the newest version of software before it enters the next Debian unstable.
 
If you are running `buster`, you cannot install it directly using `apt`. _If you would like to test the package while it is experimental,_ I will offer some instructions here. [All the usual disclaimers apply.](https://wiki.debian.org/DontBreakDebian#Don.27t_make_a_FrankenDebian) This is fairly safe since Nebula is a binary package (and doesn't have any runtime dependencies other than glibc).
 
There is a remote chance it will segfault due to binary incompatibilities with the version of glibc. If so, run `sudo apt purge nebula` and try installing from source. Building it from sources would require you to pull in a plethora of experimental build dependencies.
 
#### 1. Add `experimental` to your `sources.list` file
 
```bash
sudo sh -c "
sudo cat >/etc/apt/sources.list.d/99-tmp-nebula-overrides.list <<EOF
# Temporary pull in packages from the experimental distribution
 
deb https://deb.debian.org/debian experimental main
EOF
"
```
#### 2. Demote `experimental` in your `apt` preferences
 
```bash
sudo sh -c "
sudo cat >/etc/apt/preferences.d/99-tmp-nebula-prefer-stable <<EOF
Package: *
Pin: release o=Debian,a=experimental
Pin-Priority: -10
"
```
 
#### 3. Update
 
```bash
sudo apt update
```
 
If they above steps succeed, you should see:
```
All packages are up to date.
```
 
#### 3. Force APT to install the package from `experimental`
```bash
sudo apt install -t experimental nebula
```
 
After installing, you can continue to [creating a certificate authority](#2-creating-a-certificate-authority). Just ensure to remove nebula when you're finished testing.
 
#### 4. When you're done testing
```
sudo rm /etc/apt/sources.list.d/99-tmp-nebula-overrides.list \
       /etc/apt/preferences.d/99-tmp-nebula-prefer-stable
sudo apt purge nebula
```