Hacking:Servers

From ParabolaWiki
Jump to: navigation, search
This article or section is out of date.
Please help improve the wiki by updating the article and correcting mistakes.
Note: This page documents how the Parabola project's servers are configured. This is not about how to configure servers.

1 DNS

These are the DNS entries with he.net, as of 2016-06-18.

Record Type Value
parabola.nu. NS ns1.he.net.
parabola.nu. NS ns2.he.net.
parabola.nu. NS ns3.he.net.
parabola.nu. NS ns4.he.net.
parabola.nu. NS ns5.he.net.
parabola.nu. A 80.87.131.232
parabola.nu. MX 10 repo.parabola.nu.
_xmpp-client._tcp.parabola.nu. SRV 0 0 5222 xmpp.parabolagnulinux.org.
_xmpp-server._tcp.parabola.nu. SRV 0 0 5269 xmpp.parabolagnulinux.org.
labs.parabola.nu. A 80.87.131.232
parabola.nu. TXT "v=spf1 ip4:80.87.131.232 -all"
parabola.nu. SPF "v=spf1 ip4:80.87.131.232 -all"
labs.parabola.nu. TXT "v=spf1 -all"
_xmpp-client._tcp.parabola.nu. TXT "v=spf1 -all"
_xmpp-server._tcp.parabola.nu. TXT "v=spf1 -all"
labs.parabola.nu. SPF "v=spf1 -all"
_xmpp-client._tcp.parabola.nu. SPF "v=spf1 -all"
_xmpp-server._tcp.parabola.nu. SPF "v=spf1 -all"
repo.parabola.nu. A 80.87.131.232
wiki.parabola.nu. A 80.87.131.232
www.parabola.nu. A 80.87.131.232
lists.parabola.nu. A 80.87.131.232
projects.parabola.nu. A 80.87.131.232
repo.parabola.nu. SPF "v=spf1 ip4:80.87.131.232 -all"
repo.parabola.nu. TXT "v=spf1 ip4:80.87.131.232 -all"
lists.parabola.nu. TXT "v=spf1 ip4:80.87.131.232 -all"
lists.parabola.nu. SPF "v=spf1 ip4:80.87.131.232 -all"
projects.parabola.nu. TXT "v=spf1 -all"
wiki.parabola.nu. TXT "v=spf1 -all"
www.parabola.nu. TXT "v=spf1 -all"
projects.parabola.nu. SPF "v=spf1 -all"
wiki.parabola.nu. SPF "v=spf1 -all"
www.parabola.nu. SPF "v=spf1 -all"
localhost.parabola.nu. A 80.87.131.232
localhost.parabola.nu. TXT "v=spf1 ip4:80.87.131.232 -all"
localhost.parabola.nu. SPF "v=spf1 ip4:80.87.131.232 -all"
pur.parabola.nu. A 80.87.131.232
pur.parabola.nu. TXT "v=spf1 -all"
pur.parabola.nu. SPF "v=spf1 -all"
redirector.parabola.nu. A 80.87.131.232
repomirror.parabola.nu. A 80.87.131.232
winston.parabola.nu. A 93.95.226.249
proton.parabola.nu. A 80.87.131.232
git.parabola.nu. CNAME winston.parabola.nu.
parabolagnulinux.org. NS ns1.he.net.
parabolagnulinux.org. NS ns2.he.net.
parabolagnulinux.org. NS ns3.he.net.
parabolagnulinux.org. NS ns4.he.net.
parabolagnulinux.org. NS ns5.he.net.
parabolagnulinux.org. A 80.87.131.232
projects.parabolagnulinux.org. A 80.87.131.232
wiki.parabolagnulinux.org. A 80.87.131.232
repo.parabolagnulinux.org. A 80.87.131.232
licenses.parabolagnulinux.org. A 80.87.131.232
lists.parabolagnulinux.org. A 80.87.131.232
forum.parabolagnulinux.org. A 80.87.131.232
parabolagnulinux.org. MX 0 repo.parabola.nu.
_xmpp-server._tcp.parabolagnulinux.org. SRV 0 0 5269 xmpp.parabolagnulinux.org.
_jabber._tcp.parabolagnulinux.org. SRV 0 0 5269 xmpp.parabolagnulinux.org.
_xmpp-client._tcp.parabolagnulinux.org. SRV 0 0 5222 xmpp.parabolagnulinux.org.
list.parabolagnulinux.org. MX 0 repo.parabola.nu.
lists.parabolagnulinux.org. MX 0 repo.parabola.nu.
conference.parabolagnulinux.org. A 80.87.131.232
list.parabolagnulinux.org. A 80.87.131.232
xmpp.parabolagnulinux.org. A 80.87.131.232
parabolagnulinux.org. TXT "v=spf1 ip4:80.87.131.232 -all"
lists.parabolagnulinux.org. TXT "v=spf1 ip4:80.87.131.232 -all"
repo.parabolagnulinux.org. TXT "v=spf1 ip4:80.87.131.232 -all"
bugs.parabolagnulinux.org. TXT "v=spf1 -all"
conference.parabolagnulinux.org. TXT "v=spf1 -all"
forum.parabolagnulinux.org. TXT "v=spf1 -all"
licenses.parabolagnulinux.org. TXT "v=spf1 -all"
list.parabolagnulinux.org. TXT "v=spf1 -all"
packages.parabolagnulinux.org. TXT "v=spf1 -all"
projects.parabolagnulinux.org. TXT "v=spf1 -all"
wiki.parabolagnulinux.org. TXT "v=spf1 -all"
xmpp.parabolagnulinux.org. TXT "v=spf1 -all"
_xmpp-client._tcp.parabolagnulinux.org. TXT "v=spf1 -all"
_xmpp-server._tcp.parabolagnulinux.org. TXT "v=spf1 -all"
_jabber._tcp.parabolagnulinux.org. TXT "v=spf1 -all"
bugs.parabolagnulinux.org. SPF "v=spf1 -all"
conference.parabolagnulinux.org. SPF "v=spf1 -all"
forum.parabolagnulinux.org. SPF "v=spf1 -all"
licenses.parabolagnulinux.org. SPF "v=spf1 -all"
list.parabolagnulinux.org. SPF "v=spf1 -all"
lists.parabolagnulinux.org. SPF "v=spf1 ip4:80.87.131.232 -all"
packages.parabolagnulinux.org. SPF "v=spf1 -all"
parabolagnulinux.org. SPF "v=spf1 ip4:80.87.131.232 -all"
projects.parabolagnulinux.org. SPF "v=spf1 -all"
repo.parabolagnulinux.org. SPF "v=spf1 ip4:80.87.131.232 -all"
wiki.parabolagnulinux.org. SPF "v=spf1 -all"
xmpp.parabolagnulinux.org. SPF "v=spf1 -all"
_xmpp-client._tcp.parabolagnulinux.org. SPF "v=spf1 -all"
_xmpp-server._tcp.parabolagnulinux.org. SPF "v=spf1 -all"
_jabber._tcp.parabolagnulinux.org. SPF "v=spf1 -all"
www.parabolagnulinux.org. A 80.87.131.232
labs.parabolagnulinux.org. A 80.87.131.232
pur.parabolagnulinux.org. A 80.87.131.232
pur.parabolagnulinux.org. TXT "v=spf1 -all"
pur.parabolagnulinux.org. SPF "v=spf1 -all"
www.parabolagnulinux.org. TXT "v=spf1 -all"
labs.parabolagnulinux.org. TXT "v=spf1 -all"
labs.parabolagnulinux.org. SPF "v=spf1 -all"
www.parabolagnulinux.org. SPF "v=spf1 -all"
repomirror.parabolagnulinux.org. A 80.87.131.232

1.1 SPF

These are mail domains, i.e. ones listed in envelope From addresses or used for outgoing mail servers:

  • parabola.nu
  • lists.parabola.nu
  • repo.parabola.nu

Mail domains have TXT and SPF records with the value "v=spf1 ip4:80.87.131.232 -all", i.e. allowing only mail sent from the parabola.nu server. All other domains have "v=spf1 -all": all mail from them is fake.

We use SPF to get less spam claiming to be from our domains and to discourage spammers from using fake sender addresses.

2 proton.parabola.nu

This article or section is out of date.
Please help improve the wiki by updating the article and correcting mistakes.

proton.parabola.nu was a VPS hosted somewhere in the UK, on a machine operated by user n1md4 in cooperation with his employer, Positive Internet.

It is no longer used by Parabola.

2.1 Resources

$ uname -m
x86_64
$ free -h
             total       used       free     shared    buffers     cached
Mem:          1.9G       1.8G       100M       191M        28M       538M
-/+ buffers/cache:       1.2G       667M
Swap:         1.0G       413M       608M
$ df -h | grep sda
/dev/sda1        98G   12G   82G  13% /
/dev/sda3       150G  125G   26G  83% /srv

2.2 Reboot

Once upon a time, reboots were hard to do because of a weird issue that could make shutdown take half an hour! That's a lot of downtime and praying!

Nowadays, it takes about 3 and a half minutes (measured at the Nov 8 1:00 GTM reboot):

shut down 00:01:36
BIOS 00:00:30
boot up 00:01:21
total 00:03:27

Still slower than we would like, but so are most things on Proton.

2.3 Public-facing network sockets/services

Note: IPv4/6 support is just what lsof says. Except that systemd .socket targets are both IPv4 and IPv6, but lsof reports them as just IPv6, even for established IPv4 connections. Weird. I'm pretty sure it's a bug in lsof, so I guess that means that some of the others might be wrong?
socket IPv4 IPv6 protocol daemon other info
TCP:*:22 Yes Yes SSH sshd.service
TCP:*:25 Yes Yes SMTP postfix.service/master
TCP:*:80 Yes Yes HTTP nginx.service
TCP:*:443 Yes Yes HTTPS nginx.service
TCP:*:465 Yes Yes SMTPS postfix.service/master
TCP:*:587 Yes Yes SMTP-MSA postfix.service/master
TCP:*:655 Yes Yes tinc tincd@lvpn.service
TCP:*:875 Yes* Yes rsync rsync.socket
TCP:*:1863 Yes Yes SSH sshd.service
TCP:*:5222 Yes Yes xmpp-client prosody.service
TCP:*:5269 Yes Yes xmpp-server prosody.service
TCP:*:9418 Yes* Yes git git-daemon.socket

2.4 Inward-facing sockets

socket protocol unit
TCP4/6:localhost.localdomain:2812 HTTP monit.service
TCP4:localhost.localdomain:5432 pgsql postgresql.service
TCP4/6:localhost.localdomain:5582 telnet prosody.service
unix:/run/mailman-fcgi.sock FastCGI mailman-fcgi.socket
unix:/run/uwsgi/labs.sock uwsgi uwsgi@labs.socket
unix:/run/uwsgi/parabolaweb.sock uwsgi uwsgi@parabolaweb.socket
unix:/run/uwsgi/projects.sock uwsgi/modifier1=9 uwsgi@projects.socket
unix:/run/uwsgi/repo.sock uwsgi/modifier1=14 uwsgi@repo.socket
unix:/run/dovecot/... misc dovecot.service
unix:/var/spool/postfix/private/auth Dovecot SASL dovecot.service
unix:/var/spool/postfix/private/* misc postfix.service
System stuff
unix:/run/dbus/systemd_bus_socket D-Bus dbus.socket
unix:/run/lvm/lvmetad.socket  ??? lvm2-lvmetad.socket
unix:/run/udev/control  ??? systemd-udev-control.socket
unix:/run/systemd/... misc misc
unix:/run/user/${UID}/{bus,systemd/{notify,private}} misc user@${UID}.service

2.5 Other running services of note

  • dovecot.service
  • mailman.service
  • parabolaweb-reporead-inotify.service

2.6 Nginx "servers"

server_name HTTP HTTPS
Simple redirects
* return 301 https://$host$request_uri; return 301 https://www.parabola.nu/404;
parabolagnulinux.org N/A return 301 https://www.parabola.nu$request_uri;
*.parabolagnulinux.org N/A return 301 https://$subdomain.parabola.nu$request_uri;
list.parabolagnulinux.org N/A return 301 https://lists.parabola.nu$request_uri;
parabola.nu N/A return 301 https://www.parabola.nu$request_uri
Websites
www.parabola.nu N/A Serve /static/, /favicon.ico, /robots.txt,and /img/ statically, redirect /https to /, and hand everything else off to uWSGI
labs.parabola.nu N/A Redirect / to /projects; use uWSGI
lists.parabola.nu N/A
  • Redirect / to /mailman/
  • Redirect /mailman/ to /mailman/listinfo
  • Serve the mailman CGI programs at /mailman/* via fcgiwrap
  • Serve the static mailman icons at /icons
  • Serve /var/lib/mailman/archives/public at /pipermail
projects.parabola.nu N/A Serve cgit via uWSGI
repo.parabola.nu N/A Serve the union of /srv/repo/main and /srv/repo/http, using repoindex.php (via uWSGI) for indexes.
wiki.parabola.nu N/A Serve MediaWiki via uWSGI (https://lukeshu.com/blog/nginx-mediawiki.html)
pur.parabola.nu N/A TODO
redirector.parabola.nu N/A TODO
repomirror.parabola.nu N/A TODO

2.7 Mail configuration

Note: I (lukeshu) have no idea if this is still true for the current server. Ask fauno or mtjm.

Postfix's postscreen handles port 25 enforcing the pregreet test, checks SPF records via python2-postfix-policyd-spf, the mail is delivered via Dovecot deliver. Both deliver and the daemon need to be of the same version, so restarting the daemon after an update is needed.

Mailman handles the lists, with Postfix integration via a virtual address map. Use /usr/lib/mailman/bin/newlist to add a list.

2.8 Configuration things

There used to be several symlinks added to /var to keep things in sane places, but they have been replaced by bind mounts in /etc/fstab, because the symlinks confused pacman.

/var/lib/mailman   ->  ../../srv/mailman
/var/lib/mysql     ->  ../../srv/sql/mysql
/var/lib/postgres  ->  ../../srv/sql/postgres
/var/spool/cron    ->  ../../etc/cron.spool

That is, nothing of consequence and needing to be backed up should live anywhere but /srv or /etc. Maybe /home for personal stuff.

3 winston.parabola.nu

This article or section is out of date.
Please help improve the wiki by updating the article and correcting mistakes.

winston.parabola.nu is A VPS server which is hosted for free by 1984, a Hosting Company in Iceland.

$ uname -m
x86_64

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           4.4G         98M        2.2G        692K        2.0G        4.0G
Swap:          4.3G          0B        4.3G

$ df -h | grep ^/dev
/dev/vda3        94G  3.2G   86G   4% /
/dev/vda4       375G  5.7M  355G   1% /srv


Pardon the sloppy formatting (<code> instead of {{ic|), I wrote this in markdown and used pandoc to convert it to mediawiki.

3.1 configuration philosophy

Avoid editing files owned by a package if possible; insert things into new files. If a file must be edited, try to avoid having to change any lines; try to only add new lines.

Everything worth backing up should be in /etc, /srv, or /home.

3.2 infrastructure/management

3.2.1 base setup

Packages installed:

  • group:base except for linux-libre
  • linux-libre-lts
  • haveged
  • irqbalance

Files affected:

  • /etc/hostname : winston.parabola.nu
  • /etc/locale.conf : LANG=en_US.UTF-8 LC_COLLATE=C
  • /etc/locale.gen : en_US.UTF-8 UTF-8
  • /etc/machine-info : LOCATION=1984 Hosting Company, Iceland
  • /etc/vconsole.conf : KEYMAP=us CONSOLEMAP=8859-1 FONT_MAP=8859-1_to_uni
  • /etc/systemd/system/multi-user.target.wants/getty@tty1.service
  • /etc/systemd/system/multi-user.target.wants/haveged.service
  • /etc/systemd/system/multi-user.target.wants/irqbalance.service
3.2.1.1 bootloader

Packages installed:

  • grub

Files affected:

  • /etc/default/grub

The grub config has two overrides added to it:

GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_TERMINAL_OUTPUT=console
3.2.1.2 filesystems

Packages installed:

  • systemd-swap

Files affected:

  • /etc/fstab
  • /etc/systemd-swap.conf
  • /etc/systemd/system/local-fs.target.wants/systemd-swap.service
  • /etc/systemd/system/multi-user.target.wants/remote-fs.target

TODO: document

fstab is set up similarly to proton.parabola.nu.

3.2.1.3 networking

Files affected:

  • /etc/udev/rules.d/80-net-setup-link.rules
  • /etc/netctl/eth0-static
  • /etc/systemd/system/netctl@eth0\x2dstatic.service (auto-generated by netctl enable)
  • /etc/systemd/system/multi-user.target.wants/netctl@eth0\x2dstatic.service

/etc/udev/rules.d/80-net-setup-link.rules is a symlink to /dev/null, which disables new-style predictable network interface names (enp0s3? ens3?), and causes it to fall back to the old-style names (eth0).

The netctl profile eth0-static just has the network information from the 1984 VPS control panel.

3.2.1.4 Pacman

Files affected:

  • /etc/pacman.conf : Enable [pcr]
  • /etc/pacman.d/mirrorlist : Use repo.parabola.nu directly
  • /etc/pacman.d/.gitignore : /gnupg/
3.2.1.5 timedate

Files affected:

  • /etc/localtime : -> /usr/share/zoneinfo/UTC
  • /etc/systemd/system/sysinit.target.wants/systemd-timesyncd.service

timesyncd is an NTP client daemon.

3.2.2 etckeeper

Packages installed:

  • pristine-etc-keeper

Files affected:

  • /etc/systemd/system/multi-user.target.wants/pristine-etc-keeper.timer
  • /etc/systemd/system/multi-user.target.wants/etckeeper.timer
  • /etc/.gitignore : append !/mtab /machine-id /group- /gshadow- /passwd- /shadow- /resolv.conf
  • /etc/udev/.gitignore : /hwdb.bin

etckeeper is configured to use git to keep track of changes in /etc. The systemd unit etckeeper.timer is enabled, which makes a commit (if necessary) daily. It will also run before and after pacman via libalpm hooks.

In addition to etckeeper keeping track of the current configuration, lukeshu's pristine-etc-keeper maintains a branch of what /etc would be like if we never made any changes from the default files. The clean version of etc is available as the clean remote's master branch. Doing a git diff clean/master master should be helpful in investigating how things have been configured.

3.2.3 users

See also: keys user and group

3.2.3.1 Parabola hackers

Packages installed:

  • parabola-hackers-nshd
  • openssh

Files affected:

  • /var/lib/hackers-git/
  • /etc/ssh/sshd_config
  • /etc/nsswitch.conf
  • /etc/pam.d/passwd
  • /etc/pam.d/system-auth
  • /etc/pam.d/system-login
  • /etc/systemd/system/sockets.target.wants/nshd.socket
  • /etc/systemd/system/dbus.service.wants/nshd.service (temporary systemd bug workaround)
  • /etc/nshd/shadow

sshd is configured to force the use of keys (no password-based login), and to use parabola-hackers ssh-list-authorized-keys in addition to checking ~/.ssh/authorized_keys. ssh-list-authorized-keys returns the authorized keys from the hackers.git checkout in /var/lib/hackers-git (the path to the checkout is configured in /etc/parabola-hackers.yml).

NSS and PAM have been configured to use the ldap modules that are part of nss-pam-ldapd. However, instead of running the normal nslcd LDAP client daemon, the system has ben configured to run the parabola-hackers-nshd nshd daemon, which reads user infomation from the same hackers.git checkout (configured the same way). This way we dn't have to worry about keeping /etc/passwd in sync with hackers.git. To this end, PAM has also been configured to create a users home directory when they log in if it doesn't already exist. Because hackers.git doesn't store any password information, nshd stores password hashes in /etc/nshd/shadow.

Sometimes after something fails in PAM, you get a "User not known to the underlying authentication module" message. Like, the pam_ldap.so failed because you typed your password wrong, but it acts like it failed because it didn't "own" the user. I think that it's just a bug in PAM's message selection. But (TODO) we should actually track it down.

3.2.3.2 emergency user

Files affected:

  • /etc/passwd
  • /etc/shadow
  • /etc/sudoers.d/99-emergency
  • /home/emergency/

In case anything should ever go wrong with hackers.git, the user "emergency" has been set up. "emergency" has authorized (statically via ~/.ssh/authorized_keys) the keys of serveral (not publicly disclosed) Parabola hackers. /etc/sudoers.d/99-emergency grants special privileges in case PAM or NSS get screwed up.

3.2.3.3 other

Files affected:

  • /etc/sudoers.d/00-wheel
  • /etc/systemd/system/shadow.service.d/sort.conf

/etc/sudoers.d/00-wheel gives sudo access to everyone in the wheel group.

The shadow.service has been extended to sort the files if they otherwise are ok. This makes dealing with pacman updates and such easier.

3.2.3.4 pbot

Files affected:

  • /etc/group
  • /etc/gshadow
  • /etc/passwd
  • /etc/shadow
  • /etc/systemd/system/pbot.service <-- this script is coppied from the pbot git repo
  • /etc/systemd/system/multi-user.target.wants/pbot.service <-- created by systemctl enable

3.2.4 SSH

Packages installed:

  • openssh

Files affected:

  • /etc/ssh/sshd_config
  • /etc/systemd/system/multi-user.target.wants/sshd.service
  • /etc/ssh/.gitignore: /ssh_host_*_key /ssh_host_*_key.pub

See above for how authentication and users are set up.

sshd is also configured to listen on both ports 22 and 1863. We may turn of port 22 in the future. Not using port 22 isn't security through obscurity, it is security through keeping-the-the-logs-useful-by-keeping-noise-down.

3.2.5 SSL

Packages installed:

  • certbot

Winston uses the certbot ACME client to get certificates from Let's Encrypt.

All domains handled by the server are shoved in as Subject Alternative Names in a single certificate. This makes configuring nginx easier.

3.2.5.1 keys user and group

Files affected:

  • /etc/passwd
  • /etc/shadow
  • /etc/group
  • /etc/gshadow
  • /etc/letsencrypt
  • /var/lib/letsencrypt
  • /var/log/letsencrypt

In order to run certbot as a non-root user, the keys user and group have been created:

useradd --system --user-group --no-create-home --home-dir /etc/ssl --shell /usr/bin/nologin keys
chown -R keys:keys /etc/letsencrypt /var/log/letsencrypt /var/lib/letsencrypt
chmod 750 /etc/letsencrypt/archive /etc/letsencrypt/live

The associated keys group allows users to read the (private) keys in /etc/letsencrypt/live.

3.2.5.2 issuance, renewal, and installation

Files affected:

  • /etc/ssl/misc/certbot-get
  • /etc/ssl/misc/certbot-hook
  • /etc/sudoers.d/10-certbot
  • /etc/systemd/system/certbot-renew.service
  • /etc/systemd/system/certbot-renew.timer
  • /etc/systemd/system/timers.target.wants/certbot-renew.timer
  • /etc/nginx/snippets/ssl.conf

Unlike acmetool, certbot doesn't have an easy way of saying "please add this domain as a Subject Alternative Name". You have to re-run the same (long) command to get the cert, but with the domain added. So, I've encapsulated this into the script /etc/ssl/misc/certbot-get. Edit the array of domains at the top of the script, then run it.

Renewal, however, is very simple. It is handled by certbot-renew.service (triggered by the associated .timer). It runs certbot renew with a couple of flags.

Both certbot-get and certbot-renew.service prove ownership of the domain via the http-01 challenge. /etc/nginx/nginx.conf includes /etc/nginx/snippets/ssl.conf, which has a server{} block that handles ACME http-01 challenges.

Both certbot-get and certbot-renew.service have been written to run sudo /etc/ssl/misc/certbot-hook after certificates have been updated, and sudo has been configured to allow the keys user to do this without a password. Right now certbot-hook just runs systemctl reload nginx.service.

3.2.5.3 other

Files affected:

  • /etc/nginx/nginx.conf
  • /etc/nginx/snippets/ssl.conf
  • /etc/ssl/private/dhparam-2048.pem
  • /etc/ca-certificates/.gitignore : /trust-source/blacklist/ /trust-source/anchors/ /extracted/
  • /etc/ssl/.gitignore : /certs/*.0 /certs/*.1 /certs/*.pem /certs/ca-certificates.crt /certs/java/cacerts /.ssh/

nginx.conf includes snippets/ssl.conf, which is primarily based on the output of Mozilla Security's recommended web server configuration generator. It has had the main SSL information promoted to be directly into the http{} block, instead of having to be in each server{} block. The HTTP->HTTPS redirector has had an exception added to it to have it respond to ACME http-01 challenges.

Because certbot is only configured to use http-01 challenges, the all challenges happen over pain HTTP, which means that the configurations for each subdomain (which only serve over HTTPS/HTTP2) do not need to include anything about ACME or SSL (other than mentioning ssl in the listen directive).

ssl.conf needs to refer to a dhparam PEM file. This has been generated with the command

openssl dhparam -out /etc/ssl/private/dhparam-2048.pem 2048

3.2.6 HTTP

Packages installed:

  • nginx

Files affected:

  • /etc/nginx/nginx.conf
  • /etc/nginx/fastcgi.conf, /etc/nginx/{fastcgi,scgi,uwsgi}_params
  • /etc/nginx/mime.types
  • /etc/nginx/snippets/ssl.conf
  • /etc/nginx/sites/
  • /etc/ssl/misc/certbot-hook
  • /etc/systemd/system/multi-user.target.wants/nginx.service

See the #SSL for how nginx has been configured to do HTTPS.

fastcgi.conf, fastcgi_params, scgi_params and uwsgi_params have been edited to pass 127.0.0.1 as the client IP address to worker processes, to protect user privacy.

The log_format has been customized to redact the client IP and referrer, again to protect user privacy. Unfortunately, the libre version of nginx does not allow for customizing the format of the error log, so user IPs may still end up in the error log.

mime.types has had xz, gzip, bzip2, tar, PGP (.sig), and bittorrent types added to it. This list was based on the unmatched file extensions that find turned up on repo.parabola.nu.

Per-subdomain configurations live in /etc/nginx/sites/*.conf (which is the exact wildcard used with include at the end of nginx.conf).

"Special" rules in sites/*.conf should have the alias- or meta- prefix. For example, right now there is only meta-unknown-domain.conf which is a fallback for when no other file handles a domain (it redirects to https://www.parabola.nu/404). Normal domain handlers should be named server-${domain}.conf

Every sites/server-*.conf file should be of the format (replace subdomain with the subdomain):

/etc/nginx/sites/server-subdomain.parabola.nu.conf
# -*- Mode: nginx; nginx-indent-level: 8; intent-tabs-mode: t -*-

server {
    server_name subdomain.parabola.nu;
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    error_log  /var/log/nginx/nginx.http.subdomain.parabola.nu.error.log  error;
    access_log /var/log/nginx/nginx.http.subdomain.parabola.nu.access.log specific;

    ...
}

nginx.conf sets root /srv/http/$server_name, which means that most of the time root won't need to be set in each individual server{} block. Unfortunately, error_log and access_log don't support variable expansion, so they do need to be set each time.

The individual sites/server-*.conf files won't be documented in this section; see the section for the service that it provides.

TODO: document the rest of the settings in nginx.conf

3.2.6.1 process management

Packages installed:

  • uwsgi

Files affected:

  • /etc/systemd/system/uwsgi@.service.d/socket.conf
  • /etc/systemd/system/uwsgi@.socket.d/owner.conf
  • /etc/uwsgi/vassal.ini
  • /etc/systemd/system/sockets.target.wants/uwsgi@vassal.socket

Wherever possible, we should use uWSGI for process management of our HTTP services. This allows for much more consistent management and configuration than the hodge-podge of PHP-FPM, manage.py, fcgiwrap, et c. that we used to have on Proton.

uWSGI is the program, uwsgi is the protocol it speaks with nginx. A pool of workers is called a vassal, and is configured in /etc/uwsgi/vassal.ini, and activated by uwsgi@vassal.socket; a socket speaking the uwsgi protocol is created at /var/run/uwsgi/vassal.sock.

We use systemd socket activation rather than a uWSGI emperor because they provide largely the same functionality; the only real advantage that a uWSGI emperor would provide over systemd socket activation is if you ran it in tyrant mode, it lets you do some cool things with unpriveleged access, which would be useful for a shared web host. We aren't a shared web host, and have no reason to run emperor in tyrant mode.

Since the uwsgi@.service vassal unit is written to support socket-activated or non-socket-activated use, it is normally possible to accidentally start it without the associated .socket unit; which is an error with how our vassal configurations are written. To fix this, uwsgi@.service.d/socket.conf overrides the unit a bit to disable non-socket-activated use.

The ownership and permissions for the socket are configured in uwsgi@.socket.d/owner.conf, which sets the owner to http:http and the mode to 0600.

uWSGI supports thread pools in addition to process pools, but many of the actual workers you'll want to use aren't thread safe, so stick to process pools unless you specifically know that your worker is thread-safe (for example, PHP, at least with the modules needed for MediaWiki, is not thread-safe).

Individual vassal configurations are documented in the section for the service that they provide, not here.

3.2.7 email

Packages installed:

  • postfix

Files affected:

  • /etc/systemd/system/multi-user.target.wants/postfix.service
  • /etc/postfix/.gitignore : *.db

Configuration

  • /etc/postfix/main.cf edited to set make it accept, and locally deliver, mails to <user>@winston.parabola.nu.

Operation

  • pbot gets emails delivered qmail-style in $HOME/Maildir/ to allow him to report bug tracker changes
  • currently postfix aliases has not been set up. where should mails addressed to e.g. postmaster@winston.parabola.nu go? at the moment they get locally delivered to root user

3.3 services

You know, the things that the server is actually for.

3.3.1 System Information

Packages installed:

  • uwsgi-plugin-cgi

Files affected:

  • /srv/http/winston.parabola.nu/sysinfo.cgi
  • /etc/uwsgi/winston.ini
  • /etc/nginx/sites/server-winston.parabola.nu.conf
  • /etc/systemd/system/sockets.target.wants/uwsgi@winston.socket

Serves a simple system information page at https://winston.parabola.nu/. The source of the (very simple) script is at https://winston.parabola.nu/sysinfo.cgi.

3.3.2 ParabolaWeb

Packages installed:

  • uwsgi-plugin-python2
  • parabolaweb-utils
  • parabolaweb-dev

Files affected:

  • /etc/passwd
  • /etc/shadow
  • /srv/http/www.parabola.nu/

Winston isn't running ParabolaWeb (yet), but has it installed because other pages symlink to the branding files in it.

useradd --comment ParabolaWeb --home-dir /srv/http/www.parabola.nu/ --create-home --system --gid http parabolaweb

3.3.3 Bazaar

Packages installed:

  • bzr

Files affected:

  • /etc/passwd
  • /etc/systemd/system/bzr.service
  • /etc/systemd/system/multi-user.target.wants/bzr.service
  • /srv/bzr/

TODO: IDK, someone else set this up.

3.3.4 Git

Packages installed:

  • git
  • cgit
  • uwsgi-plugin-cgi

Files affected:

  • /etc/passwd
  • /etc/sudoers.d/10-git
  • /etc/cron.spool/git
  • /srv/git/

The "git" user has been modified:

usermod --comment "Parabola git" --shell /usr/bin/git-shell git

The git repositories live in /srv/git/ and are owned by the git user.

Metadata about all of the git repositories, as well as most of the setup, lives in the git-meta.git repository. The git user has a cron-job to update this repository every hour.

git is configured (core.hooksPath such that it looks in ~/.config/git/hooks/ for hooks, instead of /path/to/repo.git/hooks/. All of the server-side hooks exist: they all do the "same" thing: use run-parts to run all of the scripts in ~/.config/git/hooks/hook-name.d/, and then run the normal hook file /path/to/repo.git/hooks/hook-name.

Right now there are only two "real" hooks:

  • /srv/git/.config/git/hooks/post-receive.d/agefile : see #transport:_HTTPS
  • /srv/git/hackers.git/hooks/post-receive : update the hackers.git checkout (see #Parabola_hackers), then run sudo systemctl reload nshd.service to activate changes to system users (sudo has been configured to allow this to be done without a password). TODO: see the hook file itself for more details, but it should also activate the changes to email aliases with postfix, as well as rebuild the parabola-keyring package with autobuilder.
3.3.4.1 transport: git protocol

Files affected:

  • /etc/systemd/system/socket.target.wants/git-daemon.socket
3.3.4.2 transport: SSH

Files affected:

  • /srv/git/git-shell-commands/

This is the only method for which push is enabled. Only the git@ user may push; even though if you have access to git, you probably have access to ssh into yourname@winston, you can't push from that account. As far as git is concerned, all users are equal (TODO: perhaps we should implement some more granular access control. Safety rails and all that). Access to git@ is based on membership in the "git" group in hackers.git (see #Parabola_hackers).

If you need to set up custom git hooks, then ssh in to your user, and use sudo -u git to do it.

3.3.4.3 transport: HTTPS

Files affected:

  • /etc/uwsgi/git.ini
  • /etc/systemd/system/socket.target.wants/uwsgi@git.socket
  • /etc/nginx/sites/server-git.parabola.nu.conf
  • /srv/http/git.parabola.nu/
  • /srv/git/.config/git/hooks/post-receive.d/agefile

The git web interface is cgit, which is managed by uWSGI speaking uwsgi/modifier1=9, which is the variant of the uwsgi protocol for CGI requests.

uwsgi/git.ini sets CGIT_CONFIG=/srv/http/git.parabola.nu/cgitrc.

A post-receive hook that runs for all repositories updates the agefile, so that cgit has a better idea of when a repository was last updated.

The files for cgit are in /srv/http/git.parabola.nu/, which shadows /usr/share/webapps/cgit/:

# mirror /usr/share/webapps/cgit/
cgit.cgi    -> /usr/share/webapps/cgit/cgit.cgi
cgit.css    -> /usr/share/webapps/cgit/cgit.css # customized
cgit.png    -> /usr/share/webapps/cgit/cgit.png
favicon.ico -> /usr/share/webapps/cgit/favicon.ico /srv/http/www.parabola.nu/web/sitestatic/favicon.ico
robots.txt  -> /usr/share/webapps/cgit/robots.txt
# added files
cgitrc
archnavbar/ -> /srv/http/www.parabola.nu/web/sitestatic/archnavbar/
header.html              # archnavbar
syntax-highlighting.sh   # based on some old version of /usr/lib/cgit/filters/syntax-highlighting.sh

3.3.5 pbot

pbot, the Parabola IRC bot, lives at pbot.git, and runs as the user pbot

Home dir at /home/pbot/.

  • This contains all the pbot files from the git repo. Not in a subdir, just right there.
  • Also contains dir logs/ TODO add to git repo
  • Also contains 600 permissions file passord.secret containing IRC password
  • pbot writes data to info/ and announcements/. these are put in a tarball every day by cron job so they can be downloaded elsewhere if wanted (e.g. for backup).

Deps installed with pacman

  • moreutils
  • w3m
  • recode
  • inotify-tools

3.3.6 Redmine

Packages installed:

  • ruby
  • ruby-bundler
  • rubygems
  • uwsgi-plugin-rack

Files affected:

  • /etc/nginx/sites/server-labs_parabola_nu.conf
  • /etc/systemd/system/labs-restart.socket
  • /etc/systemd/system/labs-restart.timer
  • /etc/systemd/system/multi-user.target.wants/uwsgi@labs.service
  • /etc/systemd/system/socket-user.target.wants/uwsgi@labs.socket
  • /etc/systemd/system/timers.target.wants/labs-restart.timer
  • /etc/systemd/system/uwsgi@labs.service.d/psql.conf
  • /etc/uwsgi/labs.ini

See also: DeveloperWiki:Web_Admin

3.3.7 Rsync

Packages installed:

  • rsync

Files affected:

  • /etc/rsyncd.conf
  • /etc/systemd/system/rsyncd.socket
  • /etc/systemd/system/sockets.target.wants/rsyncd.socket

Installed by coadde or Emulatorman during migration of repo.parabola.nu from Proton. I resumed configuration of the server and enabled it to accommodate mirrors after the DNS change. Port number (873 -> 875) and modules ( [repos]:/srv/repo/main and [abs]:/srv/abslibre) were copied verbatim from Proton.

4 beefcake.parabola.nu

Beefcake is a virtual machine occupying roughly half of LukeShu's KGPE-D16 in Indianapolis. It runs no publicly accessible services, but is intended to be usable as a build machine for Parabola developers. It has a large data disk which archives all repos since April 2018; which are exposed to the librechroots as an HTTP service.

$ uname -m
x86_64

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           29Gi       725Mi       250Mi        13Mi        28Gi        28Gi
Swap:         7.4Gi       4.0Mi       7.4Gi

$ df -h | sed -n '1p;/^\/dev/p'
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda         10G  5.6G  4.1G  59% /
/dev/vdb        9.8G  6.8G  2.6G  73% /home
/dev/vdc        4.6T  4.6T  644K 100% /mnt/data
/dev/vdd        100G   42G   58G  43% /var/lib/archbuild

4.1 Login in Beefcake

4.1.1 Proxy via Winston

To login in Beefcake using Winston as a proxy, you need to first login in winston and forward Beefcake IP address locally:

ssh user@parabola.nu  -L localhost:1863:192.168.9.164:1863

You can then login to Beefcake with the following command:

ssh user@localhost -p 1863

Alternatively, you can also use the information from above as a 'ProxyCommand' in your SSH configuration file(s) in ~/.ssh/config or /etc/ssh/ssh_config.

More information is also available on a mailing list post.

4.1.2 LibreVPN

This is more complicated to setup; but allows accessing Beefcake and other parabola servers, without relying on any one of them as a proxy. LibreVPN also makes it fairly simple (once setup) to add any rare hardware, or useful services to the network, for other Parabola devs to access.

TODO: this is currently not well-documented

4.2 Using your local gpg-agent to sign files on beefcake

If you're building packages on beefcake, obviously you'll need to PGP-sign them when you release them. But copying your GPG secret key to Beefcake seems risky & silly. Instead, you should set up forwarding so that Beefcake can use your local gpg-agent that has access to your local secret key, rather than running gpg-agent on Beefcake.

4.2.1 Step 1: Stop Beefcake from automatically starting gpg-agent for you

user@beefcake $ mkdir -p ~/.config/systemd/user
user@beefcake $ ln -sT /dev/null ~/.config/systemd/user/gpg-agent.socket

4.2.2 Step 2: Configure SSH to forward your local gpg-agent

Unfortunately, this requires knowledge of where GPG will look for the gpg-agent socket. Which is under-documented, and relies on undocumented heuristics for magically trying to guess the appropriate place, instead of just letting the user configure it.

If /run/user/$UID/ exists (as it will on systemd systems (and elogind systems?); systemd keeps track of that directory as that user's $XDG_RUNTIME_DIR), then you insert the following in to the appropriate section of your SSH config:

user@local:~/.ssh.config
Host beefcake
	RemoteForward /run/user/1015/gnupg/S.gpg-agent /run/user/1000/gnupg/S.gpg-agent.extra
	#                       ^^^^                             ^^^^
	#                your UID in hackers.git             your local UID

	# Whatever DISPLAY or TTY name the remote `gpg` sends to gpg-agent will be on Beefcake,
	# not on the local host.  So we need to tell gpg-agent the correct DISPLAY/TTY info. 
	PermitLocalCommand yes
	LocalCommand gpg-connect-agent UPDATESTARTUPTTY /bye

If /run/user/$UID/ doesn't exist (as it won't OpenRC systems without elogind), then GnuPG puts the socket somewhere else:

user@local:~/.ssh.config
	RemoteForward /run/user/1015/gnupg/S.gpg-agent /home/luke/.gnupg/S.gpg-agent.extra
	#                       ^^^^                   ^^^^^^^^^^
	#                your UID in hackers.git     your local HOME

5 Notes

TLS certs are used by:

  • nginx
  • dovecot (only used by postfix; does it really need TLS?)
  • postfix
  • prosody

6 TODO

  • murmurd? *DONE*
  • Use NSD (or PowerDNS) to manage DNS zones from the server.
  • get proton using nshd *n/a*
  • migrate services from proton to winston *DONE*
  • set up autobuilder on winston *DONE ('your-freedom' and 'parabola-keyring' git hooks)*
  • switch most A records to CNAME records
  • ...

7 IN PROGRESS

  • re-provision proton services (DONE)
  • update the above linked articles
  • increase nesting of In-Progress-2019-11 (below) to replace this heading

7.1 Categorized List of Services:

7.1.1 web services

native

  • cgit

PHP

  • mediawiki

python

  • mailman
  • parabolaweb (packages, hackers, news)
  • pipermail (email archives)

ruby

  • redmine (issue tracker, forum)

7.1.2 essential public services

  • postfix (SMTP)
  • repo HTTP access
  • repo rsync access

7.1.3 essential private services

  • nshd (all boxes will need this)
  • tinc (at least one reliable box needs this)

7.1.4 essential internal services

  • autobuilder (hackers=>parabola-keyring/nshd, blacklist=>your-freedom)
  • duplicity (backups)
  • pbot
  • upstream repo importer

7.1.5 essential storage

  • backups (duplicity cross-mirrored 60-100GB per mirror)
  • git (1GB)
  • packages (~80GB per arch - currently 3 arches - 2 more planned)
  • ISOs (5-50 GB - depending how we slice them)

7.1.6 non-essential public services

  • bazaar
  • docecot (@parabola.nu IMAP)
  • git HTTP access
  • mumble (VOIP)
  • prosody (XMPP)

7.1.7 notes

most of the "non-essential public services" are rarely or never used; and are disabled

the ones that are most tightly coupled are autobuilder<->git-storage, and postfix<->mailman - everything labeled as "essential" is probably best kept on the same box, and perhaps replicated on another for emergency use - all of those essentials were on winston, while the web and non-essentials were split between proton and winston

lukeshu explained to me that the separation between proton and winston was such that all clients of mysql were on one box and all clients of pg were on the other - when we migrated everything to winston, lukeshu was leery that mixing these could cause performance problems; so that is something to consider with regards to re-provisioning