Hacking:Servers
Contents
1 DNS
1.1 Subdomains
2021-09 - the Newtron server is planned to replace the previous web server (Hacking:Servers/Proton), but is not yet in full operation - this 'Subdomains' section is a WIP, documenting the migration of the web services currently/temproarily hosted on Winston, from Winston to Newtron
TODO:
- check which of these are still registered and/or handled by the server
- we should re-evaluate the duplicated/confusing semantics/routing of {bugs,labs,projects,git,pur} and {conference,xmpp}
Newtron Sub-domains | Notes | TODO: |
---|---|---|
bugs.parabolagnulinux.org | legacy tracker? | ? |
conference.parabolagnulinux.org | no service (perhaps was alias for 'xmpp.') | reserve for mumble, jitsimeet, or similar - mumble is installed but not running |
forum.parabola.nu | web forum | delete forums.parabola.nu? |
forum.parabolagnulinux.org | ||
forums.parabola.nu | ||
labs.parabola.nu | issue tracker | ? |
labs.parabolagnulinux.org | ||
licenses.parabolagnulinux.org | no service | ? |
lists.parabola.nu | mailman | delete list.parabolagnulinux.org? |
lists.parabolagnulinux.org | ||
list.parabolagnulinux.org | ||
packages.parabola.nu | parabolaweb packages | ? |
packages.parabolagnulinux.org | ||
projects.parabola.nu | git server and cgit (redirects to git.parabola.nu) | split? git.parabola.nu(git)/projects.parabola.nu(cgit) - or replace cgit with pagure and delete projects.parabola.nu, replacing with forge.parabola.nu, code.parabola.nu, or similar |
projects.parabolagnulinux.org | ||
pur.parabola.nu | no service | ? |
pur.parabolagnulinux.org | ||
wiki.parabola.nu | mediawiki | ? |
wiki.parabolagnulinux.org | ||
www.parabola.nu | parabolaweb | ? |
www.parabolagnulinux.org | ||
xmpp.parabolagnulinux.org | no service (was prosody) | prosody is installed but not running |
Winston Sub-domains | Notes | TODO: |
---|---|---|
git.parabola.nu | git server and cgit | replace cgit with pagure? |
redirector.parabola.nu | redirector | OK |
repo.parabola.nu | repo rsync | OK |
repo.parabolagnulinux.org | ||
repomirror.parabola.nu | ? | ? |
repomirror.parabolagnulinux.org |
1.2 Records
These are the DNS entries with he.net, as of 2016-06-18.
Record | Type | Value |
---|---|---|
parabola.nu. | NS | ns1.he.net. |
parabola.nu. | NS | ns2.he.net. |
parabola.nu. | NS | ns3.he.net. |
parabola.nu. | NS | ns4.he.net. |
parabola.nu. | NS | ns5.he.net. |
parabola.nu. | A | 80.87.131.232 |
parabola.nu. | MX | 10 repo.parabola.nu. |
_xmpp-client._tcp.parabola.nu. | SRV | 0 0 5222 xmpp.parabolagnulinux.org. |
_xmpp-server._tcp.parabola.nu. | SRV | 0 0 5269 xmpp.parabolagnulinux.org. |
labs.parabola.nu. | A | 80.87.131.232 |
parabola.nu. | TXT | "v=spf1 ip4:80.87.131.232 -all" |
parabola.nu. | SPF | "v=spf1 ip4:80.87.131.232 -all" |
labs.parabola.nu. | TXT | "v=spf1 -all" |
_xmpp-client._tcp.parabola.nu. | TXT | "v=spf1 -all" |
_xmpp-server._tcp.parabola.nu. | TXT | "v=spf1 -all" |
labs.parabola.nu. | SPF | "v=spf1 -all" |
_xmpp-client._tcp.parabola.nu. | SPF | "v=spf1 -all" |
_xmpp-server._tcp.parabola.nu. | SPF | "v=spf1 -all" |
repo.parabola.nu. | A | 80.87.131.232 |
wiki.parabola.nu. | A | 80.87.131.232 |
www.parabola.nu. | A | 80.87.131.232 |
lists.parabola.nu. | A | 80.87.131.232 |
projects.parabola.nu. | A | 80.87.131.232 |
repo.parabola.nu. | SPF | "v=spf1 ip4:80.87.131.232 -all" |
repo.parabola.nu. | TXT | "v=spf1 ip4:80.87.131.232 -all" |
lists.parabola.nu. | TXT | "v=spf1 ip4:80.87.131.232 -all" |
lists.parabola.nu. | SPF | "v=spf1 ip4:80.87.131.232 -all" |
projects.parabola.nu. | TXT | "v=spf1 -all" |
wiki.parabola.nu. | TXT | "v=spf1 -all" |
www.parabola.nu. | TXT | "v=spf1 -all" |
projects.parabola.nu. | SPF | "v=spf1 -all" |
wiki.parabola.nu. | SPF | "v=spf1 -all" |
www.parabola.nu. | SPF | "v=spf1 -all" |
localhost.parabola.nu. | A | 80.87.131.232 |
localhost.parabola.nu. | TXT | "v=spf1 ip4:80.87.131.232 -all" |
localhost.parabola.nu. | SPF | "v=spf1 ip4:80.87.131.232 -all" |
pur.parabola.nu. | A | 80.87.131.232 |
pur.parabola.nu. | TXT | "v=spf1 -all" |
pur.parabola.nu. | SPF | "v=spf1 -all" |
redirector.parabola.nu. | A | 80.87.131.232 |
repomirror.parabola.nu. | A | 80.87.131.232 |
winston.parabola.nu. | A | 93.95.226.249 |
proton.parabola.nu. | A | 80.87.131.232 |
git.parabola.nu. | CNAME | winston.parabola.nu. |
parabolagnulinux.org. | NS | ns1.he.net. |
parabolagnulinux.org. | NS | ns2.he.net. |
parabolagnulinux.org. | NS | ns3.he.net. |
parabolagnulinux.org. | NS | ns4.he.net. |
parabolagnulinux.org. | NS | ns5.he.net. |
parabolagnulinux.org. | A | 80.87.131.232 |
projects.parabolagnulinux.org. | A | 80.87.131.232 |
wiki.parabolagnulinux.org. | A | 80.87.131.232 |
repo.parabolagnulinux.org. | A | 80.87.131.232 |
licenses.parabolagnulinux.org. | A | 80.87.131.232 |
lists.parabolagnulinux.org. | A | 80.87.131.232 |
forum.parabolagnulinux.org. | A | 80.87.131.232 |
parabolagnulinux.org. | MX | 0 repo.parabola.nu. |
_xmpp-server._tcp.parabolagnulinux.org. | SRV | 0 0 5269 xmpp.parabolagnulinux.org. |
_jabber._tcp.parabolagnulinux.org. | SRV | 0 0 5269 xmpp.parabolagnulinux.org. |
_xmpp-client._tcp.parabolagnulinux.org. | SRV | 0 0 5222 xmpp.parabolagnulinux.org. |
list.parabolagnulinux.org. | MX | 0 repo.parabola.nu. |
lists.parabolagnulinux.org. | MX | 0 repo.parabola.nu. |
conference.parabolagnulinux.org. | A | 80.87.131.232 |
list.parabolagnulinux.org. | A | 80.87.131.232 |
xmpp.parabolagnulinux.org. | A | 80.87.131.232 |
parabolagnulinux.org. | TXT | "v=spf1 ip4:80.87.131.232 -all" |
lists.parabolagnulinux.org. | TXT | "v=spf1 ip4:80.87.131.232 -all" |
repo.parabolagnulinux.org. | TXT | "v=spf1 ip4:80.87.131.232 -all" |
bugs.parabolagnulinux.org. | TXT | "v=spf1 -all" |
conference.parabolagnulinux.org. | TXT | "v=spf1 -all" |
forum.parabolagnulinux.org. | TXT | "v=spf1 -all" |
licenses.parabolagnulinux.org. | TXT | "v=spf1 -all" |
list.parabolagnulinux.org. | TXT | "v=spf1 -all" |
packages.parabolagnulinux.org. | TXT | "v=spf1 -all" |
projects.parabolagnulinux.org. | TXT | "v=spf1 -all" |
wiki.parabolagnulinux.org. | TXT | "v=spf1 -all" |
xmpp.parabolagnulinux.org. | TXT | "v=spf1 -all" |
_xmpp-client._tcp.parabolagnulinux.org. | TXT | "v=spf1 -all" |
_xmpp-server._tcp.parabolagnulinux.org. | TXT | "v=spf1 -all" |
_jabber._tcp.parabolagnulinux.org. | TXT | "v=spf1 -all" |
bugs.parabolagnulinux.org. | SPF | "v=spf1 -all" |
conference.parabolagnulinux.org. | SPF | "v=spf1 -all" |
forum.parabolagnulinux.org. | SPF | "v=spf1 -all" |
licenses.parabolagnulinux.org. | SPF | "v=spf1 -all" |
list.parabolagnulinux.org. | SPF | "v=spf1 -all" |
lists.parabolagnulinux.org. | SPF | "v=spf1 ip4:80.87.131.232 -all" |
packages.parabolagnulinux.org. | SPF | "v=spf1 -all" |
parabolagnulinux.org. | SPF | "v=spf1 ip4:80.87.131.232 -all" |
projects.parabolagnulinux.org. | SPF | "v=spf1 -all" |
repo.parabolagnulinux.org. | SPF | "v=spf1 ip4:80.87.131.232 -all" |
wiki.parabolagnulinux.org. | SPF | "v=spf1 -all" |
xmpp.parabolagnulinux.org. | SPF | "v=spf1 -all" |
_xmpp-client._tcp.parabolagnulinux.org. | SPF | "v=spf1 -all" |
_xmpp-server._tcp.parabolagnulinux.org. | SPF | "v=spf1 -all" |
_jabber._tcp.parabolagnulinux.org. | SPF | "v=spf1 -all" |
www.parabolagnulinux.org. | A | 80.87.131.232 |
labs.parabolagnulinux.org. | A | 80.87.131.232 |
pur.parabolagnulinux.org. | A | 80.87.131.232 |
pur.parabolagnulinux.org. | TXT | "v=spf1 -all" |
pur.parabolagnulinux.org. | SPF | "v=spf1 -all" |
www.parabolagnulinux.org. | TXT | "v=spf1 -all" |
labs.parabolagnulinux.org. | TXT | "v=spf1 -all" |
labs.parabolagnulinux.org. | SPF | "v=spf1 -all" |
www.parabolagnulinux.org. | SPF | "v=spf1 -all" |
repomirror.parabolagnulinux.org. | A | 80.87.131.232 |
1.3 SPF
These are mail domains, i.e. ones listed in envelope From addresses or used for outgoing mail servers:
- parabola.nu
- lists.parabola.nu
- repo.parabola.nu
Mail domains have TXT and SPF records with the value "v=spf1 ip4:80.87.131.232 -all", i.e. allowing only mail sent from the parabola.nu server. All other domains have "v=spf1 -all": all mail from them is fake.
We use SPF to get less spam claiming to be from our domains and to discourage spammers from using fake sender addresses.
2 proton.parabola.nu
proton.parabola.nu was a VPS hosted somewhere in the UK, on a machine operated by user n1md4 in cooperation with his employer, Positive Internet.
It is no longer used by Parabola.
2.1 Resources
$ uname -m x86_64 $ free -h total used free shared buffers cached Mem: 1.9G 1.8G 100M 191M 28M 538M -/+ buffers/cache: 1.2G 667M Swap: 1.0G 413M 608M $ df -h | grep sda /dev/sda1 98G 12G 82G 13% / /dev/sda3 150G 125G 26G 83% /srv
2.2 Reboot
Once upon a time, reboots were hard to do because of a weird issue that could make shutdown take half an hour! That's a lot of downtime and praying!
Nowadays, it takes about 3 and a half minutes (measured at the Nov 8 1:00 GTM reboot):
shut down | 00:01:36 |
BIOS | 00:00:30 |
boot up | 00:01:21 |
total | 00:03:27 |
---|
Still slower than we would like, but so are most things on Proton.
2.3 Public-facing network sockets/services
socket | IPv4 | IPv6 | protocol | daemon | other info |
---|---|---|---|---|---|
TCP:*:22 | Yes | Yes | SSH | sshd.service | |
TCP:*:25 | Yes | Yes | SMTP | postfix.service/master | |
TCP:*:80 | Yes | Yes | HTTP | nginx.service | |
TCP:*:443 | Yes | Yes | HTTPS | nginx.service | |
TCP:*:465 | Yes | Yes | SMTPS | postfix.service/master | |
TCP:*:587 | Yes | Yes | SMTP-MSA | postfix.service/master | |
TCP:*:655 | Yes | Yes | tinc | tincd@lvpn.service | |
TCP:*:875 | Yes* | Yes | rsync | rsync.socket | |
TCP:*:1863 | Yes | Yes | SSH | sshd.service | |
TCP:*:5222 | Yes | Yes | xmpp-client | prosody.service | |
TCP:*:5269 | Yes | Yes | xmpp-server | prosody.service | |
TCP:*:9418 | Yes* | Yes | git | git-daemon.socket |
2.4 Inward-facing sockets
socket | protocol | unit | |
---|---|---|---|
TCP4/6:localhost.localdomain:2812 | HTTP | monit.service | |
TCP4:localhost.localdomain:5432 | pgsql | postgresql.service | |
TCP4/6:localhost.localdomain:5582 | telnet | prosody.service | |
unix:/run/mailman-fcgi.sock | FastCGI | mailman-fcgi.socket | |
unix:/run/uwsgi/labs.sock | uwsgi | uwsgi@labs.socket | |
unix:/run/uwsgi/parabolaweb.sock | uwsgi | uwsgi@parabolaweb.socket | |
unix:/run/uwsgi/projects.sock | uwsgi/modifier1=9 | uwsgi@projects.socket | |
unix:/run/uwsgi/repo.sock | uwsgi/modifier1=14 | uwsgi@repo.socket | |
unix:/run/dovecot/... | misc | dovecot.service | |
unix:/var/spool/postfix/private/auth | Dovecot SASL | dovecot.service | |
unix:/var/spool/postfix/private/* | misc | postfix.service | |
System stuff | |||
unix:/run/dbus/systemd_bus_socket | D-Bus | dbus.socket | |
unix:/run/lvm/lvmetad.socket | ??? | lvm2-lvmetad.socket | |
unix:/run/udev/control | ??? | systemd-udev-control.socket | |
unix:/run/systemd/... | misc | misc | |
unix:/run/user/${UID}/{bus,systemd/{notify,private}} | misc | user@${UID}.service |
2.5 Other running services of note
- dovecot.service
- mailman.service
- parabolaweb-reporead-inotify.service
2.6 Nginx "servers"
server_name | HTTP | HTTPS |
---|---|---|
Simple redirects | ||
* | return 301 https://$host$request_uri; | return 301 https://www.parabola.nu/404; |
parabolagnulinux.org | N/A | return 301 https://www.parabola.nu$request_uri; |
*.parabolagnulinux.org | N/A | return 301 https://$subdomain.parabola.nu$request_uri; |
list.parabolagnulinux.org | N/A | return 301 https://lists.parabola.nu$request_uri; |
parabola.nu | N/A | return 301 https://www.parabola.nu$request_uri |
Websites | ||
www.parabola.nu | N/A | Serve /static/, /favicon.ico, /robots.txt,and /img/ statically, redirect /https to /, and hand everything else off to uWSGI |
labs.parabola.nu | N/A | Redirect / to /projects; use uWSGI |
lists.parabola.nu | N/A |
|
projects.parabola.nu | N/A | Serve cgit via uWSGI |
repo.parabola.nu | N/A | Serve the union of /srv/repo/main and /srv/repo/http, using repoindex.php (via uWSGI) for indexes. |
wiki.parabola.nu | N/A | Serve MediaWiki via uWSGI (https://lukeshu.com/blog/nginx-mediawiki.html) |
pur.parabola.nu | N/A | TODO |
redirector.parabola.nu | N/A | TODO |
repomirror.parabola.nu | N/A | TODO |
2.7 Mail configuration
Postfix's postscreen handles port 25 enforcing the pregreet test, checks SPF records via python2-postfix-policyd-spf
, the mail is delivered via Dovecot deliver. Both deliver and the daemon need to be of the same version, so restarting the daemon after an update is needed.
Mailman handles the lists, with Postfix integration via a virtual address map. Use /usr/lib/mailman/bin/newlist
to add a list.
2.8 Configuration things
There used to be several symlinks added to /var to keep things in sane places, but they have been replaced by bind mounts in /etc/fstab, because the symlinks confused pacman.
/var/lib/mailman -> ../../srv/mailman /var/lib/mysql -> ../../srv/sql/mysql /var/lib/postgres -> ../../srv/sql/postgres /var/spool/cron -> ../../etc/cron.spool
That is, nothing of consequence and needing to be backed up should live anywhere but /srv or /etc. Maybe /home for personal stuff.
3 winston.parabola.nu
winston.parabola.nu is A VPS server which is hosted for free by 1984, a Hosting Company in Iceland.
$ uname -m x86_64 $ free -h total used free shared buff/cache available Mem: 4.4G 98M 2.2G 692K 2.0G 4.0G Swap: 4.3G 0B 4.3G $ df -h | grep ^/dev /dev/vda3 94G 3.2G 86G 4% / /dev/vda4 375G 5.7M 355G 1% /srv
Pardon the sloppy formatting (<code> instead of {{ic|), I wrote this in markdown and used pandoc to convert it to mediawiki.
3.1 configuration philosophy
Avoid editing files owned by a package if possible; insert things into new files. If a file must be edited, try to avoid having to change any lines; try to only add new lines.
Everything worth backing up should be in /etc
, /srv
, or /home
.
3.2 infrastructure/management
3.2.1 base setup
Packages installed:
- group:
base
except forlinux-libre
-
linux-libre-lts
-
haveged
-
irqbalance
Files affected:
-
/etc/hostname
:winston.parabola.nu
-
/etc/locale.conf
:LANG=en_US.UTF-8
LC_COLLATE=C
-
/etc/locale.gen
:en_US.UTF-8 UTF-8
-
/etc/machine-info
:LOCATION=1984 Hosting Company, Iceland
-
/etc/vconsole.conf
:KEYMAP=us
CONSOLEMAP=8859-1
FONT_MAP=8859-1_to_uni
-
/etc/systemd/system/multi-user.target.wants/getty@tty1.service
-
/etc/systemd/system/multi-user.target.wants/haveged.service
-
/etc/systemd/system/multi-user.target.wants/irqbalance.service
3.2.1.1 bootloader
Packages installed:
-
grub
Files affected:
-
/etc/default/grub
The grub config has two overrides added to it:
GRUB_CMDLINE_LINUX_DEFAULT="" GRUB_TERMINAL_OUTPUT=console
3.2.1.2 filesystems
Packages installed:
-
systemd-swap
Files affected:
-
/etc/fstab
-
/etc/systemd-swap.conf
-
/etc/systemd/system/local-fs.target.wants/systemd-swap.service
-
/etc/systemd/system/multi-user.target.wants/remote-fs.target
TODO: document
fstab is set up similarly to proton.parabola.nu.
3.2.1.3 networking
Files affected:
-
/etc/udev/rules.d/80-net-setup-link.rules
-
/etc/netctl/eth0-static
-
/etc/systemd/system/netctl@eth0\x2dstatic.service
(auto-generated bynetctl enable
) -
/etc/systemd/system/multi-user.target.wants/netctl@eth0\x2dstatic.service
/etc/udev/rules.d/80-net-setup-link.rules
is a symlink to /dev/null
, which disables new-style predictable network interface names (enp0s3
? ens3
?), and causes it to fall back to the old-style names (eth0
).
The netctl profile eth0-static
just has the network information from the 1984 VPS control panel.
3.2.1.4 Pacman
Files affected:
-
/etc/pacman.conf
: Enable[pcr]
-
/etc/pacman.d/mirrorlist
: Userepo.parabola.nu
directly -
/etc/pacman.d/.gitignore
:/gnupg/
3.2.1.5 timedate
Files affected:
-
/etc/localtime
:-> /usr/share/zoneinfo/UTC
-
/etc/systemd/system/sysinit.target.wants/systemd-timesyncd.service
timesyncd is an NTP client daemon.
3.2.2 etckeeper
Packages installed:
- pristine-etc-keeper
Files affected:
-
/etc/systemd/system/multi-user.target.wants/pristine-etc-keeper.timer
-
/etc/systemd/system/multi-user.target.wants/etckeeper.timer
-
/etc/.gitignore
: append!/mtab
/machine-id
/group-
/gshadow-
/passwd-
/shadow-
/resolv.conf
-
/etc/udev/.gitignore
:/hwdb.bin
etckeeper is configured to use git to keep track of changes in /etc
. The systemd unit etckeeper.timer
is enabled, which makes a commit (if necessary) daily. It will also run before and after pacman
via libalpm hooks.
In addition to etckeeper keeping track of the current configuration, lukeshu's pristine-etc-keeper maintains a branch of what /etc
would be like if we never made any changes from the default files. The clean version of etc is available as the clean
remote's master
branch. Doing a git diff clean/master master
should be helpful in investigating how things have been configured.
3.2.3 users
See also: keys user and group
3.2.3.1 Parabola hackers
Packages installed:
-
parabola-hackers-nshd
-
openssh
Files affected:
-
/var/lib/hackers-git/
-
/etc/ssh/sshd_config
-
/etc/nsswitch.conf
-
/etc/pam.d/passwd
-
/etc/pam.d/system-auth
-
/etc/pam.d/system-login
-
/etc/systemd/system/sockets.target.wants/nshd.socket
-
/etc/systemd/system/dbus.service.wants/nshd.service
(temporary systemd bug workaround) -
/etc/nshd/shadow
sshd is configured to force the use of keys (no password-based login), and to use parabola-hackers ssh-list-authorized-keys
in addition to checking ~/.ssh/authorized_keys
. ssh-list-authorized-keys
returns the authorized keys from the hackers.git checkout in /var/lib/hackers-git
(the path to the checkout is configured in /etc/parabola-hackers.yml
).
NSS and PAM have been configured to use the ldap modules that are part of nss-pam-ldapd. However, instead of running the normal nslcd
LDAP client daemon, the system has ben configured to run the parabola-hackers-nshd nshd
daemon, which reads user infomation from the same hackers.git
checkout (configured the same way). This way we dn't have to worry about keeping /etc/passwd
in sync with hackers.git
. To this end, PAM has also been configured to create a users home directory when they log in if it doesn't already exist. Because hackers.git
doesn't store any password information, nshd
stores password hashes in /etc/nshd/shadow
.
Sometimes after something fails in PAM, you get a "User not known to the underlying authentication module" message. Like, the pam_ldap.so
failed because you typed your password wrong, but it acts like it failed because it didn't "own" the user. I think that it's just a bug in PAM's message selection. But (TODO) we should actually track it down.
3.2.3.2 emergency user
Files affected:
-
/etc/passwd
-
/etc/shadow
-
/etc/sudoers.d/99-emergency
-
/home/emergency/
In case anything should ever go wrong with hackers.git
, the user "emergency" has been set up. "emergency" has authorized (statically via ~/.ssh/authorized_keys
) the keys of serveral (not publicly disclosed) Parabola hackers. /etc/sudoers.d/99-emergency
grants special privileges in case PAM or NSS get screwed up.
3.2.3.3 other
Files affected:
-
/etc/sudoers.d/00-wheel
-
/etc/systemd/system/shadow.service.d/sort.conf
/etc/sudoers.d/00-wheel
gives sudo access to everyone in the wheel
group.
The shadow.service
has been extended to sort the files if they otherwise are ok. This makes dealing with pacman updates and such easier.
3.2.3.4 pbot
Files affected:
- /etc/group
- /etc/gshadow
- /etc/passwd
- /etc/shadow
- /etc/systemd/system/pbot.service <-- this script is coppied from the pbot git repo
- /etc/systemd/system/multi-user.target.wants/pbot.service <-- created by systemctl enable
3.2.4 SSH
Packages installed:
-
openssh
Files affected:
-
/etc/ssh/sshd_config
-
/etc/systemd/system/multi-user.target.wants/sshd.service
-
/etc/ssh/.gitignore
:/ssh_host_*_key
/ssh_host_*_key.pub
See above for how authentication and users are set up.
sshd is also configured to listen on both ports 22 and 1863. We may turn of port 22 in the future. Not using port 22 isn't security through obscurity, it is security through keeping-the-the-logs-useful-by-keeping-noise-down.
3.2.5 SSL
Packages installed:
-
certbot
Winston uses the certbot ACME client to get certificates from Let's Encrypt.
All domains handled by the server are shoved in as Subject Alternative Names in a single certificate. This makes configuring nginx easier.
3.2.5.1 keys user and group
Files affected:
-
/etc/passwd
-
/etc/shadow
-
/etc/group
-
/etc/gshadow
-
/etc/letsencrypt
-
/var/lib/letsencrypt
-
/var/log/letsencrypt
In order to run certbot as a non-root user, the keys
user and group have been created:
useradd --system --user-group --no-create-home --home-dir /etc/ssl --shell /usr/bin/nologin keys chown -R keys:keys /etc/letsencrypt /var/log/letsencrypt /var/lib/letsencrypt chmod 750 /etc/letsencrypt/archive /etc/letsencrypt/live
The associated keys
group allows users to read the (private) keys in /etc/letsencrypt/live
.
3.2.5.2 issuance, renewal, and installation
Files affected:
-
/etc/ssl/misc/certbot-get
-
/etc/ssl/misc/certbot-hook
-
/etc/sudoers.d/10-certbot
-
/etc/systemd/system/certbot-renew.service
-
/etc/systemd/system/certbot-renew.timer
-
/etc/systemd/system/timers.target.wants/certbot-renew.timer
-
/etc/nginx/snippets/ssl.conf
Unlike acmetool, certbot doesn't have an easy way of saying "please add this domain as a Subject Alternative Name". You have to re-run the same (long) command to get the cert, but with the domain added. So, I've encapsulated this into the script /etc/ssl/misc/certbot-get
. Edit the array of domains at the top of the script, then run it.
Renewal, however, is very simple. It is handled by certbot-renew.service
(triggered by the associated .timer
). It runs certbot renew
with a couple of flags.
Both certbot-get
and certbot-renew.service
prove ownership of the domain via the http-01
challenge. /etc/nginx/nginx.conf
includes /etc/nginx/snippets/ssl.conf
, which has a server{}
block that handles ACME http-01 challenges.
Both certbot-get
and certbot-renew.service
have been written to run sudo /etc/ssl/misc/certbot-hook
after certificates have been updated, and sudo has been configured to allow the keys
user to do this without a password. Right now certbot-hook
just runs systemctl reload nginx.service
.
3.2.5.3 other
Files affected:
-
/etc/nginx/nginx.conf
-
/etc/nginx/snippets/ssl.conf
-
/etc/ssl/private/dhparam-2048.pem
-
/etc/ca-certificates/.gitignore
:/trust-source/blacklist/
/trust-source/anchors/
/extracted/
-
/etc/ssl/.gitignore
:/certs/*.0
/certs/*.1
/certs/*.pem
/certs/ca-certificates.crt
/certs/java/cacerts
/.ssh/
nginx.conf
includes snippets/ssl.conf
, which is primarily based on the output of Mozilla Security's recommended web server configuration generator. It has had the main SSL information promoted to be directly into the http{}
block, instead of having to be in each server{}
block. The HTTP->HTTPS redirector has had an exception added to it to have it respond to ACME http-01 challenges.
Because certbot is only configured to use http-01 challenges, the all challenges happen over pain HTTP, which means that the configurations for each subdomain (which only serve over HTTPS/HTTP2) do not need to include anything about ACME or SSL (other than mentioning ssl
in the listen
directive).
ssl.conf
needs to refer to a dhparam PEM file. This has been generated with the command
openssl dhparam -out /etc/ssl/private/dhparam-2048.pem 2048
3.2.6 HTTP
Packages installed:
-
nginx
Files affected:
-
/etc/nginx/nginx.conf
-
/etc/nginx/fastcgi.conf
,/etc/nginx/{fastcgi,scgi,uwsgi}_params
-
/etc/nginx/mime.types
-
/etc/nginx/snippets/ssl.conf
-
/etc/nginx/sites/
-
/etc/ssl/misc/certbot-hook
-
/etc/systemd/system/multi-user.target.wants/nginx.service
See the #SSL for how nginx has been configured to do HTTPS.
fastcgi.conf
, fastcgi_params
, scgi_params
and uwsgi_params
have been edited to pass 127.0.0.1 as the client IP address to worker processes, to protect user privacy.
The log_format
has been customized to redact the client IP and referrer, again to protect user privacy. Unfortunately, the libre version of nginx does not allow for customizing the format of the error log, so user IPs may still end up in the error log.
mime.types
has had xz, gzip, bzip2, tar, PGP (.sig
), and bittorrent types added to it. This list was based on the unmatched file extensions that find
turned up on repo.parabola.nu
.
Per-subdomain configurations live in /etc/nginx/sites/*.conf
(which is the exact wildcard used with include
at the end of nginx.conf
).
"Special" rules in sites/*.conf
should have the alias-
or meta-
prefix. For example, right now there is only meta-unknown-domain.conf
which is a fallback for when no other file handles a domain (it redirects to https://www.parabola.nu/404
). Normal domain handlers should be named server-${domain}.conf
Every sites/server-*.conf
file should be of the format (replace subdomain with the subdomain):
/etc/nginx/sites/server-subdomain.parabola.nu.conf
# -*- Mode: nginx; nginx-indent-level: 8; intent-tabs-mode: t -*- server { server_name subdomain.parabola.nu; listen 443 ssl http2; listen [::]:443 ssl http2; error_log /var/log/nginx/nginx.http.subdomain.parabola.nu.error.log error; access_log /var/log/nginx/nginx.http.subdomain.parabola.nu.access.log specific; ... }
nginx.conf
sets root /srv/http/$server_name
, which means that most of the time root
won't need to be set in each individual server{}
block. Unfortunately, error_log
and access_log
don't support variable expansion, so they do need to be set each time.
The individual sites/server-*.conf
files won't be documented in this section; see the section for the service that it provides.
TODO: document the rest of the settings in nginx.conf
3.2.6.1 process management
Packages installed:
-
uwsgi
Files affected:
-
/etc/systemd/system/uwsgi@.service.d/socket.conf
-
/etc/systemd/system/uwsgi@.socket.d/owner.conf
-
/etc/uwsgi/vassal.ini
-
/etc/systemd/system/sockets.target.wants/uwsgi@vassal.socket
Wherever possible, we should use uWSGI for process management of our HTTP services. This allows for much more consistent management and configuration than the hodge-podge of PHP-FPM, manage.py, fcgiwrap, et c. that we used to have on Proton.
uWSGI is the program, uwsgi is the protocol it speaks with nginx. A pool of workers is called a vassal, and is configured in /etc/uwsgi/vassal.ini
, and activated by uwsgi@vassal.socket
; a socket speaking the uwsgi protocol is created at /var/run/uwsgi/vassal.sock
.
We use systemd socket activation rather than a uWSGI emperor because they provide largely the same functionality; the only real advantage that a uWSGI emperor would provide over systemd socket activation is if you ran it in tyrant mode, it lets you do some cool things with unpriveleged access, which would be useful for a shared web host. We aren't a shared web host, and have no reason to run emperor in tyrant mode.
Since the uwsgi@.service
vassal unit is written to support socket-activated or non-socket-activated use, it is normally possible to accidentally start it without the associated .socket
unit; which is an error with how our vassal configurations are written. To fix this, uwsgi@.service.d/socket.conf
overrides the unit a bit to disable non-socket-activated use.
The ownership and permissions for the socket are configured in uwsgi@.socket.d/owner.conf
, which sets the owner to http:http
and the mode to 0600.
uWSGI supports thread pools in addition to process pools, but many of the actual workers you'll want to use aren't thread safe, so stick to process pools unless you specifically know that your worker is thread-safe (for example, PHP, at least with the modules needed for MediaWiki, is not thread-safe).
Individual vassal configurations are documented in the section for the service that they provide, not here.
3.2.7 email
Packages installed:
-
postfix
Files affected:
-
/etc/systemd/system/multi-user.target.wants/postfix.service
-
/etc/postfix/.gitignore
:*.db
Configuration
-
/etc/postfix/main.cf
edited to set make it accept, and locally deliver, mails to <user>@winston.parabola.nu.
Operation
- pbot gets emails delivered qmail-style in
$HOME/Maildir/
to allow him to report bug tracker changes - currently postfix aliases has not been set up. where should mails addressed to e.g. postmaster@winston.parabola.nu go? at the moment they get locally delivered to root user
3.3 services
You know, the things that the server is actually for.
3.3.1 System Information
Packages installed:
-
uwsgi-plugin-cgi
Files affected:
-
/srv/http/winston.parabola.nu/sysinfo.cgi
-
/etc/uwsgi/winston.ini
-
/etc/nginx/sites/server-winston.parabola.nu.conf
-
/etc/systemd/system/sockets.target.wants/uwsgi@winston.socket
Serves a simple system information page at https://winston.parabola.nu/. The source of the (very simple) script is at https://winston.parabola.nu/sysinfo.cgi.
3.3.2 ParabolaWeb
Packages installed:
-
uwsgi-plugin-python2
-
parabolaweb-utils
-
parabolaweb-dev
Files affected:
-
/etc/passwd
-
/etc/shadow
-
/srv/http/www.parabola.nu/
Winston isn't running ParabolaWeb (yet), but has it installed because other pages symlink to the branding files in it.
useradd --comment ParabolaWeb --home-dir /srv/http/www.parabola.nu/ --create-home --system --gid http parabolaweb
3.3.3 Bazaar
Packages installed:
-
bzr
Files affected:
-
/etc/passwd
-
/etc/systemd/system/bzr.service
-
/etc/systemd/system/multi-user.target.wants/bzr.service
-
/srv/bzr/
TODO: IDK, someone else set this up.
3.3.4 Git
Packages installed:
-
git
-
cgit
-
uwsgi-plugin-cgi
Files affected:
-
/etc/passwd
-
/etc/sudoers.d/10-git
-
/etc/cron.spool/git
-
/srv/git/
The "git" user has been modified:
usermod --comment "Parabola git" --shell /usr/bin/git-shell git
The git repositories live in /srv/git/
and are owned by the git user.
Metadata about all of the git repositories, as well as most of the setup, lives in the git-meta.git
repository. The git user has a cron-job to update this repository every hour.
git is configured (core.hooksPath
such that it looks in ~/.config/git/hooks/
for hooks, instead of /path/to/repo.git/hooks/
. All of the server-side hooks exist: they all do the "same" thing: use run-parts
to run all of the scripts in ~/.config/git/hooks/hook-name.d/
, and then run the normal hook file /path/to/repo.git/hooks/hook-name
.
Right now there are only two "real" hooks:
-
/srv/git/.config/git/hooks/post-receive.d/agefile
: see #transport:_HTTPS -
/srv/git/hackers.git/hooks/post-receive
: update the hackers.git checkout (see #Parabola_hackers), then runsudo systemctl reload nshd.service
to activate changes to system users (sudo has been configured to allow this to be done without a password). TODO: see the hook file itself for more details, but it should also activate the changes to email aliases with postfix, as well as rebuild theparabola-keyring
package with autobuilder.
3.3.4.1 transport: git protocol
Files affected:
-
/etc/systemd/system/socket.target.wants/git-daemon.socket
3.3.4.2 transport: SSH
Files affected:
-
/srv/git/git-shell-commands/
This is the only method for which push is enabled. Only the git@ user may push; even though if you have access to git, you probably have access to ssh into yourname@winston, you can't push from that account. As far as git is concerned, all users are equal (TODO: perhaps we should implement some more granular access control. Safety rails and all that). Access to git@ is based on membership in the "git" group in hackers.git (see #Parabola_hackers).
If you need to set up custom git hooks, then ssh in to your user, and use sudo -u git
to do it.
3.3.4.3 transport: HTTPS
Files affected:
-
/etc/uwsgi/git.ini
-
/etc/systemd/system/socket.target.wants/uwsgi@git.socket
-
/etc/nginx/sites/server-git.parabola.nu.conf
-
/srv/http/git.parabola.nu/
-
/srv/git/.config/git/hooks/post-receive.d/agefile
The git web interface is cgit, which is managed by uWSGI speaking uwsgi/modifier1=9, which is the variant of the uwsgi protocol for CGI requests.
uwsgi/git.ini
sets CGIT_CONFIG=/srv/http/git.parabola.nu/cgitrc
.
A post-receive hook that runs for all repositories updates the agefile, so that cgit has a better idea of when a repository was last updated.
The files for cgit are in /srv/http/git.parabola.nu/
, which shadows /usr/share/webapps/cgit/
:
# mirror /usr/share/webapps/cgit/ cgit.cgi -> /usr/share/webapps/cgit/cgit.cgi cgit.css-> /usr/share/webapps/cgit/cgit.css# customized cgit.png -> /usr/share/webapps/cgit/cgit.png favicon.ico ->/usr/share/webapps/cgit/favicon.ico/srv/http/www.parabola.nu/web/sitestatic/favicon.ico robots.txt -> /usr/share/webapps/cgit/robots.txt # added files cgitrc archnavbar/ -> /srv/http/www.parabola.nu/web/sitestatic/archnavbar/ header.html # archnavbar syntax-highlighting.sh # based on some old version of /usr/lib/cgit/filters/syntax-highlighting.sh
3.3.5 pbot
pbot, the Parabola IRC bot, lives at pbot.git, and runs as the user pbot
Home dir at /home/pbot/
.
- This contains all the pbot files from the git repo. Not in a subdir, just right there.
- Also contains dir
logs/
TODO add to git repo - Also contains 600 permissions file passord.secret containing IRC password
- pbot writes data to
info/
andannouncements/
. these are put in a tarball every day by cron job so they can be downloaded elsewhere if wanted (e.g. for backup).
Deps installed with pacman
- moreutils
- w3m
- recode
- inotify-tools
3.3.6 Redmine
Packages installed:
- ruby
- ruby-bundler
- rubygems
- uwsgi-plugin-rack
Files affected:
- /etc/nginx/sites/server-labs_parabola_nu.conf
- /etc/systemd/system/labs-restart.socket
- /etc/systemd/system/labs-restart.timer
- /etc/systemd/system/multi-user.target.wants/uwsgi@labs.service
- /etc/systemd/system/socket-user.target.wants/uwsgi@labs.socket
- /etc/systemd/system/timers.target.wants/labs-restart.timer
- /etc/systemd/system/uwsgi@labs.service.d/psql.conf
- /etc/uwsgi/labs.ini
See also: DeveloperWiki:Web_Admin
3.3.7 Rsync
Packages installed:
- rsync
Files affected:
- /etc/rsyncd.conf
- /etc/systemd/system/rsyncd.socket
- /etc/systemd/system/sockets.target.wants/rsyncd.socket
Installed by coadde or Emulatorman during migration of repo.parabola.nu from Proton. I resumed configuration of the server and enabled it to accommodate mirrors after the DNS change. Port number (873 -> 875) and modules ( [repos]:/srv/repo/main and [abs]:/srv/abslibre) were copied verbatim from Proton.
4 beefcake.parabola.nu
Beefcake is a virtual machine occupying roughly half of LukeShu's KGPE-D16 in Indianapolis. It runs no publicly accessible services, but is intended to be usable as a build machine for Parabola developers. It has a large data disk which archives all repos since April 2018; which are exposed to the librechroots as an HTTP service.
$ uname -m x86_64 $ free -h total used free shared buff/cache available Mem: 29Gi 725Mi 250Mi 13Mi 28Gi 28Gi Swap: 7.4Gi 4.0Mi 7.4Gi $ df -h | sed -n '1p;/^\/dev/p' Filesystem Size Used Avail Use% Mounted on /dev/vda 10G 5.6G 4.1G 59% / /dev/vdb 9.8G 6.8G 2.6G 73% /home /dev/vdc 4.6T 4.6T 644K 100% /mnt/data /dev/vdd 100G 42G 58G 43% /var/lib/archbuild
4.1 Login in Beefcake
4.1.1 Proxy via Winston
To login in Beefcake using Winston as a proxy, you need to first login in winston and forward Beefcake IP address locally:
ssh user@parabola.nu -L localhost:1863:192.168.9.164:1863
You can then login to Beefcake with the following command:
ssh user@localhost -p 1863
Alternatively, you can also use the information from above as a 'ProxyCommand' in your SSH configuration file(s) in ~/.ssh/config or /etc/ssh/ssh_config.
More information is also available on a mailing list post.
4.1.2 LibreVPN
This is more complicated to setup; but allows accessing Beefcake and other parabola servers, without relying on any one of them as a proxy. LibreVPN also makes it fairly simple (once setup) to add any rare hardware, or useful services to the network, for other Parabola devs to access.
TODO: this is currently not well-documented
4.2 Using your local gpg-agent to sign files on beefcake
If you're building packages on beefcake, obviously you'll need to PGP-sign them when you release them. But copying your GPG secret key to Beefcake seems risky & silly. Instead, you should set up forwarding so that Beefcake can use your local gpg-agent that has access to your local secret key, rather than running gpg-agent on Beefcake.
4.2.1 Step 1: Stop Beefcake from automatically starting gpg-agent for you
user@beefcake $ mkdir -p ~/.config/systemd/user user@beefcake $ ln -sT /dev/null ~/.config/systemd/user/gpg-agent.socket
4.2.2 Step 2: Configure SSH to forward your local gpg-agent
Unfortunately, this requires knowledge of where GPG will look for the gpg-agent socket. Which is under-documented, and relies on undocumented heuristics for magically trying to guess the appropriate place, instead of just letting the user configure it.
If /run/user/$UID/ exists (as it will on systemd systems (and elogind systems?); systemd keeps track of that directory as that user's $XDG_RUNTIME_DIR), then you insert the following in to the appropriate section of your SSH config:
user@local:~/.ssh.config
Host beefcake RemoteForward /run/user/1015/gnupg/S.gpg-agent /run/user/1000/gnupg/S.gpg-agent.extra # ^^^^ ^^^^ # your UID in hackers.git your local UID # Whatever DISPLAY or TTY name the remote `gpg` sends to gpg-agent will be on Beefcake, # not on the local host. So we need to tell gpg-agent the correct DISPLAY/TTY info. PermitLocalCommand yes LocalCommand gpg-connect-agent UPDATESTARTUPTTY /bye
If /run/user/$UID/ doesn't exist (as it won't OpenRC systems without elogind), then GnuPG puts the socket somewhere else:
user@local:~/.ssh.config
RemoteForward /run/user/1015/gnupg/S.gpg-agent /home/luke/.gnupg/S.gpg-agent.extra # ^^^^ ^^^^^^^^^^ # your UID in hackers.git your local HOME
5 Notes
TLS certs are used by:
- nginx
- dovecot (only used by postfix; does it really need TLS?)
- postfix
- prosody
6 TODO
-
murmurd?*DONE* - Use NSD (or PowerDNS) to manage DNS zones from the server.
-
get proton using nshd*n/a* -
migrate services from proton to winston*DONE* -
set up autobuilder on winston*DONE ('your-freedom' and 'parabola-keyring' git hooks)* - switch most A records to CNAME records
- ...
7 IN PROGRESS
- re-provision proton services (DONE)
- update the above linked articles
- increase nesting of In-Progress-2019-11 (below) to replace this heading
7.1 Categorized List of Services:
7.1.1 web services
native
- cgit
PHP
- mediawiki
python
- mailman
- parabolaweb (packages, hackers, news)
- pipermail (email archives)
ruby
- redmine (issue tracker, forum)
7.1.2 essential public services
- postfix (SMTP)
- repo HTTP access
- repo rsync access
7.1.3 essential private services
- nshd (all boxes will need this)
- tinc (at least one reliable box needs this)
7.1.4 essential internal services
- autobuilder (hackers=>parabola-keyring/nshd, blacklist=>your-freedom)
- duplicity (backups)
- pbot
- upstream repo importer
7.1.5 essential storage
- backups (duplicity cross-mirrored 60-100GB per mirror)
- git (1GB)
- packages (~80GB per arch - currently 3 arches - 2 more planned)
- ISOs (5-50 GB - depending how we slice them)
7.1.6 non-essential public services
- bazaar
- docecot (@parabola.nu IMAP)
- git HTTP access
- mumble (VOIP)
- prosody (XMPP)
7.1.7 notes
most of the "non-essential public services" are rarely or never used; and are disabled
the ones that are most tightly coupled are autobuilder<->git-storage, and postfix<->mailman - everything labeled as "essential" is probably best kept on the same box, and perhaps replicated on another for emergency use - all of those essentials were on winston, while the web and non-essentials were split between proton and winston
lukeshu explained to me that the separation between proton and winston was such that all clients of mysql were on one box and all clients of pg were on the other - when we migrated everything to winston, lukeshu was leery that mixing these could cause performance problems; so that is something to consider with regards to re-provisioning