Hacking:Servers/Winston

From ParabolaWiki
Jump to: navigation, search
This article or section is out of date.
Please help improve the wiki by updating the article and correcting mistakes.

winston.parabola.nu is A VPS server which is hosted for free by 1984, a Hosting Company in Iceland. 

$ uname -m
x86_64

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           4.4G         98M        2.2G        692K        2.0G        4.0G
Swap:          4.3G          0B        4.3G

$ df -h | grep ^/dev
/dev/vda3        94G  3.2G   86G   4% /
/dev/vda4       375G  5.7M  355G   1% /srv


Pardon the sloppy formatting (<code> instead of {{ic|), I wrote this in markdown and used pandoc to convert it to mediawiki.

1 configuration philosophy

Avoid editing files owned by a package if possible; insert things into new files. If a file must be edited, try to avoid having to change any lines; try to only add new lines.

Everything worth backing up should be in /etc, /srv, or /home.

2 infrastructure/management

2.1 base setup

Packages installed:

  • group:base except for linux-libre
  • linux-libre-lts
  • haveged
  • irqbalance

Files affected:

  • /etc/hostname : winston.parabola.nu
  • /etc/locale.conf : LANG=en_US.UTF-8 LC_COLLATE=C
  • /etc/locale.gen : en_US.UTF-8 UTF-8
  • /etc/machine-info : LOCATION=1984 Hosting Company, Iceland
  • /etc/vconsole.conf : KEYMAP=us CONSOLEMAP=8859-1 FONT_MAP=8859-1_to_uni
  • /etc/systemd/system/multi-user.target.wants/getty@tty1.service
  • /etc/systemd/system/multi-user.target.wants/haveged.service
  • /etc/systemd/system/multi-user.target.wants/irqbalance.service
2.1.1 bootloader

Packages installed:

  • grub

Files affected:

  • /etc/default/grub

The grub config has two overrides added to it: 

GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_TERMINAL_OUTPUT=console
2.1.2 filesystems

Packages installed:

  • systemd-swap

Files affected:

  • /etc/fstab
  • /etc/systemd-swap.conf
  • /etc/systemd/system/local-fs.target.wants/systemd-swap.service
  • /etc/systemd/system/multi-user.target.wants/remote-fs.target

TODO: document

fstab is set up similarly to proton.parabola.nu.

2.1.3 networking

Files affected:

  • /etc/udev/rules.d/80-net-setup-link.rules
  • /etc/netctl/eth0-static
  • /etc/systemd/system/netctl@eth0\x2dstatic.service (auto-generated by netctl enable)
  • /etc/systemd/system/multi-user.target.wants/netctl@eth0\x2dstatic.service

/etc/udev/rules.d/80-net-setup-link.rules is a symlink to /dev/null, which disables new-style predictable network interface names (enp0s3? ens3?), and causes it to fall back to the old-style names (eth0).

The netctl profile eth0-static just has the network information from the 1984 VPS control panel.

2.1.4 Pacman

Files affected:

  • /etc/pacman.conf : Enable [pcr]
  • /etc/pacman.d/mirrorlist : Use repo.parabola.nu directly
  • /etc/pacman.d/.gitignore : /gnupg/
2.1.5 timedate

Files affected:

  • /etc/localtime : -> /usr/share/zoneinfo/UTC
  • /etc/systemd/system/sysinit.target.wants/systemd-timesyncd.service

timesyncd is an NTP client daemon.

2.2 etckeeper

Packages installed:

  • pristine-etc-keeper

Files affected:

  • /etc/systemd/system/multi-user.target.wants/pristine-etc-keeper.timer
  • /etc/systemd/system/multi-user.target.wants/etckeeper.timer
  • /etc/.gitignore : append !/mtab /machine-id /group- /gshadow- /passwd- /shadow- /resolv.conf
  • /etc/udev/.gitignore : /hwdb.bin

etckeeper is configured to use git to keep track of changes in /etc. The systemd unit etckeeper.timer is enabled, which makes a commit (if necessary) daily. It will also run before and after pacman via libalpm hooks.

In addition to etckeeper keeping track of the current configuration, lukeshu's pristine-etc-keeper maintains a branch of what /etc would be like if we never made any changes from the default files. The clean version of etc is available as the clean remote's master branch. Doing a git diff clean/master master should be helpful in investigating how things have been configured.

2.3 users

See also: keys user and group

2.3.1 Parabola hackers

Packages installed:

  • parabola-hackers-nshd
  • openssh

Files affected:

  • /var/lib/hackers-git/
  • /etc/ssh/sshd_config
  • /etc/nsswitch.conf
  • /etc/pam.d/passwd
  • /etc/pam.d/system-auth
  • /etc/pam.d/system-login
  • /etc/systemd/system/sockets.target.wants/nshd.socket
  • /etc/systemd/system/dbus.service.wants/nshd.service (temporary systemd bug workaround)
  • /etc/nshd/shadow

sshd is configured to force the use of keys (no password-based login), and to use parabola-hackers ssh-list-authorized-keys in addition to checking ~/.ssh/authorized_keys. ssh-list-authorized-keys returns the authorized keys from the hackers.git checkout in /var/lib/hackers-git (the path to the checkout is configured in /etc/parabola-hackers.yml).

NSS and PAM have been configured to use the ldap modules that are part of nss-pam-ldapd. However, instead of running the normal nslcd LDAP client daemon, the system has ben configured to run the parabola-hackers-nshd nshd daemon, which reads user infomation from the same hackers.git checkout (configured the same way). This way we dn't have to worry about keeping /etc/passwd in sync with hackers.git. To this end, PAM has also been configured to create a users home directory when they log in if it doesn't already exist. Because hackers.git doesn't store any password information, nshd stores password hashes in /etc/nshd/shadow.

Sometimes after something fails in PAM, you get a "User not known to the underlying authentication module" message. Like, the pam_ldap.so failed because you typed your password wrong, but it acts like it failed because it didn't "own" the user. I think that it's just a bug in PAM's message selection. But (TODO) we should actually track it down.

2.3.2 emergency user

Files affected:

  • /etc/passwd
  • /etc/shadow
  • /etc/sudoers.d/99-emergency
  • /home/emergency/

In case anything should ever go wrong with hackers.git, the user "emergency" has been set up. "emergency" has authorized (statically via ~/.ssh/authorized_keys) the keys of serveral (not publicly disclosed) Parabola hackers. /etc/sudoers.d/99-emergency grants special privileges in case PAM or NSS get screwed up.

2.3.3 other

Files affected:

  • /etc/sudoers.d/00-wheel
  • /etc/systemd/system/shadow.service.d/sort.conf

/etc/sudoers.d/00-wheel gives sudo access to everyone in the wheel group.

The shadow.service has been extended to sort the files if they otherwise are ok. This makes dealing with pacman updates and such easier.

2.3.4 pbot

Files affected:

  • /etc/group
  • /etc/gshadow
  • /etc/passwd
  • /etc/shadow
  • /etc/systemd/system/pbot.service <-- this script is coppied from the pbot git repo
  • /etc/systemd/system/multi-user.target.wants/pbot.service <-- created by systemctl enable

2.4 SSH

Packages installed:

  • openssh

Files affected:

  • /etc/ssh/sshd_config
  • /etc/systemd/system/multi-user.target.wants/sshd.service
  • /etc/ssh/.gitignore: /ssh_host_*_key /ssh_host_*_key.pub

See above for how authentication and users are set up.

sshd is also configured to listen on both ports 22 and 1863. We may turn of port 22 in the future. Not using port 22 isn't security through obscurity, it is security through keeping-the-the-logs-useful-by-keeping-noise-down.

2.5 SSL

Packages installed:

  • certbot

Winston uses the certbot ACME client to get certificates from Let's Encrypt.

All domains handled by the server are shoved in as Subject Alternative Names in a single certificate. This makes configuring nginx easier.

2.5.1 keys user and group

Files affected:

  • /etc/passwd
  • /etc/shadow
  • /etc/group
  • /etc/gshadow
  • /etc/letsencrypt
  • /var/lib/letsencrypt
  • /var/log/letsencrypt

In order to run certbot as a non-root user, the keys user and group have been created: 

useradd --system --user-group --no-create-home --home-dir /etc/ssl --shell /usr/bin/nologin keys
chown -R keys:keys /etc/letsencrypt /var/log/letsencrypt /var/lib/letsencrypt
chmod 750 /etc/letsencrypt/archive /etc/letsencrypt/live

The associated keys group allows users to read the (private) keys in /etc/letsencrypt/live.

2.5.2 issuance, renewal, and installation

Files affected:

  • /etc/ssl/misc/certbot-get
  • /etc/ssl/misc/certbot-hook
  • /etc/sudoers.d/10-certbot
  • /etc/systemd/system/certbot-renew.service
  • /etc/systemd/system/certbot-renew.timer
  • /etc/systemd/system/timers.target.wants/certbot-renew.timer
  • /etc/nginx/snippets/ssl.conf

Unlike acmetool, certbot doesn't have an easy way of saying "please add this domain as a Subject Alternative Name". You have to re-run the same (long) command to get the cert, but with the domain added. So, I've encapsulated this into the script /etc/ssl/misc/certbot-get. Edit the array of domains at the top of the script, then run it.

Renewal, however, is very simple. It is handled by certbot-renew.service (triggered by the associated .timer). It runs certbot renew with a couple of flags.

Both certbot-get and certbot-renew.service prove ownership of the domain via the http-01 challenge. /etc/nginx/nginx.conf includes /etc/nginx/snippets/ssl.conf, which has a server{} block that handles ACME http-01 challenges.

Both certbot-get and certbot-renew.service have been written to run sudo /etc/ssl/misc/certbot-hook after certificates have been updated, and sudo has been configured to allow the keys user to do this without a password. Right now certbot-hook just runs systemctl reload nginx.service.

2.5.3 other

Files affected:

  • /etc/nginx/nginx.conf
  • /etc/nginx/snippets/ssl.conf
  • /etc/ssl/private/dhparam-2048.pem
  • /etc/ca-certificates/.gitignore : /trust-source/blacklist/ /trust-source/anchors/ /extracted/
  • /etc/ssl/.gitignore : /certs/*.0 /certs/*.1 /certs/*.pem /certs/ca-certificates.crt /certs/java/cacerts /.ssh/

nginx.conf includes snippets/ssl.conf, which is primarily based on the output of Mozilla Security's recommended web server configuration generator. It has had the main SSL information promoted to be directly into the http{} block, instead of having to be in each server{} block. The HTTP->HTTPS redirector has had an exception added to it to have it respond to ACME http-01 challenges.

Because certbot is only configured to use http-01 challenges, the all challenges happen over pain HTTP, which means that the configurations for each subdomain (which only serve over HTTPS/HTTP2) do not need to include anything about ACME or SSL (other than mentioning ssl in the listen directive).

ssl.conf needs to refer to a dhparam PEM file. This has been generated with the command 

openssl dhparam -out /etc/ssl/private/dhparam-2048.pem 2048

2.6 HTTP

Packages installed:

  • nginx

Files affected:

  • /etc/nginx/nginx.conf
  • /etc/nginx/fastcgi.conf, /etc/nginx/{fastcgi,scgi,uwsgi}_params
  • /etc/nginx/mime.types
  • /etc/nginx/snippets/ssl.conf
  • /etc/nginx/sites/
  • /etc/ssl/misc/certbot-hook
  • /etc/systemd/system/multi-user.target.wants/nginx.service

See the #SSL for how nginx has been configured to do HTTPS.

fastcgi.conf, fastcgi_params, scgi_params and uwsgi_params have been edited to pass 127.0.0.1 as the client IP address to worker processes, to protect user privacy.

The log_format has been customized to redact the client IP and referrer, again to protect user privacy. Unfortunately, the libre version of nginx does not allow for customizing the format of the error log, so user IPs may still end up in the error log.

mime.types has had xz, gzip, bzip2, tar, PGP (.sig), and bittorrent types added to it. This list was based on the unmatched file extensions that find turned up on repo.parabola.nu.

Per-subdomain configurations live in /etc/nginx/sites/*.conf (which is the exact wildcard used with include at the end of nginx.conf).

"Special" rules in sites/*.conf should have the alias- or meta- prefix. For example, right now there is only meta-unknown-domain.conf which is a fallback for when no other file handles a domain (it redirects to https://www.parabola.nu/404). Normal domain handlers should be named server-${domain}.conf

Every sites/server-*.conf file should be of the format (replace subdomain with the subdomain): 

/etc/nginx/sites/server-subdomain.parabola.nu.conf
# -*- Mode: nginx; nginx-indent-level: 8; intent-tabs-mode: t -*-

server {
    server_name subdomain.parabola.nu;
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    error_log  /var/log/nginx/nginx.http.subdomain.parabola.nu.error.log  error;
    access_log /var/log/nginx/nginx.http.subdomain.parabola.nu.access.log specific;

    ...
}

nginx.conf sets root /srv/http/$server_name, which means that most of the time root won't need to be set in each individual server{} block. Unfortunately, error_log and access_log don't support variable expansion, so they do need to be set each time.

The individual sites/server-*.conf files won't be documented in this section; see the section for the service that it provides.

TODO: document the rest of the settings in nginx.conf

2.6.1 process management

Packages installed:

  • uwsgi

Files affected:

  • /etc/systemd/system/uwsgi@.service.d/socket.conf
  • /etc/systemd/system/uwsgi@.socket.d/owner.conf
  • /etc/uwsgi/vassal.ini
  • /etc/systemd/system/sockets.target.wants/uwsgi@vassal.socket

Wherever possible, we should use uWSGI for process management of our HTTP services. This allows for much more consistent management and configuration than the hodge-podge of PHP-FPM, manage.py, fcgiwrap, et c. that we used to have on Proton.

uWSGI is the program, uwsgi is the protocol it speaks with nginx. A pool of workers is called a vassal, and is configured in /etc/uwsgi/vassal.ini, and activated by uwsgi@vassal.socket; a socket speaking the uwsgi protocol is created at /var/run/uwsgi/vassal.sock.

We use systemd socket activation rather than a uWSGI emperor because they provide largely the same functionality; the only real advantage that a uWSGI emperor would provide over systemd socket activation is if you ran it in tyrant mode, it lets you do some cool things with unpriveleged access, which would be useful for a shared web host. We aren't a shared web host, and have no reason to run emperor in tyrant mode.

Since the uwsgi@.service vassal unit is written to support socket-activated or non-socket-activated use, it is normally possible to accidentally start it without the associated .socket unit; which is an error with how our vassal configurations are written. To fix this, uwsgi@.service.d/socket.conf overrides the unit a bit to disable non-socket-activated use.

The ownership and permissions for the socket are configured in uwsgi@.socket.d/owner.conf, which sets the owner to http:http and the mode to 0600.

uWSGI supports thread pools in addition to process pools, but many of the actual workers you'll want to use aren't thread safe, so stick to process pools unless you specifically know that your worker is thread-safe (for example, PHP, at least with the modules needed for MediaWiki, is not thread-safe).

Individual vassal configurations are documented in the section for the service that they provide, not here.

2.7 email

Packages installed:

  • postfix

Files affected:

  • /etc/systemd/system/multi-user.target.wants/postfix.service
  • /etc/postfix/.gitignore : *.db

Configuration

  • /etc/postfix/main.cf edited to set make it accept, and locally deliver, mails to <user>@winston.parabola.nu.

Operation

  • pbot gets emails delivered qmail-style in $HOME/Maildir/ to allow him to report bug tracker changes
  • currently postfix aliases has not been set up. where should mails addressed to e.g. postmaster@winston.parabola.nu go? at the moment they get locally delivered to root user

3 services

You know, the things that the server is actually for.

3.1 System Information

Packages installed:

  • uwsgi-plugin-cgi

Files affected:

  • /srv/http/winston.parabola.nu/sysinfo.cgi
  • /etc/uwsgi/winston.ini
  • /etc/nginx/sites/server-winston.parabola.nu.conf
  • /etc/systemd/system/sockets.target.wants/uwsgi@winston.socket

Serves a simple system information page at https://winston.parabola.nu/. The source of the (very simple) script is at https://winston.parabola.nu/sysinfo.cgi.

3.2 ParabolaWeb

Packages installed:

  • uwsgi-plugin-python2
  • parabolaweb-utils
  • parabolaweb-dev

Files affected:

  • /etc/passwd
  • /etc/shadow
  • /srv/http/www.parabola.nu/

Winston isn't running ParabolaWeb (yet), but has it installed because other pages symlink to the branding files in it. 

useradd --comment ParabolaWeb --home-dir /srv/http/www.parabola.nu/ --create-home --system --gid http parabolaweb

3.3 Bazaar

Packages installed:

  • bzr

Files affected:

  • /etc/passwd
  • /etc/systemd/system/bzr.service
  • /etc/systemd/system/multi-user.target.wants/bzr.service
  • /srv/bzr/

TODO: IDK, someone else set this up.

3.4 Git

Packages installed:

  • git
  • cgit
  • uwsgi-plugin-cgi

Files affected:

  • /etc/passwd
  • /etc/sudoers.d/10-git
  • /etc/cron.spool/git
  • /srv/git/

The "git" user has been modified: 

usermod --comment "Parabola git" --shell /usr/bin/git-shell git

The git repositories live in /srv/git/ and are owned by the git user.

Metadata about all of the git repositories, as well as most of the setup, lives in the git-meta.git repository. The git user has a cron-job to update this repository every hour.

git is configured (core.hooksPath such that it looks in ~/.config/git/hooks/ for hooks, instead of /path/to/repo.git/hooks/. All of the server-side hooks exist: they all do the "same" thing: use run-parts to run all of the scripts in ~/.config/git/hooks/hook-name.d/, and then run the normal hook file /path/to/repo.git/hooks/hook-name.

Right now there are only two "real" hooks:

  • /srv/git/.config/git/hooks/post-receive.d/agefile : see #transport:_HTTPS
  • /srv/git/hackers.git/hooks/post-receive : update the hackers.git checkout (see #Parabola_hackers), then run sudo systemctl reload nshd.service to activate changes to system users (sudo has been configured to allow this to be done without a password). TODO: see the hook file itself for more details, but it should also activate the changes to email aliases with postfix, as well as rebuild the parabola-keyring package with autobuilder.
3.4.1 transport: git protocol

Files affected:

  • /etc/systemd/system/socket.target.wants/git-daemon.socket
3.4.2 transport: SSH

Files affected:

  • /srv/git/git-shell-commands/

This is the only method for which push is enabled. Only the git@ user may push; even though if you have access to git, you probably have access to ssh into yourname@winston, you can't push from that account. As far as git is concerned, all users are equal (TODO: perhaps we should implement some more granular access control. Safety rails and all that). Access to git@ is based on membership in the "git" group in hackers.git (see #Parabola_hackers).

If you need to set up custom git hooks, then ssh in to your user, and use sudo -u git to do it.

3.4.3 transport: HTTPS

Files affected:

  • /etc/uwsgi/git.ini
  • /etc/systemd/system/socket.target.wants/uwsgi@git.socket
  • /etc/nginx/sites/server-git.parabola.nu.conf
  • /srv/http/git.parabola.nu/
  • /srv/git/.config/git/hooks/post-receive.d/agefile

The git web interface is cgit, which is managed by uWSGI speaking uwsgi/modifier1=9, which is the variant of the uwsgi protocol for CGI requests.

uwsgi/git.ini sets CGIT_CONFIG=/srv/http/git.parabola.nu/cgitrc.

A post-receive hook that runs for all repositories updates the agefile, so that cgit has a better idea of when a repository was last updated.

The files for cgit are in /srv/http/git.parabola.nu/, which shadows /usr/share/webapps/cgit/: 

# mirror /usr/share/webapps/cgit/
cgit.cgi    -> /usr/share/webapps/cgit/cgit.cgi
cgit.css    -> /usr/share/webapps/cgit/cgit.css # customized
cgit.png    -> /usr/share/webapps/cgit/cgit.png
favicon.ico -> /usr/share/webapps/cgit/favicon.ico /srv/http/www.parabola.nu/web/sitestatic/favicon.ico
robots.txt  -> /usr/share/webapps/cgit/robots.txt
# added files
cgitrc
archnavbar/ -> /srv/http/www.parabola.nu/web/sitestatic/archnavbar/
header.html              # archnavbar
syntax-highlighting.sh   # based on some old version of /usr/lib/cgit/filters/syntax-highlighting.sh

3.5 pbot

pbot, the Parabola IRC bot, lives at pbot.git, and runs as the user pbot

Home dir at /home/pbot/.

  • This contains all the pbot files from the git repo. Not in a subdir, just right there.
  • Also contains dir logs/ TODO add to git repo
  • Also contains 600 permissions file passord.secret containing IRC password
  • pbot writes data to info/ and announcements/. these are put in a tarball every day by cron job so they can be downloaded elsewhere if wanted (e.g. for backup).

Deps installed with pacman

  • moreutils
  • w3m
  • recode
  • inotify-tools

3.6 Redmine

Packages installed:

  • ruby
  • ruby-bundler
  • rubygems
  • uwsgi-plugin-rack

Files affected:

  • /etc/nginx/sites/server-labs_parabola_nu.conf
  • /etc/systemd/system/labs-restart.socket
  • /etc/systemd/system/labs-restart.timer
  • /etc/systemd/system/multi-user.target.wants/uwsgi@labs.service
  • /etc/systemd/system/socket-user.target.wants/uwsgi@labs.socket
  • /etc/systemd/system/timers.target.wants/labs-restart.timer
  • /etc/systemd/system/uwsgi@labs.service.d/psql.conf
  • /etc/uwsgi/labs.ini

See also: DeveloperWiki:Web_Admin

3.7 Rsync

Packages installed:

  • rsync

Files affected:

  • /etc/rsyncd.conf
  • /etc/systemd/system/rsyncd.socket
  • /etc/systemd/system/sockets.target.wants/rsyncd.socket

Installed by coadde or Emulatorman during migration of repo.parabola.nu from Proton. I resumed configuration of the server and enabled it to accommodate mirrors after the DNS change. Port number (873 -> 875) and modules ( [repos]:/srv/repo/main and [abs]:/srv/abslibre) were copied verbatim from Proton.

Retrieved from "https://wiki.parabola.nu/index.php?title=Hacking:Servers/Winston&oldid=22472"