Systems - WEBDB

Basics

See also https://infradocs.cacert.org/critical/webdb.html. Currently very basic, but system documentation should move to that page...

Purpose

The webdb server provides the web front-end to the world for CAcert's certificate services. It also runs the database with all subscriber data. It communicates with the signer server for submission of certificate signing requests and retrieval of signed certificates, certificate revocation lists and signed PGP keys. This is one of CAcert's critical servers, and is operated under the requirements of the CAcert Security Policy and Security Manual.

Physical Location

The webdb server is located on physical machine webdb (alsp known as sun2) in the CAcert rack at BIT 2, Ede.

Physical Configuration

The webdb machine is a Sun X4100 server containing 4 AMD Opteron processors running at 2.4 GHz, 4 GB of RAM and 2 hot-swappable SAS disks of 146 GB.

See also SystemAdministration/EquipmentList

Connections

This system has four ethernet connections, of which currently three are used: one to the CAcert internal administrative network (172.28.50), one to the CAcert critical services network (172.16.3), one to the public internet. A special fifth ethernet connection is used to connect the ILO-based management interface of the server to the CAcert internal administrative network (172.28.50).

The keyboard, mouse and video monitor connections are normally not connected, but can optionally be connected when visiting the data center.

The system is also connected by a USB cable to the signer server. The USB cable is a USB-Link cable based on the Prolific Technology Inc. PL2501 chipset, which can be found in USB 2.0 NET Link Cable from EdNet. A custom-developed software module named CommModule is used to transfer controlled data over this link between the webdb server and the signer server.

Applicable Documentation

  1. CommModule - Documentation and Operator Manual (to be put online somewhere)

  2. Security Policy

  3. Security Manual

  4. Disk Encryption

  5. Disk Mirroring

  6. Drive Retirement

Administration

Critical System Administrator Team:

Services

Listening services

Protocol

Port

Remarks

HTTP

TCP/80

web server for main CAcert application

HTTPS

TCP/443

web server for main CAcert application in secure mode

RSYNC

TCP/873

rsync server for CRLs; only accessible by CAcert OCSP and CRL servers

SSH

TCP/22

only from two hosts on internal admin network; remote system maintenance

DNS

External names are:

Internal names are:

Connected Systems

Outbound network connections

Protocol

Port

Remarks

DNS

UDP/53 + TCP/53

DNS lookups by main CAcert application and utilities

SMTP

TCP/25

outgoing mail sent by main CAcert application

WHOIS

TCP/43

domain name lookup by main CAcert application

HTTP

TCP/80

web lookups, mainly for system updates

NTP

UDP/123

time synchronization with internet time servers

SYSLOG

UDP/514

only to logger.intra.cacert.org; for centralized system logs

boxbackup

TCP/2201

only to backup.intern.cacert.org; for on-line backups

SSH

TCP/22

only via hopper.intra.cacert.org to translations.cacert.org, for translation updates of main CAcert application

Security

Privileged Access: Critical System Administrators

Other Access: CAcert support engineers

Software installation

The base OS for the signing server is a minimum install of Debian 7.9 (Wheezy) During the installation, LUKS volume encryption must be activated, as described in Disk Encryption.

After performing the base install, the set of installed packages needs to be adjusted to conform to the guideline defined by the pkg-conform script, which can be found at pkg-conform.

The only custom software to be installed is the main CAcert web application. The source for this can be downloaded from http://www.cacert.org/src-lic.php. For setting up the chroot environment additional tools and configuration files are available from http://svn.cacert.org/CAcert/SystemAdministration/webdb as explained below in more detail.

Software Configuration

All configuration files should be kept under RCS control wherever possible, i.e. when you want to modify a pristine configuration file, do this: 

   # mkdir RCS
   # vi configfile
   # ci -u configfile

Next time you want to change the configuration file, do this: 

   # co -l configfile
   # vi configfile
   # ci -u configfile

Boxbackup

To configure boxbackup-client (assuming a UID of 0x50104 check with server admin): 

# bbackup-config /etc/boxbackup lazy 50104 SERVERNAME /var/lib/bbackupd /home /etc/ var

Send the resulting 50104-csr.pem to SERVERNAME administrator for signing and install two returned files. Then run 

# bbackupd

chroot environment for main CAcert web application

The main CAcert web application is written in PHP and served by an Apache web server running in a restricted chroot environment for security reasons. This environment must be setup under the directory /home/cacert, and contains a minimal subset of the standard Debian tools required for running the service, plus of course the application-specific parts.

To setup the environment, use the mkchrootenv script which is found at mkchrootenv. Read the first lines of the script to check whether the variables defined there (ROOT, SPEC, SSL_CACERT) need to be adjusted. In particular the SPEC variable requires some attention: this points to a directory which should contain a couple of files and software packages not easily found elsewhere. An initial cut of its content can be found at spec. The file spec/distro/www.tar.bz2 should be replaced by an up-to-date copy of the bzip2-compressed tar ball of the main CAcert application source code obtained from http://www.cacert.org/src-lic.php.

Running the mkchrootenv script will perform most of the necessary steps automatically, and even generate a set of fake certificates for a test server at the end (this requires some user interaction). When installing a production server, these fake certificates should obviously be replaced by the real ones.

Additional manual configuration

  1. /etc/default/commmodule: needs to be setup with definitions for COMMROOT and COMMROLE e.g.: 

      COMMROOT=/home/cacert/www/CommModule
  COMMROLE=client

  2. /etc/init.d/commmodule: needs to be copied from /home/cacert/www/CommModule/commmodule. After that, the init script should be enabled with: 

      /usr/sbin/update-rc.d commmodule defaults 90 20

    NOTE: the full effect of step 1 and 2 can simply be achieved by: 

      cd /home/cacert/www/CommModule; ./commmodule setup

  3. /etc/logrotate.d/apache2: needs to be adjusted with pathnames reflecting chroot for apache to /home/cacert. Use dateext to keep stable names for old logfiles. The following context diff shows the details: 

      @@ -1,8 +1,9 @@
  -/var/log/apache/*.log {
  +/home/cacert/var/log/apache2/*.log {
        weekly
        missingok
        rotate 52
        compress
  +     dateext
        delaycompress
        notifempty
        create 640 root adm

  4. /etc/postfix/main.cf: needs a couple of adjustments: 

      append_dot_mydomain = yes
  relayhost =
  # inet_interfaces = loopback-only (i.e. comment out or remove!)
  sender_canonical_maps = hash:/etc/postfix/sender_rewrite

    while etc/postfix/sender_rewrite should contain: 

      www-data returns@cacert.org
  root     returns@cacert.org

  5. /etc/init.d/apache2: needs to be adjusted for running the Apache webserver in a chroot environment. Reference svn.cacert.org/CAcert/SystemAdministration/webdb/apache2 for an example.

  6. /etc/init.d/networking: needs to be extended with a hook to invoke a custom firewall script, like this: 

      @@ -120,6 +120,8 @@
          process_options
          check_ifstate
  
  +       sh /root/firewall.sh            # enable CAcert custom firewall
  +
          if [ "$CONFIGURE_INTERFACES" = no ]
          then
              log_action_msg "Not configuring network interfaces, see /etc/default/networking"

    NOTE: it will probably be better to merge this into /etc/network/interfaces rather than modifying a standard system script.

  7. /etc/network/interfaces: needs to be configured with two additional IPv4 addresses for the equivalents of secure.cacert.org and tverify.cacert.org (in addition to www.cacert.org). For example: 

      # The external network interface
  allow-hotplug eth3
  iface eth3 inet static
       address 213.154.225.245
       netmask 255.255.255.224
       gateway 213.154.225.225
       network 213.154.225.224
       broadcast 213.154.225.255
       up   ip addr add 213.154.225.246/24 dev eth3 label eth3:1
       down ip addr del 213.154.225.246/24 dev eth3 label eth3:1
       up   ip addr add 213.154.225.247/24 dev eth3 label eth3:2
       down ip addr del 213.154.225.247/24 dev eth3 label eth3:2
 iface eth3 inet6 static
       address 2001:7b8:3:9c::245
       netmask 64
       gateway 2001:7b8:3:9c::2
       up   ip -f inet6 addr add 2001:7b8:3:9c::246/64 dev eth3
       down ip -f inet6 addr del 2001:7b8:3:9c::246/64 dev eth3
       up   ip -f inet6 addr add 2001:7b8:3:9c::247/64 dev eth3
       down ip -f inet6 addr del 2001:7b8:3:9c::247/64 dev eth3

  8. /etc/ssh/sshd_config: should have PermitRootLogin, PasswordAuthentication and UsePAM turned off, and have ListenAddress set explicitly to listen only on the internal admin network.

  9. /root/firewall.sh: is a custom firewall script for the webdb server.

  10. /home/cacert/etc/apache2/sites-available/cacert: will be generated automatically by the mkchrootenv script, but please check its contents before using it!

    Beside the obvious differences (different hostnames, IP adresses and certificates) the test servers have only one IP address available, so different secure hosts (cacert1 and secure1) have to be handled with SNI. The  SSLStrictSNIVHostCheck on  directives are implementing this.

  11. /home/cacert/www/includes/mysql.php: needs to be created as a copy of /home/cacert/www/includes/mysql.php.sample and edited to set the username and password for accessing the CAcert MySQL database, and set normalhostname, securehostname and tverify for a test system. For example: 

    @@ -16,15 +16,15 @@
     Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
 */
 
-       $_SESSION['mconn'] = mysql_connect("127.0.0.1", "username", "password");
+        $_SESSION['mconn'] = mysql_connect("127.0.0.1", "cacert", "klodder");
        if ($_SESSION['mconn'] != FALSE)
        {
-               mysql_select_db("database");
+               mysql_select_db("cacert");
                $_SESSION['mconn'] = TRUE;
        }
-       $_SESSION['_config']['normalhostname'] = "www.cacert.org";
-       $_SESSION['_config']['securehostname'] = "secure.cacert.org";
-       $_SESSION['_config']['tverify'] = "tverify.cacert.org";
+       $_SESSION['_config']['normalhostname'] = "cacert1.it-sls.de";
+       $_SESSION['_config']['securehostname'] = "secure1.it-sls.de";
+       $_SESSION['_config']['tverify'] = "tverify1.it-sls.de";
 
        function sendmail($to, $subject, $message, $from, $replyto = "", $toname = "", $fromname = "", $errorsto = "returns@cacert.org", $extra="")
        {

Disk mirroring

Disk mirroring must be setup as described in Disk Mirroring, to ensure that a usable replacement disk is available in case the primary disk fails. Failover is not automatic, and requires system administrator intervention.

Cron jobs

The following CAcert-specific cron jobs are used on the server, and setup in /etc/cron.d/cacert:

Common Tasks

Logfile Inspection

Full Backup

Log File Extraction

Password Changes

Planned Changes

The base OS needs to be upgraded to Debian Jessy (status "stable") before May 2016.

Changelog

All modifications to this system must be logged to the cacert-systemlog mailing list, which is primarily archived here.

Notes

Configuring normalhostname and securehostname in /home/cacert/www/includes/mysql.php

(Contributed by Markus Warg)

My "knowledge" about the two hostnames:

normalhostname is used for all regular operation on the http part of the page. normalhostname is even used for regular operation of all https parts of the page, if the user NOT logged in by certificate.

As you can see, the securehostname is linked under certificate login, so if the user wants to log in by crt, he is redirected to securehostname.

The code now seperates actions, if HTTP_HOST == securehostname and id=4, try to authenticate the user by crt, otherwise redirect to login page UNLESS he already logged in.

Both hosts point to the same document root, its just another virtual host to prevent the ssl renegotiation problem, I think. If you log in by crt, you will need to stay on securehost, because there is no session login, if you crt log in and manually switch from secure to www, you will be forced to log in.

If you log in and switch to securehost, you will be asked for an crt.

Further Documentations

CategorySystems

SystemAdministration/Systems/Webdb (last edited 2021-02-13 18:10:17 by BernhardFröhlich)