Systems - Signer

Basics

Purpose

The signing server signs appropriately formatted Certificate Signing Requests with one of CAcert's (sub)root keys. It also performs PGP signing of appropriately formatted PGP keys submitted by CAcert's users. This is one of CAcert's critical servers, and is operated under the requirements of the CAcert Security Policy and Security Manual.

Physical Location

The signing server is located on physical machine signer in the CAcert rack at BIT 2, Ede.

Physical Configuration

The signer machine has been replaced by a new server on September 11, 2009. The new server contains an Intel Dual Core Xeon E3113 processor (3.0 GHz, 6 MB, 1333 MHz FSB), 2 GB of RAM, and two hotplug SATA disks of 160 GB each. The machine broke in March 2020.

The signer machine has been replaced by a new server on May 4, 2020. The new server contains an Intel Quad Core Xeon X3323 processor (2.5 GHz, 6 MB, 1333 MHz FSB), 2 GB of RAM. The original drives of the old signer had been used.

See also SystemAdministration/EquipmentList

Connections

This system does not have any network connections. There is normally no keyboard, mouse or video monitor attached to it either. Only for maintenance purposes, a critical systems administrator may connect a crash-cart-based keyboard/mouse/video monitor to the system. The only connections of this system with the outside world is a serial cable and/or USB cable to the webdb server. The serial cable is a shielded and crossed RS-232 cable. The USB cable is a USB-Link cable based on the Prolific Technology Inc. PL2501 chipset, which can be found in USB 2.0 NET Link Cable from EdNet. A custom-developed software module named CommModule is used to transfer controlled data over this link between the signing server and the webdb server. As of September 11, 2009, the USB cable connection is used instead of the serial cable.

Applicable Documentation

  1. CommModule - Documentation and Operator Manual (to be put online somewhere). README of CommModule on CAcert SVN

  2. Security Policy

  3. Security Manual

  4. Disk Encryption

  5. Disk Mirroring

  6. Drive Retirement

  7. Full Backup Restore

  8. Password Management

  9. Installing subroot keys

Administration

Critical System Administrator Team:

Services

Listening services

Not applicable -- system has no network connections.

DNS

Not applicable -- system has no network connections.

Connected Systems

Outbound network connections

Not applicable -- system has no network connections.

Security

Privileged Access: Critical System Administrators, only when physically present after connecting crash-cart-based keyboard/mouse/video monitor.

Other Access: none.

Software installation

The base OS for the signing server is a minimum install of Debian 5.0 (Lenny). During the installation, LUKS volume encryption must be activated, as described in Disk Encryption.

The following packages need to be added after performing the base install:

To add a package, use: 

# aptitude install packagename

The following packages need to be removed after performing the base install:

To remove a package, use: 

# aptitude remove packagename_

NOTE: need to check the above lists again for Debian Lenny

The only custom software to be installed is the CommModule server side, which is currently not found in the CVS or SVN tree (NOTE: add a reference once it is there). This is installed in /root/CommModule.

Software Configuration

All configuration files should be kept under RCS control wherever possible, i.e. when you want to modify a pristine configuration file, do this: 

   # mkdir RCS
   # vi configfile
   # ci -u configfile

Next time you want to change the configuration file, do this: 

   # co -l configfile
   # vi configfile
   # ci -u configfile

X.509 signing configuration

Specific configuration files for the signing server are installed under the /etc/ssl directory. This directory contains a .cnf file for each combination of signing key and certificate type, and an associated subdirectory. The config file structure looks like: 

   [ CAcert original root ]
   /etc/ssl/openssl-client.cnf
   /etc/ssl/openssl-client-codesign.cnf
   /etc/ssl/openssl-client-org.cnf
   /etc/ssl/openssl-ocsp.cnf
   /etc/ssl/openssl-server.cnf
   /etc/ssl/openssl-server-org.cnf
   /etc/ssl/openssl-subca.cnf
   /etc/ssl/openssl-timestamp.cnf

   [ CAcert original class3 subroot ]
   /etc/ssl/class3-client.cnf
   /etc/ssl/class3-client-codesign.cnf
   /etc/ssl/class3-client-org.cnf
   /etc/ssl/class3-ocsp.cnf
   /etc/ssl/class3-server.cnf
   /etc/ssl/class3-server-org.cnf
   /etc/ssl/class3-subca.cnf
   /etc/ssl/class3-timestamp.cnf

   [ CAcert new class1 subroot ]
   /etc/ssl/root3/client.cnf
   /etc/ssl/root3/client-codesign.cnf
   /etc/ssl/root3/client-org.cnf
   /etc/ssl/root3/ocsp.cnf
   /etc/ssl/root3/server.cnf
   /etc/ssl/root3/server-org.cnf
   /etc/ssl/root3/subca.cnf
   /etc/ssl/root3/timestamp.cnf

   [ CAcert new class3 subroot ]
   /etc/ssl/root4/client.cnf
   /etc/ssl/root4/client-codesign.cnf
   /etc/ssl/root4/client-org.cnf
   /etc/ssl/root4/ocsp.cnf
   /etc/ssl/root4/server.cnf
   /etc/ssl/root4/server-org.cnf
   /etc/ssl/root4/subca.cnf
   /etc/ssl/root4/timestamp.cnf

There are also subdirectories here containing the signing key, signer certfificate, next serial to be used, certificate index and issued certificates for each main key: 

   [ CAcert original root ]
   /etc/ssl/CA/cacert.crt
   /etc/ssl/CA/cacert.pem
   /etc/ssl/CA/index.txt
   /etc/ssl/CA/index.txt.attr
   /etc/ssl/CA/serial
   /etc/ssl/CA/newcerts

   [ CAcert original class3 subroot ]
   /etc/ssl/class3/cacert.crt
   /etc/ssl/class3/cacert.pem
   /etc/ssl/class3/index.txt
   /etc/ssl/class3/index.txt.attr
   /etc/ssl/class3/serial
   /etc/ssl/class3/newcerts

   [ CAcert new class1 subroot ]
   /etc/ssl/root3/cacert.crt
   /etc/ssl/root3/cacert.pem
   /etc/ssl/root3/serial
   /etc/ssl/root3/index.txt
   /etc/ssl/root3/index.txt.attr
   /etc/ssl/root3/newcerts

   [ CAcert new class3 subroot ]
   /etc/ssl/root4/cacert.crt
   /etc/ssl/root4/cacert.pem
   /etc/ssl/root4/serial
   /etc/ssl/root4/index.txt
   /etc/ssl/root4/index.txt.attr
   /etc/ssl/root4/newcerts

The actual Certificate Revocation List for each signing key is kept in these files: 

   /root/CommModule/revoke-root0.crl  CAcert original root
   /root/CommModule/revoke-root1.crl  CAcert original class3 subroot
   /root/CommModule/revoke-root3.crl  CAcert new class1 subroot
   /root/CommModule/revoke-root4.crl  CAcert new class3 subroot

A delta file mechanism is used to communicate updates of the CRLs to the webdb server. The following additional files are used by this mechanism: 

   /root/CommModule/currentcrls
   /root/CommModule/currentcrls/[0134]/*
   /root/CommModule/delta[0134].diff

PGP signing configuration

Configuration files for PGP signing by the signing server are installed in the /root/CommModule directory. This directory contains the secret and public key rimgs for the CAcert PGP signing entity gpg@cacert.org: 

   /root/CommModule/secring0.gpg
   /root/CommModule/pubring0.gpg

These files should not be confused with the files in /root/.gnupg, which contains only the public key needed for creating pgp-encrypted system backups.

Disk mirroring

Disk mirroring must be setup as described in Disk Mirroring, to ensure that a usable replacement disk is available in case the primary disk fails. Failover is not automatic, and requires system administrator intervention.

Cron jobs

The following cron jobs are used on the server:

Migration to new hardware

When migrating the server to new hardware, the following steps are needed:

  1. verify reliable operation of new hardware

  2. install base OS on new hardware as described in Software Installation above; do not forget to use new passwords for the encrypted file systems!

  3. stop operation of signing server on old hardware
  4. transfer the following directories from old hardware to new hardware:
    • /root/CommModule
    • /etc/ssl
    This should preferably by done by connecting disks of old hardware to new hardware. If it is necessary to copy the data to an intermediate medium, e.g. USB stick, the medium must be destroyed securely afterwards.
  5. ensure that the required cron jobs are enabled

  6. ensure that the startup script for the CommModule is enabled

  7. put new hardware in operation

  8. decommission disks of old hardware as documented in Drive Retirement

  9. schedule a new off-site backup of the signing server as documented in Full Backup Restore

Common Tasks

Time Synchronisation

The signing server is not attached to the Internet and thus has no way to verify its time externally. But the webdb server has an external time synchronisation mechanism. Therefore a side task of the CommModule software is to transport current time information from the webdb server to the signing server on every cycle. It stores this time information in a shell script /root/CommModule/timesync.sh on the signing server. The signing server should be inspected at least once every three months, and time synchronisation must be effected manually by the system administrator like this:

  1. inspect whether /root/CommModule/timesync.sh is updated regularly
  2. inspect whether /root/CommModule/timesync.sh contains correct time
  3. issue time synchronization command like this 
    •      # date; sh /root/CommModule/timesync.sh; date
  4. record the amount of time correction applied and report this

Note that the time transmission protocol currently only works when there is no queue of outstanding signing requests. The implication of this is that before running the timesync.sh script, e.g. after restarting the signing server, one should wait for the queue of outstanding signing requests to be fully processed, and only after that the timesync.sh file can be relied upon. Hence steps 1 and 2 in the above recipe are mandatory!

Full Backup

Since this system does not have any suitable connections for taking automatic on-line backups, only full offsite backups are kept. The procedure for this is described in Full Backup Restore.

Log File Extraction

TBD

Password Changes

Please check Password Management.

Changelog

All modifications to this system must be logged to the cacert-systemlog mailing list, which is primarily archived here.

Changes Needed

Critical system administration has identified a number of performance / scaling issues in the current implementation which will need to be addressed in the medium term.

Unbounded directory growth

The following directories are growing linearly with the number of certificates issued in each category, since every issued certificate is saved in there:

directory

purpose

size in June 2009

data in June 2009

/etc/ssl/CA/newcerts

certs issued from original root

7 MB

2 GB

/etc/ssl/class3/newcerts

certs issued from original class3 subroot

1 MB

0.26 GB

/etc/ssl/root3/newcerts

certs issued from new class1 subroot

-

-

/etc/ssl/root4/newcerts

certs issued from new class3 subroot

-

-

Incomplete cleanup of work directories

For each signing request, a new work directory is created, whose name is based on an incrementing counter:

where NNN = counter / 1000, and MMM = counter % 1000. On completion of the request, an unlink is done of the work directory. This should be an rmdir instead, and some checking might be necessary to verify whether the work directory is empty at the end of the work cycle. In June 2009, we found about 115.000 unused work directories lying around on the server.

Critics

BerndEckenfels: First of all, sorry for using this official page, but since Moin has no discussion pages, and I dont really know where to discuss this, let me add some points:

Bernd: I propose to trust the webdb host less and minimize attack surfaces on the system:

  1. For example the timesync should work differently: the signing server maintains it's own clock and compares it to the WebDB's timestamp in the CommServer protocol. Once the difference gets too big, it refuses to work (and requires adjustment of the clock). It may adjust the time if smaller drifts are present, but I think it is better to not invest the time to make this work. Especially since stepping time while signing is very critical. Maybe it is better to have an external Timestamping Service generate singned timestamps in the CSR packets from webdb to signhost?

    wytze: the above seems to assume that the signing server has more reliable time than the webdb server -- this is normally not the case, the signing server has only its internal clock (which drifts over time), while the webdb server is synchronized through NTP. The only exception would be when the webdb server was attacked to manipulate its time. In the latter case the signing server would be undisturbed, continuing to use its (somewhat drifting) time. With the manual time synchronisation as documented there is no problem -- the sysadmin will check validity and changes are only applied at quiet moments. Stopping the time server automatically on detecting a time difference is NOT an option: it would cause many outages at unpredictable moments and require many expensive visits to the data centre by at least two people.

    • Bernd: well, I guess the policy should define how critical time skew in the signature is. But yes I agree having a manual sync process is fine. What is contained in timesync, is it using adjtime file?

  2. The documentation is not really complete, so forgive me when I make falso asumptions, but I think it is a absolute critical violation to run the CommModule or any of the maintenance jobs as root.

    wytze: Actually, this is not an issue at all on the signing server, because it is totally isolated (with the exception of the serial link). The only real secrets on the signing server are the private keys, and they need to be readable by the CommModule -- whether it does that running as root or otherwise is immaterial (and one could even argue that running as root reduces the attack surfacce.

    • Bernd: from my point of few the CommModule does not need to modify system files, and it also does not need to read the most valuable asset on that server. I would run the communication handler as an unpriveledged user and do the signing in a totally different user context. This can be done by a file system or fifo based interface between signer loop and comm module handler. Least priveldge principle.

  3. You should publish a list of actual installed packages, only.

    wytze: this is a work in progress ... once we have completed the installation of the new signing server, the list of package which 'probably should not be installed' can go away. Please take into account that there is virtually no room for experimentation here -- the signing server is locked up in the data centre and can only be inspected or modified at great cost.

  4. How are CRL entries generated? (I.e. why do they flow from sign to webdb and not the other way around?)

    wytze: When you want to revoke a certificate, the webdb server will submit the revoke request to the signer, which will update the CRL with a new entry and sign the CRL. Obviously that signature can only be made by the signer. This is the way openssl works, I don't know whether that could be done differently as you suggest.

    • Bernd: ok thats true. One possible thing would be to only sent hashes to the sign server, but I guess its cleaner to have a full openssl sign operation on this server.

  5. why dont you publish the tripwire/integrit Hash database monthly for external review

    wytze: First of all: we are not running tripwire or integrit at this moment, but I agree that it is probably a good idea to do so. However, we would not be able to extract the database monthly except by going to the data centre each month and do it manually. That is too expensive. On the other hand, we are looking into a way to get syslog information out of the signer server channeled into a central logging server over a serial cable; this would allow us to get some integrity information out of the signer, and publish it automatically in some fashion.

    • Bernd: maybe on a bi-monthly schedule or something. I learned it is a good method to catch uncontroled emergency fixes and accidential policy violations.

CategorySystems

SystemAdministration/Systems/Signer (last edited 2020-05-10 10:18:13 by dirkastrath)