= Incident Reports = This wiki page contains security incident reports in chronological order. <> == 1. 20140407 Heartbleed OpenSSL. == * [[Arbitrations/a20140408.1|Arbitration a20140408.1]] * [[https://blog.cacert.org/2014/04/openssl-heartbleed-bug/|blog post]] including English, German version. * [[https://lists.cacert.org/wws/arc/cacert-board/2014-04/msg00011.html|report to board]] * [[https://bugs.cacert.org/view.php?id=1265|bug-1265]] bugtracker entry for mail-script * [[https://blog.cacert.org/wp-content/uploads/2014/04/CAcert-PM-Heartbleed-en.pdf|CAcert-PM-Heartbleed-EN.pdf]] press release English === Overview of actions === || 2014-04-07 23:19 CEST || -1:01h || iang reports Heartbleed bug on cacert-sysadm mailing list <
> https://lists.cacert.org/wws/arc/cacert-sysadm/2014-04/msg00001.html <
> || 2014-04-08 0:10 CEST || -0:10h || CAcert critical team checks OpenSSL version on webdb and signer and confirms that these critical machines are '''not''' vulnerable <
> https://lists.cacert.org/wws/arc/cacert-sysadm/2014-04/msg00002.html <
> || 2014-04-08 0:20 CEST || +0:00h || golem reports about bug <
> || 2014-04-08 0:36 CEST || +0:16h || CAcert personnel gets aware that something was up with OpenSSL <
> || 2014-04-08 3:30 CEST || +3:10h || start to seriously discuss the issue <
> || 2014-04-08 3:50 CEST || +3:30h || As it was already checked that the root key could not have been affected and the central systems were safe as well, it was agreed that a double tracked strategy should be followed, as two things were likewise important: <
> a. recovery strategy for CAcert systems <
> * investigate what infrastructure systems were affected <
> * update them <
> * get new keys for them <
> a. informing our members <
> * to prepare information about root keys being safe, but member probably were likely to be affected, anyway, and what the status on our systems is <
> * via blog post <
> * via mail to all members (this was later narrowed down) <
> * for this the authorization of an Arbitrator was needed <
> * maybe other information strategies || 2014-04-08 8:50 || +8:30h || At this time already was done: * risks analysed * most relevant systems fixed (and upgraded certs) * blog post sent * tweet sent * main cacert mailinglist informed * members of the following teams had been involved: * infrastructure, arbitration, critical, support, software * the following teams were informed (enough to be able to react) * infrastructure, critical, board, audit, pr, support, software * (correct documented) arbitration ruling to inform members per automated mail || 2014-04-08 10:50 || +10:30h || CAcert critical team reports detailed versions of OpenSSL for all CAcert critical machines and confirms that they are '''not''' vulnerable <
> https://lists.cacert.org/wws/arc/cacert-sysadm/2014-04/msg00004.html <
> || 2014-04-09 10:45 || +34:25h || At this time was already done * only two non-critical systems fixed waiting for new cert * automated mail send to members who either have subscribed "general announcements" or had active keys since the bug was introduced This was mostly done by a handful of people - during their free time (at night). === Timeline === (all times of when systems were fixed should be added) || 2014-04-07 23:19 || bug reported on cacert-sysadm by iang || || 2014-04-08 00:10 || critical team confirms that webdb and signer are '''not''' vulnerable || || 2014-04-08 00:20 || bug is announced by golem || || 2014-04-08 00:36 || CAcert personnel gets aware that something was up with OpenSSL || || 2014-04-08 ??? || persons starte to take it as something possibly serious, and to look at "their" systems || || 2014-04-08 3:30 || VoIP session initiated to discuss the issue. || || 2014-04-08 3:50 || decisions on reaction strategy || || 2014-04-08 ??? || contact with critical, verification about status of central systems || || 2014-04-08 4:10 || dispute send to arbitration || || 2014-04-08 5:00 || cacert.eu fixed, but no new cert || || 2014-04-08 6:05 || arbitration case a20140408.1 (previously transfered to Wiki) picked up by A and started || || 2014-04-08 7:00 || internal auditor is informed || || 2014-04-08 7:25 || ruling in arbitration case to inform members with possibly affected server certificates or had the "general announcement flag" set via scripted mail (goes to software, critical, board, support) || || 2014-04-08 7:45 || support team provided with more details || || 2014-04-08 8:00 || mail to PR team lead || || 2014-04-08 8:10 || blog post is send || || 2014-04-08 8:10 || tweet sent || || 2014-04-08 8:20 || infrastructure team for systems not covered by present persons are informed and asked to also upgrade their systems || || 2014-04-08 8:45 || mail an cacert@lists.cacert.org || || 2014-04-08 10:20 || cacert-de@lists.cacert.org informed with german post || || 2014-04-08 10:45 || status report to open board list || || 2014-04-08 10:50 || critical team confirms that '''none''' of critical systems are vulnerable || || 2014-04-08 17:30 || bug 1265 for scripted mails to members || || 2014-04-08 17:45 || coding started || || 2014-04-09 5:30 || first review for script, transfered to testsystem || || 2014-04-09 5:35 || script started on testsystem || || 2014-04-09 6:45 || press release send out || || 2014-04-09 8:40 || first test report for script (was updated later) || || 2014-04-09 9:35 || second review for script || || 2014-04-09 9:35 || second test for script || || 2014-04-09 9:45 || changes to text approved by CM of a20140408.1 || || 2014-04-09 10:00 || mail from software to critical to execute script referencing a20140408.1 || || 2014-04-09 10:45 || A of a20140408.1 approves execution request (with reference to ruling) || || 2014-04-09 10:45 || script executed on production system, mails are going out || || 2014-04-09 21:00 || cacert.eu receives new cert || == 2. 20140828 CAcert website showing incomplete names == * [[https://lists.cacert.org/wws/arc/cacert-systemlog/2014-08/msg00019.html]] * [[https://lists.cacert.org/wws/arc/cacert-systemlog/2014-08/msg00021.html]] * [[https://bugs.cacert.org/view.php?id=1301]] === Overview of actions === ||2014-08-28 17:02:53 CEST|| CAcert critical sysadmin team completes upgrade of CAcert chroot application environment to Debian Wheezy on the main webserver, after having installed the patch for [[https://bugs.cacert.org/view.php?id=1297]], deemed to be the last blocking issue for this upgrade. The webserver is started in the new upgraded environment, but a backup of the old environment is kept online just in case. ||2014-08-28 18:10:01 CEST|| A well-known CAcert user sends e-mail to support@cacert.org, reporting that the last name and username in the account of this user are showing up as empty. The user is requesting clarification, and is hinting that the problem might have to do with the recent change by the critical sysadmin team. ||2014-08-28 around 20:00 CEST|| Support@cacert.org opens case s20140828.90 - Last name gone ||2014-08-28 20:19 CEST|| Support@cacert.org tries to phone critical sysadmin teamleader on his mobile phone, but gets no response. A voice mail is left. (Unfortunately, no attempt was made to phone the critical sysadmin teamleader on his landline -- which would have resulted in a response in this case --, or to phone another critical sysadmin team member). ||2014-08-28 20:30 CEST|| Support @cacert.org sends e-mail to critical-admin@cacert.org stating "we do have a problem with the database. All entries with special characters are no visible.", and including a copy of the e-mail of the original problem reporter. ||2014-08-29 08:41 CEST|| Critical sysadmin teamleader checks mobile phone voice mail, and starts investigating the reported problem. His initial suspicion is that some php5 environment settings concerning character encoding may have changed between the old and new CAcert chroot application environment. After some tests on the cacert2 test server (which has a copy of the production software environment), this turns out not quite to be the case. But further investigation reveals that the php5 function htlmentities() has different behaviour between PHP 5.3 (used in the old environment) and PHP 5.4 (used in the new environment) when its "encoding" parameter is left unspecified. This function is used by the CAcert application function sanitizeHTML(). It turns out that invoking this function with a string containing non-ascii characters results in an empty string as the return value. ||2014-08-29 09:46:41 CEST|| Critical team shuts down the main webserver, and re-activates the old CAcert chroot application environment which was a kept as a backup after the upgrade on 2014-08-28. ||2014-08-28 09:46:54 CEST|| Main webserver is running again with Debian Squeeze-based CAcert chroot application environment. The "dropping names" issue is resolved. ||2014-08-29 10:11|| Critical team reports [[https://bugs.cacert.org/view.php?id=1301|bug #1301]] to software development, including a detailed analysis of the problem, and including a suggested bug fix (i.e. explicitly specifying the "decoding" parameter for htmlentities()). ||2014-08-29 later that day|| Problem is picked up by software development. Further progress can be monitored in [[https://bugs.cacert.org/view.php?id=1301]]. === Impact analysis === From 2014-08-28 17:02:53 CEST until 2014-08-29 09:41 CEST the production server has been running with a software configuration causing incomplete names to be displayed to website users. Two groups of users were affected by this: 1. users with non-ascii characters in one or more of their name fields 2. users trying to assure a user in group #1 For the first group of users we can state that the only damage done will have been some confusion for the user -- suddenly seeing (part of) his/her name dropped makes a bad impression. However it cannot result in bad values being recorded in the CAcert database. Note that only the '''display''' of these names was affected, the name values used internally were not affected at all. For the second group of users, the issue can be more severe: an assurer may have been presented with an incomplete name of the user which he/she was attempting to assure. In the worst case this may have caused an assurer to refuse to perform such an assurance. We have investigated how many assurances have actually been recorded in the system in the vulnerable time period. This resulted in the identification of some 11 assurances. For these assurances the true names have been checked. Fortunately, none of these true names contained any non-ascii characters, so they were not affected by the bug. Summarizing we think that we can conclude that the integrity of the CAcert database, and particularly the assurance part, has not been damaged by this incident.