• Arbitrations
• a20120324.1

• Case Number: a20120324.1
• Status: closed
• Claimants: Eggert E
• Respondents: CAcert

• Case Manager: BernhardFröhlich

• Arbitrator: UlrichSchroeter

• Date of arbitration start: 2012-03-25
• Date of ruling: 2012-04-15
• Case closed: 2012-04-15
• Complaint: Valid certificate revoked?
• Relief: TBD

Before: Arbitrator UlrichSchroeter (A), Respondent: CAcert (R), Claimant: Eggert E (C), Case: a20120324.1

History Log

• 2012-03-24 (issue.c.o) case s20120324.6

• 2012-03-25 (A): added to wiki, request for CM / A
• 2012-03-25 (A): I'll take care about this case as (A)
• 2012-03-25 (A): appointing Ted as (CM) to this case
• 2012-03-25 (A): init mailing sent to (C)
• 2012-03-25 (C): accepts CCA/DRP under this arbitration
• 2012-03-26 (A): requesting detail infos about revoked certs from (C)
• 2012-03-26 (A): reqest from (Support) primary email of user account that is linked to given (C)'s email address
• 2012-03-26 (A): request to (Critical team) for logfile infos and special domaincerts record
• 2012-03-26 (C): response to (A)'s questions, one revocation did happen 2012-03-20 01:21:35
• 2012-03-26 (Support): [s20120326.5], sends primary email of (C)'s account

• 2012-03-27 (CriticalAdmin1): cannot give out logs, manual revocation does not log anything about the requesting user

• 2012-03-27 (CriticalAdmin2): discovered a manual domain delete request, that consequences was revoke server certs also

• 2012-03-27 (A): summarize findings to (Critical Team) and thanks for their assistance
• 2012-03-27 (A): request to (C) for connectivity infos
• 2012-03-27 (C): responded to request regarding connectivity infos
• 2012-03-27 (A): report from (Critical team) shows a domain deleted. What was the name of the domain deleted? execute sql query request to (Critical team)
• 2012-03-27 (A): question to (C): which of 3 listed domains is still active under (C)'s account?
• 2012-03-27 (C): 2 of 3 domains are still active, the 3rd is not in use anymore, I have deleted the domain via ISP
• 2012-03-27 (A): question to (C) regarding domain maintenance on domain #3 under (C)'s CAcert account
• 2012-03-27 (C): actions on domain in question were earlier (removal request to ISP back in Jan '12)
• 2012-03-27 (A): asking (C), to do a deep rethinking what did happen in the night of 2012-03-20
• 2012-03-27 (C): response to questions
• 2012-03-27 (Critical team): confirms that 3rd of 3 given domains was affected and summarize of findings
• 2012-03-27 (A): rephrasing summary and some details info about current findings to (Critical team)
• 2012-03-28 (A): sending long report of current findings to (C)
• 2012-04-15 (A): requesting a statement from (C) regarding the "long report"
• 2012-04-15 (C): yes please close the case

Original Dispute, Discovery (Private Part)

• Link to Arbitration case a20120324.1 (Private Part)

EOT Private Part

Discovery

• if this case relates to a system bug, system infos should be collected asap to analyze this issue.
• a related "strange" behavior was reported one day before (bug #1025): domain dispute with removal of certs that shouldn't be deleted. This case couldn't yet be reproduced in test cases on the testserver.
• both issues, reported bug #1025 and bug #1026, relates to a timeframe: 2012-03-19 - 2012-03-24
• if this current arbitration case is a side effect of reported bug #1025 (domain dispute), informations can be probably discovered from the webserver logs, tables disputedomain, domaincerts, domains, domlink
• Current Server certs serial number ranges (comparable cert serial numbers)

  • Root class1	~ 0A:68:6B
    Subroot Class3	~ 00:D6:A5

• the class1 server cert in question was revoked: 2012-03-20 01:21:35, expires 2013-01-28 16:16:19
• Potential procedures where a server cert gets revoked
  1. user triggers server cert revocation under server certs - view
  2. a domain dispute has been accepted
  3. (probably) an email dispute has been accepted
  4. builtin delete account procedure under the admin console (account can no longer be accessed)
• Certificate serial numbers may not be unique in the *certs tables of the webdb. One bug has been fixed regarding login to a wrong account. The reason was, that the Root or Class3 selection did not happen before the fix. Only serial number checks may lead in wrong results by find one serial number associated with Root cert and a 2nd identical serial number to be associated with the Class3 subroot cert.
• software code and functions (and related log entry)

  • 1.	GET /account.php?id=9	Domains - View	[20/Mar/2012:01:21:12 +0100]
    2.	POST /account.php	delete selected domain(s)	[20/Mar/2012:01:21:24 +0100]

• Server cert revocation was triggered by a delete domain action
  • analyze of affected server cert reveals a server cert with multiple SAN's (10) from 3 different domains
  • delete request of one domain affects this server cert in question, that also included other SAN's from other domains.
  • In first report from (Critical team), the domain id has been reported, but not the domain name
• sql query to display the domain name that has been deleted

  •     select domains.domain from domains where id=<domain-id in question>;

  • (C) reports that 2 of 3 domains are still active in (C)'s CAcert account domain list
  • (Critical team) reports, that the delete domain trigger affected domain #3, that is now missed in (C)'s domain list
• IP ranges
  • (C) reported the current IP of the dynamic ip provider
  • (Critical team) reported another IP, but from within the same IP range of the same service provider
  • so its more likely, that (C) has triggered the delete domain request by himself then an attack
  • one reason is, that the potential attacker only removed the domain, that is under delete domain procedure by (C)'s ISP. This was triggered back in January 2012 and, that the attacker did no other activities on (C)'s account: eg. resetting the password, adding other certs
• The function "delete domain" under the main website menu "Domains - View" only lists the added domains by the user, checkboxes besides each domain and a clickable button "Delete".
  • Clicking the "Delete" button once triggered only reports on a new page:

    • The following domains have been removed:
      (Any valid certificates will be revoked as well)
      domain.tld

  • no confirmation before starting the procedure, no big fat warning on page one, while removing a domain this automaticly also revokes all domain linked certificates. Especialy if certificates are configured by multiple SAN's where the direct domain link is unknown.
  • recommendations:
    1. add a big fat warning on the page where the user selects domains for deleting and pressing the "Delete" button -or-
    2. add a 2nd confirmation page into the workflow process with a sidenote, that also related certificates will be automaticly revoked once started -or-
    3. add a list of affected certificates (similar to Server certs - View page) to the confirmation page, so the user becomes a visible overview which of the certs are affected by the delete domain procedure. This becomes helpful especialy in cases, where the domain link is hidden by the certs individual SAN configuration

  • read also bug #823

Ruling

• According to SP 5 Incident Response, this running arbitration does not revealed any security breaches. All operations are identified as default actions triggered by a member using our software that results in the reported outcome.
• A deep analyze by our Critical team discovered following informations from different system logs:
  1. We have one identified revoked server certificate with serno 09:C1:2C
  2. The revocation date and time can be read from the server certs table. The action taken was around: 2012-03-20 01:21:35
  3. From the Webserver log: someone opened the Domain - View window (at the main CAcert website) in the middle of the night from 2012-03-19 to 2012-03-20 (close after midnight). 2 requests were logged

     • These 2 logged requests
        a. GET /account.php?id=9
       and
        b. POST /account.php
       reads in clear text:
       
        a. open Domain - View
        b. manualy iniated/triggered "delete selected domain(s)"
       
       From within the Domain - View form, there is only one POST request possible.
       And this is a "delete selected domain(s)" POST request.
       All other selections or actions have to show other command lines.
       
       The "active" trigger of the "delete selected domain(s)" POST request
       started the "delete selected domain(s)" process, including to revoke
       all related issued server certificates.

  4. The logged IP (dynamic IP range) from which this trigger was started was the same ISP provider IP range as (C) uses
  5. Related entries has been found in the mysql database logs by the critical team.
     • 2012-03-20 01:21:28 Revoking certs ... *.csr and *.crt
  6. The "magic" number in this case is 374323. Its the Id of affected server certificate that lists more then one Subject Alternative Name's (SAN's) so the revoked server certificate affects a multiple SAN cert that also includes the domain in question.
  7. As a domain revocation has been triggered, one of 3 potential domains that relates to the SAN's had been affected.

  8. By removal of one of the domains in question, the server cert that lists the affected domain gets revoked. The side effect on multiple SAN certificates is, that the remaining domains are also affected by the certificate revocation as the domain that has been revoked invalidates all server certs that has a link (SAN) to the domain in question.

  9. Claimant reported that he has sent a domain delete request to his ISP for the domain in question back in January 2012.
  10. Critical team executed the sql query based on following findings:
      • from the mysql log there is one line, that points to the domain that has been deleted.

      • this was: update domains set deleted=NOW() where id='192831'/*!*/;

      • select domains.domain from domains where id=192831; lists the domain in question that was also one of the domains in the list of multiple SAN's server cert
• Conclusion:
  • So the web log entry identified including the users used IP address that triggered the delete domain request links to the domain in question that has been deleted by the request.
  • The request was initiated from the main CAcert website under Domain - View by setting the checkbox in line for the domain in question and clicking the "Delete" box.
  • As this domain was one of several domains that was included in the server certificate with the id 374323 (with 10 SAN's and 3 domains listing), this server cert becomes invalidated and therefor was undergone an automated system revocation of this cert.
  • This is a default behavior in removing domains to also revoke all affected certificates. The impact in removing domains where server certs are involved with several different SAN's from several domains is the one we've discovered - revocation of a server certificate
  • 3 reasons, why the reported issue is probably no attack nor a hack:
    1. there might be a potential hack, but its very very unlikely, especialy the user works in the same IP range (dynamic ip provider) as the user who triggered the delete domain request
    2. its unlikely in an attack, that the hacker only removes domain in question and leaves all other domains untouched. and made no other certs actions (adding other certs), or doing a password change and so on
    3. (C) initiated a delete domain for the domain in question to his ISP back in January 2012, so the removal of the domain from the members account is a logical and forseen issue while leaving all other 2 domains that are linked to the cert in question.
  • There is no automatic procedure to revoke certificates based on whois checks in the background. Instead, there are reported bugs, to also doing checks on renewing certificates. The log files and the database records identifies this reported issue as a manual triggered delete domain issue. The relation is given only by the administrative process of delete a domain, here the domain in question. The actions needs to be done separately by the user who owns the domain:
    1. delete the domain at the ISP
    2. delete the domain linked to the CAcert account
  • So one situation is possible: The domain has been deleted by the ISP and the domain still remains in a users CAcert account.
  • But, in the current issue, the domain has been deleted from (C)s CAcert account.
  • Deleting a domain at the DNS provider doesn't automaticly kill the domain in a members CAcert account. This needs to be made manualy by the member itself, similar to keep the members primary email address in good working order. If the primary email address changes, the member has to change his primary email address on his account. So dns domain maintenance is another task by the user that isn't that defined in the CCA.
  • We don't have to interfer between deleting a domain by the dns registrar vs. deleting a domain in a CAcert account. The domain entered into a users CAcert account only validates the domain in the add domain process by pinging the whois entries by selection. An unregistered domain cannot be added to a users CAcert account.

  • As the domain delete of the domain in question did happen exactly 2012-03-20 01:21:35 its impossible that the domain delete for the domain in question has been triggered back in January 2012. Otherwise the server cert has been revoked back in January 2012. The delete domain process and the delete affected certs by this domain is a transactional process, once the domain is deleted, the database record will be set to date 1970-01-01 (otherwise) This signals the signer to revoke the certificate, update the CRL and the signer overwrites the date with the revocation in effect timestamp -> sql now(). This timestamp (C) finds in his server certs listing.

  • The delete domain request to the ISP was handled back around 2012-01-21 by (C)
  • Delete the domain is decoupled in the main CAcert website menues from the server certs view. So deleting a domain affects also server certificates that are linked with this domain in question. The opposite is the revocation of a single server cert. Therefor you have to open the Server certs - View and selecting a server cert and clicking the delete button. As the server certs list does not list the SAN's that are linked to different domains, there is no visible relation to other listed domains that might affected by a domain removal.

  • Also the circumstance that a member still can have an email address associated with his account for domain in question might increase the headache, as the email addresses and client certs linked with this email address aren't affected by the domain removal of a domain in question

  • As the web log, the sql log, the database entries all reports the same date and aediquate time - around 2012-03-20 0:30 the initial trigger has been set in this reported night.
  • The reported impact is not only possible, its by design as every removed domain also has all their related certificates to be revoked. In a server certificate, that has only one CN and SAN this is easy to understand. But working with certificates with multiple SAN's (10 in our case) and multiple domains (3 in our case), the impact is much more unforeseeable as it also affects other SANs and other domains.
  • So the delete domain trigger happened at Feb 20. This process cannot be rolled back. Once a cert is revoked, its revoked and nothing can recover this cert.
  • What (C) can do, if he has the private key of the certificate to create a new csr from this private key, and create a new cert (instead of renewing the old one). Renewing will be triggered under the CAcert account by selecting the cert in question and sending the renew request to the signer. If a cert is no longer listed ('cause it was revoked before) a new cert has to be created. Re-Creating a new cert is the next logical step. Except for the domain that has been gone ...
  • Most likely is that (C) can't remember exactly that he has triggered the delete domain process. Especialy the remove domain also triggered automaticly certificates revocations. As there is no sign of tamper by a third party, this case doesn't fall under the SP5 Incident response regime.
• A long report has been sent to (C). (C) accepted the report and requested to close the case.
• No further actions to take by arbitration.

Frankfurt/Main, 2012-04-15

Execution

• 2012-04-15 (A): ruling sent to (C), (Board), (Critical team). This case is now closed.

Similiar Cases

Arbitration cases

• a20111204.3

  Wrong DoB in account (includes sys tasks)

Bug Numbers

• bug #1026, (Server Certificate was revoked but not by the user)

  bug #1025, (Domain Dispute strange behaviour / Domain Dispute issue)

  bug #823, (No warning when removing e-mail address from account that certificates will be revoked)

• CategoryArbitration

• CategoryArbCaseSystemTasks

• CategoryArbCaseOthers