- Case Number: a20141118.1
- Status: running
- Claimant: Benny B as Software Assessor
- Respondent: CAcert
initial Case Manager: EvaStöwe
Case Manager: MartinGummi
Arbitrator: EvaStöwe
- Date of arbitration start: 2014-11-18
- Date of ruling: 201Y-MM-DD
- Case closed: 201Y-MM-DD
- Complaint: Investigation on bug 1339
- Relief: TBD
Before: Arbitrator EvaStöwe (A), Respondent: CAcert (R), Claimant: Benny B as Software Assessor (C), Case: a20141118.1
History Log
- 2014-11-18 (issue.c.o): case [s20141118.123]
- 2014-11-18 (iCM): added to wiki, request for CM / A
- 2014-11-18 (iCM): sent notification about case to C
2014-11-18 (A): I'll take care about this case as A and MartinGummi will be CM
- 2014-11-18 (A): send init mail to C
- 2014-11-18 (A): short chat with internal Auditor, that at first glance no "follow-up actions as necessary for audit" can be seen (as requested in the dispute)
- 2014-11-18 (C): mail to iCM with more details
- 2014-11-19 (A): send questions and requests for sql-queries to C
- 2014-11-19 (C): provided first set of answers and first query
- 2014-11-19 (a member): a draft for a blog-post by this member and C proposed to A
- 2014-11-19 (A): informs the member, that she will not answer directly, because A has first to read answers from C, but post should be soon
- 2014-11-19 (A): discussed some points of the answers with C per voice-interface, later CM joined
- 2014-11-20 (A): minor change to blog-post proposal, agreed that it could be posted
- 2014-11-21 blog post was posted
- 2015-01-17 (A): reminds C about outstanding queries [deadline: 2015-01-31, passed without response]
- 2015-02-08 (A): asks all active SAs to provide the needed queries [deadline: 2015-02-22, passed without response]
- 2015-03-01 (C): tells A to get As "priorities straight" regarding As cases in the context of another case
- 2015-03-03 (A): orders the SAs to: provide the needed queries until 2015-03-17; weekly report about outstanding open related critical bug 1341; else A has to consider a fee, makes C responsible for answers, as there is no TL at this time
- 2015-03-07 (SA1): complaints about harsh tone of A, does not know anything about the issue, so cannot give a time estimate, does not agree that SAs are responsible for queries in such cases, other people could do this; is confused about responsibility of C and rest of the team regarding required reports
- 2015-03-07 (A): explains harsh tone; C had reported that all SAs were informed about issue a long time ago; tells SAs to coordinate who dives into the subject; posts SP 7.2 and points to SP 5.4 - SA team is responsible for fixing bug and investigation; even if other people could help, C had requested to handle this case private until bug 1341 is fixed; SA team has to organise themselves how to answer
- 2015-03-07 (SA1): surprised by SP; sees A excused for As mail; annouces this to be probably SA1s last job as SA
- 2015-03-08 (A): asks critical team to execute the one provided query, even if there are no reviews
- 2015-03-08 (A): explains SA team some more to organise themselves; informs SA team about execution fo query, without further reviews
- 2015-03-08 (Crit): executed query, provides results
- 2015-03-08 (A): thanks Crit
- 2015-03-08 (A): informs SA team about result of first query; proposes query to inform affected members (basic version of one part of what SA team was asked to provide)
- 2015-03-08 (SA2): not sure if big part of queries can be done, suggests investigating bin-logs; ok for query from A, provides alternative query
- 2015-03-09 (Crit): informs A about own investigations, 4 weeks with greatly increased activity on login pages identified
- 2015-03-09 (A): answers SA2: not sure if requested queries are possible, but it seemed possible bases on answers from C; A would prefer own query (deleted accounts)
- 2015-03-09 (A): tries to contact C about findings from critical team, via direct channels (voice, chat) without result
- 2015-03-09 (A): phone call with SA2 about findings of critical team, SA2 proposes to create a script for bin-log investigations
- 2015-03-09 (A): informs Crit about possible bin-log investigation, asks critical team to prepare bin-logs and to not deleted them until further notice by A
- 2015-03-10 (Crit): agrees that bin-log is only way, even if it is a challange; bin-logs were secured
- 2015-03-10 (A): informs SA-team about findings from Crit, and current status; got no status report about 1341 so far
- 2015-03-10 (C): agrees that binlog analysis will be most relevant at the momtent, but logs should also be investigated, further, there should also be investigations for bug 1341; a second SA is asked to review patch for bug 1341, afterwards it should be tested
- 2015-03-11 (C): 2nd query proposed by A not ok, provided alternative
- 2015-03-13 (Crit): urgent security patch for bug 1341 was applied
- 2015-03-15 (A): asks board for agenda item, about this case and a20140422.1, and to have this private becasue of security or privacy issues
- 2015-03-15 (A): private discussion with two board members (private voice channel)
- 2015-03-17 (A): status report about a20141118.1 and a20140422.1 to board (private)
- 2015-03-22 (A): call with internal Auditor because of same issues, that were discussed with board
- 2015-03-22 (SA2): proposes script 3 for binlog analysis
Private Part
Link to Arbitration case a20141118.1 (Private Part), Access for (CM) + (A) only
EOT Private Part
original Dispute
> Dear arbitration, > > in the course of handling a support case on the public mailing > list[1][2] a source review has been triggered. In the course of > reviewing the source several more or less critical issues have been > found. Where every one of those issues did not pose a high risk on > their own; It was the combination of all the issues that led to the > decision to remove the affected feature from the software completely > as discussed in [3]. > > In order to check the feasibility of one particular issue of the set a > notice was sent to [a member] to implement a PoC on the test system > environment and prepare precautions in Gigi & Cassiopeia to mitigate > similar issues should they be applicable. Commits (especially to Gigi) > containing changes that might potentially disclose the nature of the > issues being fixed by 1339 were to be kept under embargo until they > are fixed on the production system. > > A limited number of selected people besides me had prior knowledge of > the issue > > I therefore seek an investigation of this issue to answer the > following questions: > > 1. How many users are affected by the removal of the feature? > > 2. Should a mailing be sent (affected users, other group?) to notify > on this issue? What content should it include? > > 3. Discuss the contents of a public announcement on the blog. > > 4. Discuss how this issue might have been used and symptoms that > indicate it being used? > > 5. Discuss parts of our policy regarding the precautions to avoid > future incidents of this kind? Especially are there requirements in > place to require a certain strength for the authentication? > > 6. Discuss further investigative steps to take for analysis of this > problem. > > 7. Discuss possibly related issues related to the one found. > > 8. Discuss follow-up actions as necessary for audit. > > 9. Investigate further aspects related to the issue at hand. > > More details will follow in encrypted form once an arbitration number > has been allocated.
Discovery
about bugs 1339, 1341
- bug 1339 allowed to "easily" log into foreign accounts when OTP was used on that account
- bug 1341 patched that there was no limit at all about how many login tries one can make for any account in any given time
- this bugs were there since OTP was introduced (long before the GIT was set up), the code seems to have a lot of issues, not only the critical ones
relevant dates
- 2014-11-14:
- bug 1339 regarding the One Time Password (OTP) feature and related bug 1341 (no limit for login tries for an account) was discovered based on a mail on the support mailing list
- 2014-11-18:
- bug 1339 was closed by removing the OTP freature
- dispute (regarding 1339) was filed
- private handling of the case was requested by C as there was an open related bug 1341
- 2014-11-19:
- A asked C a set of questions, including a request for some queries to search for exploits and to inform members who used OTP
- most of the questions were answered by C but only 1st query was provided, with a promise to provide others, soon
- a patch for bug 1341 was created by a member and reviewed by C (not moved to testserver)
- 2014-11-21:
- Blog post was posted
- 2015-03-08:
- 1st query was executed without further reviews
- 2nd query proposed by A
- 2015-03-09:
- critical team informed A about an investigation of logfiles, which revealed that there were at least 4 weeks with drastically increased activity on the login pages, which may be indicators for exploits of one of the bugs, at least in two of those weeks the activity could be relayed to specific pool of hosts.
- SA2 proposed to write a script to analyse the binlogs
- 2015-03-13:
- bug 1341 was patched
- 2015-03-23:
- SA2 provides script for binlog analysis
queries and scripts
1st query: number of affected users
SELECT COUNT(*) FROM users WHERE otphash != '';
- Result: 184
- It was created by (C). Even as the other SAs did not seem to be able to provide a review, (A) decided to let it be executed.
- (A) feels confident enough to understand that this query will not change anything in the DB, so that it's execution is not a security issue.
- The query does not violate the privacy of our members, as it is just a counter.
- It also provides information that is needed to answer the first question asked in the dispute, which is a sensible question in the context of bug 1339.
2nd query: provides needed data to infrom affected members via mail
- current version, proposed by C:
SELECT users.id, users.fname, users.lname, users.email FROM users WHERE `otphash` != '' AND `deleted` = '0000-00-00 00:00:00';
This query is not executed. It currently does not have another review and there also needs to be a mail-text created.
queries executed by critical team on logfiles
- Critical team:
Based on the pattern we have noticed on the test server to provoke the bug #1339 issue, we have concluded that webserver accesses starting with "/index.php?id=4" are possibly suspect, in particular when issued repeatedly in a very short time. Based on that assumption we have analyzed the web server access logs with the following simple script:
for i in /home/cacert/var/log/apache2/access.log* do echo -n `zgrep 'GET /index.php?id=4' $i | wc -l` echo " $i" done | sort -n
- Result described by critical team:
The results of this reveal that in a few specific weeks, *much* more queries matching the pattern were issued than in the "average" week. This concerns occurences in four different weeks (the logfile is rotated once per week), ending on these dates: 20140112, 20140810, 20140817 ad 20141019. Note that the bug was patched on 20141018.
for i in /home/cacert/var/log/apache2/access.log* do echo "$i:" zgrep 'GET /index.php?id=4' $i | cut -d ' ' -f -1 | sort | uniq -c | sort -rn |\ while read nr host do if [ ${nr} -ge 1000 ] then echo $nr $host fi done done
- Result described by critical team:
The results of running that script reveal that indeed a specific host can be linked to each occurrence, with the matches for 20140810 and 20140817 coming from a pool of hosts with similar addresses.
3rd script: analyse bin-log
[Will be added after analysis/reviews/discussion]
Ruling
Execution
Similiar Cases
to be done