* Case Number: a20140114.1 * Status: closed * Claimants: Support (represented by Marcus M) * Respondents: CAcert * iCM: MartinGummi * Case Manager: BernahrdFröhlich * Arbitrator: EvaStöwe * Date of arbitration start: 2014-01-20 * Date of ruling: 2016-09-01 * Case closed: 2016-09-16 * Complaint: Insufficient system storage * Relief: 1. review of emergency actions (deletion of emails) 2. "advise" about what to do with the remaining emails Before: Arbitrator Eva Stöwe (A), Respondent: CAcert (R), Claimant: Support (represented by Marcus M) (C), Case: a20140114.1 == History Log == . 2014-01-19 (issue.c.o) case [s20140114.67] . 2014-01-19 (iCM): added to wiki, request for CM / A . 2014-01-19 (iCM): note to Private Part . 2014-01-20 (A): I'll take care of this case and select Bernhard Fröhlich as (CM) . 2014-01-20 (A): mail to support member about relevance to free disk space and the nature of the remaining mails . 2014-01-20 (support member): answer to the questions - remaining mails take about 1,5 GB . 2014-01-20 (A): asks email-admin about relevance to free disk space and related questions . 2014-01-20 (A): asks infrastructure admins about relevance of the problem . 2014-01-21 (email-admin): answer to the questions - mails take a lot of space . 2014-01-22 (A): asks internal auditor for advice on keeping or deleting support mails . 2014-01-24 (interal auditor): mails probably should be kept for 6 years because of legal issues . 2014-01-24 (A): additional questions to intermal auditor . 2014-01-24 (A): additional questions about backup of mailsystems and OTRS to infrastructure admins . 2014-01-25 (infrastructure admin): answers to general questions - issue should be solved soon because of move of infrastructure systems to new location with a lot more space . 2014-01-25 (infrastructure admin): all infrastructure systems are backuped together for 120 days . 2014-02-01 (A): short discussion with internal auditor on the matter (in person) . 2014-02-03 (A): intermediate ruling I + execute order - infrastructure should keep last backup before incident . 2014-02-04 (A): intermediate ruling II - support should not delete any mails until further notice by an arbitrator . 2014-02-04 (representant of Support): intermediate ruling II was send to all support team members (info per voice) . 2014-02-04 (infrastructure admin1): ack mail server has moved . 2014-02-05 (infrastructure admin2): ack mail server has moved, disk storage issue resolved & backup safely stored . 2014-02-15 (otrs-admin): OTRS upgraded . 2014-11-08 (A): face to face meeting with C, asks C about possible partial withdraw of the dispute, as there may not be a need for a decission of this part . 2016-09-01 (A): ruling - send to C, email-admin, support, board, infrastructure and critical team; thanks everybody involved . 2016-09-02 (A): asks CM if the case can be closed . 2016-09-16 (A): closes case == Original Dispute, Discovery (Private Part) == * '''Link to Arbitration case [[Arbitrations/priv/a20140114.1|a20140114.1 (Private Part)]], Access for (CM) + (A) only''' ## ==> INCLUDE SECTION BOT <> ## <== INCLUDE SECTION EOT ==== EOT Private Part ==== == Original Dispute == * mail-admin to support: please delete mails {{{ Hi all, Please check whether you can delete old emails on cacert.org. Currently we managed to free 800 MB hard disk space, so it's only a question of hours until we run into trouble again. Thanks in advance }}} * support: note for dispute to look over emergency action and advice for further actions {{{ Created: 2014-01-14 16:39:04 I deleted all mails in the folders returns and request passphrase for the years 2011 to 2013 as all this inormation is stored in OTRS as well. Please advise support how to handle the 100000 mails still in the inbox. }}} == Discovery == === Communication === [[Arbitrations/a20140114.1/Communication|Communication in this case]] === Regarding the Dispute === The case was filed by Marcus as support engineer because of some emergency action he did as support engineer on the request of an email-admin. The dispute seems to ask to review the emergency action (following according ideas in the Security Policy) and especially asks for advise how support should handle the other mails in the inbox. While this specific kind of action is probably not covered directly by the provisions of the Security Policy, a wish for a review of special actions is sensible. Especially as the Security Policy states that support engineers only are allowed to act based on user request, procedures or arbitration rulings and none of this was present, here. So the first part of the dispute is understood like a request for such a review. === Description of Emergency Action === At 2014-01-14, the disc space of the email server was full so that any email had to bounce and would not be able to be processed correctly. As by this no email communication within or to CAcert was possible, the email admin had asked a member of the support team, if support could free some disk space by deleting some of their mail as that was the biggest folder on the system. Prior to this he had done a check if other files could be deleted but there were even no old enough log files to delete. The support team member then deleted all mails in the folders returns and request passphrase for the years 2011 to 2013 as all this information is also stored in the OTRS. The latest backup before that action was secured on request of the Arbitrator. Later the email system was replaced by a system whith much more space, that according to the infra structure team will always have sufficient space for emails at least until the end of the lifetime of the system. By this any possible urgency to decide in this matter was removed. Before it was clear that the need for more or less direct decisions about the handling of the other emails was over, some further investigation in the topic of a possible advise about how emails should be handled was done. The gained information could be used for setting up according procedures. It is part of the communication linked, above. == Rulings == === Interemediate Ruling I === To take out the pressure of time out of the case, I ask the infrastructure team to keep the last backup before 2014-01-14 of the server containing the mail system until further notice. -- Cologne, 2014-02-03 === Intermediate Ruling II === I hearby ask support team to refrain from deleting any mails regarding support until further notice by an arbitrator. -- Cologne, 2014-02-04 === Ruling === I hereby rule: 1. The deletion of the emails was an appropriate and sufficient action to deal with the issue. 1.a) A situation where CAcert cannot get or answer emails is critical enough to allow some special activities to solve the situation. Also no real damage was done as the affected emails can be recovered, if necessary. 1.b) The email administrator already had checked other options to deleting emails, before he send out the request. As email administrator it was his responsibility to search for possible ways to free enough space so that the system could work, agai. His request to support was sensible and has to be approved. 1.c) The deleted mails were of no extreme relevance for the business of CAcert or any individual members especially as they are also available in backups and in the OTRS. It was also in the interest and a task of support to help to enable email again so that members requests could be dealt with, again. This is especially true as there were already complaints about this situation reaching support. The decision to delete the respective mails to free enough space or the near future, was appropriate and has to be approved. 1.d) The danger of the file system running full was already registered by the infrastructure team (and board). New systems were already on their way to be installed. Regrettably they arrived with some delay and by this were not there in time to prevent that the file system was running full. Even as this was not optimal the involved teams tried their best to prevent the situation, they are not to blame for the file system running full. 1.e) All in all, the involved teams acted in the best interests of CAcert and its community. 2. The provisions of the intermediate rulings are removed. The teams are free to handle and delete any backups or mails that were frozen in this case, again. The teams are advised to handle the last backup before the incident that was set up aside in a manner as the deleted mails would be handled. 3. The request to get Arbitration to "advise how support should handle the other mails in the inbox" is dismissed. 3.a) There is no urgency any more to get a quick decision. Especially as Arbitration was asked to handle the case with low priority. 3.b) In general the teams should set up and document their processes themselves. Some general guidelines can be (or even should be) set by or coordinated with board. This seems to be the case regarding the question about how long business data should be kept directly and in backups and when and how it should be destroyed. As this is likely to be related to the decision in which country the servers are - which is likely to be a board decision - this is hard to define in a general manner independent from decisions that are with board. 3.c) If there is a need for a general decision, the appropriate authority to set up a policy is Policy Group. 3.d) Arbitration should only interfere with such decisions if there is some urgency or other specific need for (direct) clarification or if some violation of a policy is claimed/ruled. As the urgency is over, nothing like this can be seen in this case any more. 3.e) The Arbitrator is following a comment from the Case Manager, that to request a decision or "advise" from Arbitration "regarding technical parameters" that fall in the area of the individual teams or board "can be considered an abuse of position most of the time". As the specific request was done under the impression of some urgency and direct need, it is not considered to be abusive, even as the deletion itself already removed the urgency for some while. But it is not appropriate to hold up such a request after a direct need for a quick decision is over. (*) 4. Even as no specific advise is given in this case about how to deal with the remaining emails, the need to clarify how long business data / emails should be kept directly and in backups and when and how it should be destroyed both for support and other teams is confirmed, as no such processes seem to be available. Because of this board, infrastructure and critical team are advised to create according guidelines or processes. If a policy is considered to be a sensible solution, the teams are advised to propose according ideas to Policy Group. (*) If the focus of such a request is about some indication of possible violations of policies or if Arbitration rules that a parameter violates a policy such a request for advise from Arbitration could be appropriate. But then it should be restricted to what could be acceptable in that context and not about how specific details should be. Eva Stöwe - 2016-09-01 == Execution == . 2014-02-03 (A): intermediate ruling I + execute order - infrastructure should keep last backup before incident . 2014-02-04 (A): intermediate ruling II - support should not delete any mails until further notice by an arbitrator . 2014-02-04 (representant of Support): intermediate ruling II was send to all support team members (info per voice) . 2014-02-05 (infrastructure adimn2): last backup before incident stored safely . 2016-09-01 (A): ruling - send to C, email-admin, support, board, infrastructure and critical team . 2016-09-02 (A): asks CM if case can be closed . 2016-09-16 (A): closes case == Similiar Cases == ---- . CategoryArbitration . CategoryArbCaseSystemTasks