Communication of a20140114.1
- A: mail to representant of C on the relevance of the deletion of said mails
Date: Mon, 20 Jan 2014 21:42:45 +0100
Dear [representant of C],
you filed a dispute in the name of support for the arbitration case a20140114.1.
I'll take this case as Arbitrator. The Case Manager is [CM].
The status of the case is recorded at [1]. If you notice any missing or wrong information, feel free to provide us your point of view.
Since both parties are either CAcert or a part of CAcert (Support) and you yourself are also experienced with arbitration I'll skip the normal formalities.
As far as I understand the dispute, the case is about the deletion of some (100.000) mails from the support mailing list archive as reaction to a mail from an email admin who asked that you free some space on the hard disk.
To be able to decide on this case I have to ask you some questions.
Can you give an estimate how much disk-space was freed by doing so?
Can you give me a short description about the nature of the other 100.000 mails in the support inbox you asked about?
Can you give an estimate about how much space those remaining mails take on the disc?
Are there any direct backups of the inbox? Is every mail that gets into the support inbox stored in the OTRS?
Is there any "normal" mail deletion routine setup for support?
Do you see any reasons to leave those mails in the mailbox - or to delete them - beside of disk space?
[1] http://wiki.cacert.org/Arbitrations/a20140114.1
- answers from representant of C
Date: Mon, 20 Jan 2014 21:56:14 +0100
Hi arbitration,
> As far as I understand the dispute, the case is about the deletion of
> some (100.000) mails from the support mailing list archive as reaction
> to a mail from an email admin who asked that you free some space on the
> hard disk.
I know we have a mailbox size of 6 GB.
> To be able to decide on this case I have to ask you some questions.
>
> Can you give an estimate how much disk-space was freed by doing so?
>
> Can you give me a short description about the nature of the other
> 100.000 mails in the support inbox you asked about?
All mails which where not automatically moved into the area junk and
lost password phrases [a].
> Can you give an estimate about how much space those remaining mails take
> on the disc?
1,5 GB
> Are there any direct backups of the inbox? Is every mail that gets into
> the support inbox stored in the OTRS
To my knowledge there is no direct backup and every mail is handed over
to OTRS.
>
> Is there any "normal" mail deletion routine setup for support?
No there is no deletion routine established.
>
> Do you see any reasons to leave those mails in the mailbox - or to
> delete them - beside of disk space?
Not for any mails after 2010-01-01. Before that might be interesting for
audit reasons.
>
> [1] http://wiki.cacert.org/Arbitrations/a20140114.1
>
[a] https://wiki.cacert.org/Support/Triage Section The Buckets
- A: mail to the email admin who wrote the original request about the relevance to free disk space
Date: Mon, 20 Jan 2014 22:55:57 +0100
Dear [Email Admin],
you get this mail because your funktion as email admin, since I know you are familiar with the arbitration procedures I'll skip the formalities.
I'm the arbitrator of the case a20140114.1. [CM] is the case manager. The status of the case is recorded at [1].
The case was issued by support after they deleted about 100.000 mails as response to your mail to support to free some disk space from
2014-01-14. Support asked by their dispute how they should proceed with the remaining mails in their inbox.
To be able to decide on the matter I need some more informations.
As far as I know you issued the mail because there was no free disk space left on the system the mail server is running on, so the mail server could not work anymore. Is this correct?
Can you give an estimate about the importance to free more disk space in the near future? (How much free space do we have? What do you think how long it will take to fill up that space again?)
Do you have any idea when the mail server will be moved to the new server? (And the problem will be solved for some time.)
Are there any other - easy - ways to free a relevant amount of disk space for the mail server?
Can you give an estimate how much disk-space was freed by the deletion of support mails at 2014-01-14?
Do you know if there is a direct backup to the inbox of support other than the OTRS?
Do you see any reasons to leave - or delete - the remaining mails in the mailbox - beside of disk space?
[1] http://wiki.cacert.org/Arbitrations/a20140114.1
- A: mail to infrastructure admins about relevance of the problem
Date: Mon, 20 Jan 2014 23:50:10 +0100
Dear Infrastructure Admins,
I'm the arbitrator of the arbitration case a20140114.1. [CM]
is the case manager. The status of the case is recorded at [1].
The case was issued by support after they deleted about 100.000 mails as
response to a mail from the email-admins to free some disk space because
the email-server had not enough space to work at 2014-01-14. Support
asked by their dispute how they should proceed with the remaining mails
in their inbox.
To be able to decide on the matter I need some more information about
the relevance of the problem. Please be so kind and help me there.
How much free disk space do we have on the infrastructure server?
Can you estimate how long it takes to fill that space?
Are there realistic ways to free a relevant amount of disk space in the
near future beside of deletion of mails by support?
Do you have an estimate when the infrastructure systems will be moved to
the new location, so the problem will be resolved for some time?
Do you have any other relevant information on the matter?
[1] http://wiki.cacert.org/Arbitrations/a20140114.1
- email-admin: answers questions
Datum: Tue, 21 Jan 2014 07:13:15 +0100
Dear Eva,
Thanks for handling this arbitration case.
> As far as I know you issued the mail because there was no free disk
> space left on the system the mail server is running on, so the mail
> server could not work anymore. Is this correct?
This is correct. I got a phone call from [representant of C], who in turn got user
complaints about error messages coming up while trying to send mails.
Because I am not aware of any monitoring of this machine (at least none
that sends me automated error messages to a mobile device or something
that can keep me informed when I'm not using the internet) and there's
no shift planning for email admins, none of us could forsee when or
whether this will happen.
> Can you give an estimate about the importance to free more disk space in
> the near future? (How much free space do we have? What do you think how
> long it will take to fill up that space again?)
The current disk usage is as follows:
Filesystem Size Used Avail Use% Mounted on
tmpfs 2.0G 0 2.0G 0% /lib/init/rw
tmpfs 2.0G 0 2.0G 0% /dev/shm
rootfs 55G 53G 1.2G 98% /
The last disk space incident we had was some months ago (for sufficently
precise values of "some", I have to dig in my emails for a while), so I
tend to say, we're not in immediate danger. In order to get a better
feeling how far we're running out of space, I set up a small cron job
that logs the current disk usage every night at 12 am. Please note that
this is not a replacement for a real monitoring system.
> Do you have any idea when the mail server will be moved to the new
> server? (And the problem will be solved for some time.)
Please ask [other email admin] about this. As [representan of C] will confirm, [other email admin] and I agreed
upon the following strategy a couple of months ago:
1) [other email admin] has a very detailed idea of how the future system should look
like, so he will set up a POC system and will document every step.
2) After the POC system turned out to work properly, I will take [other email admin]'s
document and will try a second setup. If the second system works the
same way as the POC server, we can be sure to have a setup that we know
precisely and can reproduce whenever needed. Every future change on this
system should be going through a proper change management so we can be
sure that there is no hidden cofiguration that no one is really aware of.
> Are there any other - easy - ways to free a relevant amount of disk
> space for the mail server?
Regrettably I already checked when I tried to free space and found
nothing really promising besides deleting mail. Please find below the
current disk space usage on /:
[other directories]
16G /home
[other directories]
As you can see, most space is being consumed in /home. I already checked
whether we can delete some old logs in /var, but there is no stuff older
than one week, and besides that it does not consume that much space
> Can you give an estimate how much disk-space was freed by the deletion
> of support mails at 2014-01-14?
A really rough guess, because I didn't take notes: Less than 1 GB. Here
are the current home directory sizes sorted by KB. Please handle this
information with care, because it's personal data, although not really
sensitive one.
[anonymization]
list of all other home directories - all of them use less space than support
[/anonymization]
5762568 /home/support
> Do you know if there is a direct backup to the inbox of support other
> than the OTRS?
None that I'm aware of.
> Do you see any reasons to leave - or delete - the remaining mails in the
> mailbox - beside of disk space?
For archiving or legal purposes it might be important to store relevant
mails somewhere. Regrettably a production system's hard drive is only a
makeshift (I guess that's what your last question is heading at). As far
as I know, if we loose this data, we can only rely on the personal
copies the arbitrators made. So it would be a good idea to have some
kind of archive somewhere, but I'm afraid the mail server is not the
best place. Above that, a monitoring system might turn out helpful for
incident handling and capacity planning (although, as I'm fully aware,
it might also be a bit over the top ;)
Hopefully this answers at least some of your questions. Please feel free
to ask me for further details.
CARS and best regards,
- A: asks internal auditor for advice on the matter
Date: Wed, 22 Jan 2014 00:01:56 +0100
Dear [internal Auditor],
I'm the arbitrator of the case a20140114.1. [CM] is the case manager. The status of the case is recorded at [1].
The case was issued by support after they deleted about 100.000 mails as response to a mail from the email-admins to free some disk space because the email-server had not enough space to work at 2014-01-14. Support asked by their dispute how they should proceed with the remaining mails in their inbox.
I would like to get your view as intermal auditor about the relevance to keep or delete support mails.
As far as I learned, no backup is taken of the mailserver. But all support mails are stored in the OTRS-system automatically. Currently there is no mail deletion routine established for support at all. Neither for the mailbox nor for the OTRS.
To be honest I'm not happy with the idea that mails that probably contain a lot of personal data will be stored for an indifinite period of time. Especially since according to Security Manual the logs of the mail system should be deleted after 12 months.
On the other hand support has to be documented well, because they can do a lot and it would be hard to control support if they could delete everything directly. Software is working on a support log, so that one will be able to see the changes support did, but since the reasons behind those changes would be stated in the mails, they are relevant in this regard, as well.
Can you give some advice in this regard?
[1] http://wiki.cacert.org/Arbitrations/a20140114.1
- internal auditor: mails probably should be kept for 6 years
Date: Thu, 23 Jan 2014 23:09:12 +0000
Dear Eva,
That's a tough question. On the one hand e-mails might contain personal or
even sensitive data, that needs to be protected from illicit access; on the
other hand, a track record needs to be kept.
First I checked German law to get an understanding. § 257 para. 4 HGB [1]
says that send and received commercial letters needs to be kept for six
years by merchants; if CAcert is a merchant in the sense of the law is
unclear. Moreover, § 147 para. 1 No 2 & 3 with para. 3 S 2 AO [2], has
also a six year storage time for send and received commercial letters.
I'm not sure about Australian law. I found a webpage about record keeping
[3] in Queensland, stating five years for business - not financial -
related records. I could not find anything about NSW's rules for private
business, but assume the same times.
Taking this into account, I would recommend, to keep every send and
received e-mail for six years and delete them securely after this time. If
this ruling means, that e-mail amount is still to big, a technical solution
(e.g. additional storage) or a organisational measurement (e.g. backup to
another system after 3 years) should apply.
No e-mail older than six years should be deleted if it is not granted, that
the OTRS database really keeps all conversation. Younger mails should not
be deleted at all because the cited law does need the letters in their
original form (here e-mail).
By the way, on the case's documentation site you wrote "2014-01-22 (A):
asks internal arbitrator for advice" should be "internal auditor". ;)
I hope I could help you with my advice.
Best Regards
[1, DE] http://www.gesetze-im-internet.de/hgb/__257.html
[2, DE] http://www.gesetze-im-internet.de/ao_1977/__147.html
[3]
http://www.business.qld.gov.au/business/starting/starting-a-business/record-keeping-business
- A: more detailed questions to internal auditor
Date: Fri, 24 Jan 2014 18:37:08 +0100
Dear ´[internal Auditor],
thank you for your quick and quite informative answer.
> First I checked German law to get an understanding. § 257 para. 4 HGB [1]
> says that send and received commercial letters needs to be kept for six
> years by merchants; if CAcert is a merchant in the sense of the law is
> unclear. Moreover, § 147 para. 1 No 2 & 3 with para. 3 S 2 AO [2], has
> also a six year storage time for send and received commercial letters.
I do not have a HGB at hand, currently, but as far as I remember the HGB
only applies to merchants which are defined in §1HGB (and following). As
far as I remember a "eingetragener Verein" (e.V.) normally does not
apply to this. (Currently there is an interesting discussion going on if
the ADAC with the legal form "e.V." may have to change to another legal
form, because they maybe did to much commerce.)
A "e.V." is probably the legal form we would have if CAcert Inc. would
be a german association. So I do not think that the german HGB really
matches, but I'm not sure. (This may be relevant since our servers are
maintained by secure-u which is a german "e.V.", but I doubt it. It is
one step too far.)
While I like to be on the safe side with the laws, I don't like the idea
to order to keep privacy related data longer than needed.
> I'm not sure about Australian law. I found a webpage about record keeping
> [3] in Queensland, stating five years for business - not financial -
> related records. I could not find anything about NSW's rules for private
> business, but assume the same times.
Thanks for looking up those rules.
> Taking this into account, I would recommend, to keep every send and
> received e-mail for six years and delete them securely after this time. If
> this ruling means, that e-mail amount is still to big, a technical solution
> (e.g. additional storage) or a organisational measurement (e.g. backup to
> another system after 3 years) should apply.
>
> No e-mail older than six years should be deleted if it is not granted, that
> the OTRS database really keeps all conversation. Younger mails should not
> be deleted at all because the cited law does need the letters in their
> original form (here e-mail).
As far as I know the OTRS currently keeps everything it gets. And I
guess it gets backuped regularly, but I will ask infrastructure about
this, to be sure.
Does it really have to be a mailbox, where the emails are stored in?
Especially since the answers of support are created in the OTRS and
mailed from there. They are stored with all meta-information as emails
in the OTRS and are much easier to access than through a mailbox. I
doubt that big companies use mailboxes for this, since they also use
some other tools to communicate and organize their communication.
I would also prefere an OTRS-"only" solution out of some other
considerations - not only disk space. (While there was an emergency
regarding disk space when support deleted the mails last week, this
should be no general issue and no reason for a regular decision.) That
is becaus of who manages the OTRS and who the mailing-infrastructure. I
consider the chances, that OTRS-admins are ABCed persons much higher
than those for the mailing-infrastructure-admins. Just because OTRS is
the bread & butter tool for support. (Currently they are both
administrated by ABCed persons.)
> By the way, on the case's documentation site you wrote "2014-01-22 (A):
> asks internal arbitrator for advice" should be "internal auditor". ;)
Thanks for the remark. I was a little bit tired when I did the
documentation (as far as it went anyway - there will be more, when I
have the time to write it).
> I hope I could help you with my advice.
Yes, you could. :)
- A: two more questions to infrastructure admins about backup of mailsystems and OTRS
Date: Fri, 24 Jan 2014 18:37:12 +0100
Dear Infrastructure Admins,
as I informed you in my last mail, I'm the arbitrator of the case a20140114.1. While I did not get an answer to my questions from my last mail, I have to add two more.
Do you know if there is an automated backup for our mailing systems?
Do you know if there is an automated backup for the OTRS?
If there are backups, please inform me about the relevant time frames.
- Infrastructure Admin: answers to first question-set (infrastructure servers should be moved before the disk is filled up the next time)
Date: Sat, 25 Jan 2014 15:01:39 +0100
Dear Eva,
> How much free disk space do we have on the infrastructure server?
Right now, there are 1,2 GB free disk space on infra01, the only server
used for infrastructure when the issue arised. Currently, there is
approx. another 1 GB used by a backup that could be freed if required.
> Can you estimate how long it takes to fill that space?
Without any actions that should suffice a month or two.
> Are there realistic ways to free a relevant amount of disk space in the
> near future beside of deletion of mails by support?
Now, infra02 is up and we are ready to move services there. This would
be my preferred way to solve this issue permanently.
> Do you have an estimate when the infrastructure systems will be moved to
> the new location, so the problem will be resolved for some time?
It is currently in progress so I would consider the problem as solved.
We had a delay there on almost any process in the chain.
> Do you have any other relevant information on the matter?
I consider deleting data in order to free disk space as the worst
solution to this problem. Of course, users and admin of CAcert services
should care about freeing up resources they do no longer require.
The disk space in question is shared for all infrastructure services on
infra01. There was a different initial setup, but we got rid of because
of shortcomings in disk space. We return to this setup for infra02 that
offers plenty of disk space.
--
Mit freundlichen Grüßen / Best regards
Infrastructure Team Leader
- infrastructure admin: all infrastructure systems are backuped for 120 days
Date: Sat, 25 Jan 2014 15:14:50 +0100
Dear Eva,
> Do you know if there is an automated backup for our mailing systems?
> Do you know if there is an automated backup for the OTRS?
I will answer these together: There is an automated backup for all
infrastructure services for the whole file system. There is no separate
backup of databases etc. from infrastructure side.
> If there are backups, please inform me about the relevant time frames.
Backups are automatically kept for 120 days. It is sometimes adjusted to
available backup space. Currently there are 119 backups dated between
2013-09-01 and 2014-01-25.
--
Mit freundlichen Grüßen / Best regards
Infrastructure Team Leader
- infrastructure adim1: ack mail server has moved
Date: Tue, 4 Feb 2014 00:43:20 +0100
Hello Eva,
> I heared, that the mail server is moved to the new location, so the
> issue with disk storage is resolved for some time. Can you please
> acknowledge that?
yes the mail server as well as the other infrastructure systems has been
moved to the new infrastructure machine where we have plenty of disk space
available.
- infrstructure admin2: ack mail server has moved, disk storage issue resolved, backup stored away
Datum: Wed, 05 Feb 2014 23:19:17 +0100
Dear Eva,
> I heared, that the mail server
is moved to the new location, so the
> issue with disk storage is resolved for some time. Can you please
> acknowledge that?
Right, as [infrastructure admin1] already mentioned. I would even drop the time constraint.
The new server offers enough disk space I could ever imagine CAcert
might require for emails, at least for the lifetime of the server.
> To take out the pressure of time out of the case, I ask the
> infrastructure team to keep the last backup before 2014-01-14 of the
> server containing the mail system until further notice.
The most recent available backup before 2013-01-14 is dated [date].
Unfortunately, some backups were missed due to technical difficulties. A
copy of the email backup is kept in the following locations to ensure
deletion during the regular backup process.
[old location] (machine might be wiped soon)
[new location]
Please give me notice latest by finishing the case whether the backup
should be deleted or for how long it should be archived.
