As we plan to move Infrastructure services to a new Hosting provider its a fair time to re-evaluate the design. Currently these are only ideas, tbd.
Requirements
Deficiencies with current design
- multiple ssh hops causing latency and configuration pain for a few admins
- firewall configuration out of control of local admins
- lack of centralised account management making it a bit labouring to remove/add admins
- lack of centralised configuration making standard configurations on ssh/web service/logging options difficult to acheive/maintain
- lack of configuration history on services/machines
- lack of a clear tested backup that is available to local admins
- logging checked for abnormal events
- monitoring of systems for expected statue
- email standards - defined email address, dkim signing
- bandwidth accounting /montoring isn't done - needs to be kept in check
Security Requirements
Be guided by SM
- indications when package updates are required
- restrictive firewall rules in and out
- strong authentication for admins to access services)
- strong identity associated with logins (no shared or root accounts)
- logging with strong integrity
- file modification detection
Easy of Maintainence Requirements
- preference for distribution maintained packages
- non-distribution packages need some approval process
- formation of templates for commonly deployed tasks
- easier integration of application maintainers
Design
Proposed Solutions
There are multiple ways to achieve some of these requirements - lets look at them.
Deficiencies
- multiple ssh hops:
- direct ssh access - security requirement mitigated by: standardised config, monitoring
- hopper with more open input firewall rules
- Single Packet Authentication / Port knocking
- firewall config:
- locally controlled at VM with alerts on rule changes just to be sure they are authorized/consistant with security requirements
- account management:
- LDAP server with shared /home directory exported across all guest VMs (Samba?)
- puppet account management
- standardization:
- puppet to manage config in OS independent manner
- config history:
- puppet supports config history?
- RCS on critical files
- other VC?
- backup/restoration:
- daily lvm snapshots of running hosts
- individual backup daemons on hosts - preferences?
- better SQL backups
- logging:
- admins commit rules for their system to categorize log messages
- centralised logging for integrity. logs readable by local admins
- common alert rules for hits on prohibited firewall rules (outbound)
- monitoring:
- local admins place nagios rules to validate system functionality
- common rules for monitoring to check only designed listening services are running
- email standards:
- puppet managed
- dedicate local VM for emailout
- bandwidth accounting /monitoring
- iptables accounting by IP
- specialised log monitoring / webanalyiser/ mailgraph
Security Requirements
- package updates
- cron-apt or similar
- monitoring agent detects this
- restrictive firewall
- rules validated by monitoring
- rules occasionally tested
- strong authentication for admins
- ssh key access only
- strong identity associated with logins (no shared or root accounts)
- no root logins, sudo access for root access
- logging with strong integrity
- remote log with log directory exported back to machine readonly
- file modification detection
- lvm snapshots are examined for modified files. Excludes distribution updates and logging/database files
Maintainence Requirements
Design
Host OS
- ssh access to infrastructure sysadmin managers
Common Services
- puppet
- ldap/samba share/ kerberos(?)
- logging alert rules repository
- monitoring alert rules repository
- dns cache
- emailout