Good security practices from Slack - Distributed security alerting
May 22, 2016
In Distributed Security Alerting Ryan Huber describes some pitfalls of common security 'best practices' of monitoring systems.
Most of the things he has issues with, I've experienced first-hand in security and sysadmin work in general.
He then explains how at Slack they have a reactive system where the automation asks you whether you just ran some sensitive commands or not and then requires two factor auth to prove that you did. (Or else it escalates.)
This is quite a handy paradigm of monitoring, but I personally have been wanting a way to say to monitoring "so, I'm going to do stuff now and all problems it raises in the next 5 mins are my fault" -- I mean, Nagios sort of has that with scheduled downtime but Nagios is not a security tool.
Maybe my idea of a blanket statement is a bit too naive and might end up masking unintended consequences so reacting to things that actually did happen is a more mature approach.