Alerting is a tricky business. Too many alerts and they will be ignored. Too many false positives and they become a hindrance and a nuisance. The best approach is to have the ability to identify that quantity of “not too many”, and understand the alerts that are received are relevant and controllable.
This can be done in many ways with several tactics. The first way is setting up threshold. For example, a developer can set a high threshold for alerting in development which can go up and down as software is added and vetted. If it goes down for a large period of time, the system will alert.
The second control to add value is the frequency in which the system is checked. Many systems mainly need an hourly checkup, while some Customer facing systems need to-the-minute monitoring because of SLA's and a need for rapid response in case of an outage.
The third option is the ability to control the quantity of messaging. It is counterproductive to blow up everyone's accounts and devices when there is an issue. We alert an issue after the appropriate threshold, then continue to monitor and alert again in a reasonable time-frame as needed. You have the ability to log in and pause alerting until the issue had been resolved.
The right people need to get the notification. How many times has an old employee account been the sole receiver of the critical alerts? Since we have one unified system with unlimited access (unlimited accounts) everyone can make sure the right accounts are updated and routed to the correct individual. Gone are the days of being limited to just one login for a set price. It is a terrible policy to share a single login among the entire team because of cost per user.
We give the right medium for the right alerting visibility. Some alerts work just fine with email but when you need higher visibility, you want to be on a platform that will get the attention the alert deserves. This might be on Slack or Teams. This could be connecting to an existing alerting provider or through text. Our connect service gives the developer the ability to set up different connection options which can define web hooks and API calls. This is promote-able; therefore, different steps in your environment can have different options.