Wednesday, February 19, 2014
In defense of regular alerting
There have been many different alerts that I've designed, and for the most part they are setup to run many times a day and sometimes even several times an hour. This means that you can depend on your alerts coming out at regular intervals, and even if they may have different content each time you are receiving information which in and of itself tells you something about your system. It is up and working as expected. In this way, the alert itself becomes an indicator of your system regardless of what information it holds. Take what happened to us today, where we had a report of a problem by the business and the absence of any of our regular alerting during this time tells us there was a serious problem going on which is above and beyond any of the business reporting. Looking at our system we see that anything going out to a remote database encountered issues including our alerts which help us keep track of several databases at the same time using database links. In this case I was able to use the absence of the alert to start me down the right path of finding the root cause, instead of actually using any information in the alert.