Linux OS: Logwatch crontab is sending several e-mails per day instead of one

If you are on a Linux OS then make sure that you check out the article "Crontab Called Logwatch Sends More Email Then Expected" on My Oracle Support via note ID 2114498.1 because there is a very simple reason this is happening and can be easily rectified!

EM: Ops Center shuts down every day around the same time

Sometimes we get abstract lessons which we can apply, and the MOS article "The Enterprise Manager Ops Center Node is Shutting Down automatically approximatively at the same time of the day" from note ID 2026395.1 has that kind of lesson for us where our automation might get away from us if we let it!

Weekend Learning: Allowing access to crontab with non-root user

In our environment we have non-root user accounts that can list and modify the crontab entry, and while I never knew the how behind it, today I know after reading My Oracle Support article "How To Allow Only Specific Non-root User(s) to Use Crontab" from document ID 2002877.1 which may help you modify your setup to use specific power user accounts that do not need root access to the OS.

Converting cron jobs to EM jobs

Have you ever been frustrated with how little configuration there is for cron/crontab jobs?  Well boy are you in luck!  Check out this article from Oracle's Enterprise Manager blog for a great how-to guide for converting your old cron jobs from simple lines in the crontab to complex entries in the EM Job Activities.  This is particularly useful for getting cron jobs to throw out alerts when something is going wrong, and helps take you to the next level of supporting your applications.

Memory leak causing database crashes

After upgrading our database to 11.2.0.3 and applying both BP11 and EX10 Exadata patches we started having database node crashes every 2 to 3 weeks.  As a result we started looking closer at the monitoring information that was being shown before the crashes.  First we identified that only a single database node was starting to run out of virtual memory (swap space) in what appeared to be a random pattern, yet it appeared to happen the most at 9PM for some reason we couldn't fathom.  We even had an outlier happen on Saturday morning where the system recovered free memory around 8 PM, and then ran out of free memory again an hour later!

Time	Swap	Free Mem(G)	Usable Mem(G)
7:47 AM	2317	16	43
8:02 AM	3136	1	44
8:17 AM	4508	0	45
8:47 AM	6315	0	45
9:23 AM	7036	0	45
9:47 AM	7865	0	45
10:17 AM	8915	0	47
12:17 PM	9822	0	47
2:02 PM	10254	0	48
3:47 PM	10250	1	48
8:02 PM	9625	0	47
8:17 PM	9605	16	45
9:02 PM	9515	17	46
9:17 PM	9536	5	46
9:23 PM	9581	0	46

Another mystery was why our "usable" memory never really wavered at all, even though the system was under pressure.  Regardless of that, we all agreed that this was likely not an OS bug since the symptom was only presenting itself on a single node.  An option presented to us by Oracle was changing the sysctl parameter to increase the memory allocated to the database processes in order to try and eliminate the stresses on swap space usage but a nagging question being asked was if we added any new stress or load to the system which would cause this node to under perform.  Obviously, we didn't think so but we kept digging to see if maybe a rogue ETL process or other external partner were hitting us at the same time every night when a pattern of this occurring on a Wednesday or Saturday night started to emerge.  Having dug into this a few times, our scope expanded trying to understand the problem (this is why database load was added to this next trend analysis):

Time	Swap	Free Mem(G)	Usable Mem(G)	DB load (1 m avg)
9:02 PM	8353	29	37	1.19
9:17 PM	8455	1	37	4.04
9:22 PM	9097	0	38	4.39
9:47 PM	11144	0	38	5.65
10:02 PM	11840	0	38	3.19
10:17 PM	12160	0	39	3.19
10:23 PM	12142	0	39	1.72

This type of load jump was abnormal and led the DBAs to look closer at possible suspects like cron jobs to find this:

5 21 * * 3,6 /s_u01/app/applmgr/scripts/tar_tops.sh > /s_u01/app/applmgr/scripts/tar_tops.logs

Since the 3,6 parameter meant the script was only running on Wednesday and Saturday at 21:00 (9:00 PM), QA testing commenced.  Guess what we found out?  The script is the culprit for our memory depletion because after a while free memory became 0!!  Here is how QA looked before the cron job was run:

[applmgr@server midtier]$ free -g

             total       used       free     shared    buffers     cached

Mem:            94         68         26          0          1         14

-/+ buffers/cache:         51         42

Swap:           23          4         19

After some time of the cron job running in QA, you can see the difference:

[applmgr@server midtier]$ free -g

             total       used       free     shared    buffers     cached

Mem:            94         94          0          0          1         40

-/+ buffers/cache:         52         42

Swap:           23          5         18

Once the DBAs changed the cron commenting out this job in PROD, we stopped having regular database crashes.