Wednesday, September 18, 2013

CMCLEAN: Zombie Hunter?

The other day I was tracking down a mystery (more on that later) when one of our Exadata database nodes hard crashed and we had to pick up the pieces.  While I was tracking down that mystery, I happened to be on our concurrent manager nodes and saw that neither node had more than 3 processes at the OS layer marked as zombie processes.  What do you think I found after the crash?

We now had 38 zombie processes "alive" on one of the nodes!  As we were cleaning up the system, I kept asking "what about the zombie processes?" to which the DBA told me "don't worry, CMCLEAN will take care of it" but he didn't exactly tell me how or if he really knew for a fact that CMCLEAN would take care of the zombies.  After we took down the concurrent manager nodes and ran CMCLEAN, guess what: all the zombie processes were gone.

So this leaves me to wonder if they disappeared when the nodes were taken down, or if CMCLEAN actually did the job.  Next time, I'll have to watch a bit closer to find out for sure but if you know the answer share in the comments!

EDIT:  For some reason the Blogger iOS app doesn't include changing the "Published On" date, and Safari on my phone wouldn't load the site right/completely, so this showed up yesterday on the blog as created on 9/12/13 instead of when I posted it 9/18/13.  Sorry for burying it on the site!

1 comment:

  1. cmclean does clean up the zombies, in Oracle applications there is a table called fnd_concurrent_requests which keeps track of concurrent processes, cmclean marks the request as complete, terminate and the corresponding processes disappear by pmon cleanup process on db

    ReplyDelete