12c: CRS processes restart periodically on cluster nodes

From MOS note ID 2074515.1 we find the article "CRS Processes Periodically Restart In Some Cluster Nodes in 12.1.0.2" which may be very important to keep an eye out for this showing up in your alert/trace files because it is caused by several slow ASM I/Os in a row!

ASM: After disk comes online resync takes excessive amount of time

If you want to learn a bit more about ASM, especially if you have Exadata because it is affected as well, check out MOS article "ASM disk resync takes long time after disk online" from document ID 2061405.1 including the object GV$ASM_OPERATION which if you investigate on your system maybe you could develop alerts based on it!

Unable to communicate with ASM mirror copy if Exadata Storage Server is offline

Tonight the MOS document "(DB32) Bug 20217875 - When using normal redundancy Oracle is unable to perform I/O to an ASM mirror copy when an Exadata Storage Server is offline" from note 2057731.1 seems straightforward BUT shows how complicated ASM and Storage Servers are with even more reasons to be constantly monitoring these components!

ASM: ORA-600 due to corruption in ASM staleness registry

Lately ASM has been a popular topic, and document 2028222.1 on My Oracle Support continues that recent tradition with article "(DB30) Bug 20904530 - During ASM disk resync ORA-600 [kfdsBlk_verCb] occurs due to corruption in ASM staleness registry" which can teach us about how ASM corruption can be verified, fixed, and repaired!

Exadata: Fixes needed for ASM File System due to data corruption

Fresh off the presses, and on the heels of my post about ASM allowing disks of 2TB, is article "Important Fixes required to implement Oracle Automatic Storage Management File System (ACFS) on Oracle Exadata Database Machine" found on My Oracle Support via document ID 2022172.1 which explains there are multiple concerns of data corruption in ASM including one affecting disks which are larger than 2TB!

Exadata: DB/ASM I/O request hangs if one cell has a problem

Very similar to my post on LCK0 hanging, there is a specific slant to the My Oracle Support document "Exadata Database Or ASM Instance Hung On IO Request When Single Cell Is Having Problem" via note ID 2002084.1 is worth checking out because there are some log entries we can look for to alert us about the next time we have this problem PLUS there is the bug that we can patch our system with too to prevent it!

ASM: ORA-15196 error encountered when disk is over 2TB

Always talking about getting to a better proactive position, My Oracle Support note 736891.1 gives us yet another angle to approach this from document "ORA-15196 WITH ASM DISKS LARGER THAN 2TB" because it affects many different versions of the database AND the Exadata platform!  There are patches available to prevent this from happening to you, and the aforementioned Exadata platform has patches available which allows a larger ASM disk!

Weekend Learning: ASM not starting after upgrade to 11.2.0.4

My Oracle Support article ID 2006534.1 has note "ASM fail to start with error ORA-00132 After Upgrade From 11.2.0.3 To 11.2.0.4" which has a very simple solution to the problem.  Now my next challenge is to understand why the spfile has to be modified in the way Oracle suggested!

Weekend Learning: ASM capacity and free space for Exadata

Add to your toolbox this weekend by getting up to speed on both ASM and Exadata considerations of capacity and free space by reading this very good MOS document "Understanding ASM Capacity and Reservation of Free Space in Exadata" from note ID 1551288.1 with some information about both High Capacity and High Performance.

Weekend Learning: Whitepaper on validation of Diskgroups and ASM

In our efforts to get more understanding of Oracle technology for our toolbox, I suggest that you read the MOS whitepaper "How to Validate Normal or High Redundancy Diskgroups and ASM Disks Partnership (White Paper)" from article ID 1961372.1 to add to that toolbox!  This really takes us step by step on the validation procedure, and makes me think of ways we can monitor this (if OEM does not take care of it!).

11.2.0.3: Instance termination by LMON during node eviction

From the "I wonder if this is from our SR" pages, I have seen firsthand the bug detailed in MOS article "While Evicting One of the Instance, the Remaining instances Terminated by LMON with "LMON is running too slowly and in the middle of reconfiguration"" via note ID 1949505.1 where one instance in a RAC cluster takes the other nodes in the cluster down as well.  Not only is there a bug patch to resolve this, or that it is fixed in 12.2, but there is a database parameter which can help resolve the issue too!

Frits Hoogland: DB parameters from spfile

I've mentioned a few of Frits' articles before, and while this one may be a little out in left field for somebody in a Support role that never really gets to deal with pfile/spfile or even ASM, I think it is a good exercise to follow along with!  Frits has a shortcut to use ASM to get the information out of a spfile using the strings command, but as you will see shortly something is not 100% correct so he goes digging into the guts of how Oracle is working with the spfile to come up with a very interesting (and probably right) hypothesis.

ASM connections fail with ORA-01031 if LDAP enabled in OS

While I do not need to connect to an ASM instance, it is interesting to see a lesson that can be applied otherwise to a future problem or project.  If you read article "Connections to the ASM Instance Fail With ORA-01031 When LDAP Is Enabled at OS Level" from document ID 1927489.1 on My Oracle Support you will see there is a very basic lesson to learn about LDAP and how it needs to be setup for RAC High Availability.

Database Week: ASM notes

Today in Database Week I want to shine a light on these My Oracle Support articles regarding Automatic Storage Management (ASM) because they may help you learn something/more about it!  While I learned that parameters for Exadata mean something on a non-Exadata installation, my real take away was about the crashing ASM diskgroup as we just had something like this happen a few weeks ago so I need to try and correlate this just in case we need to apply the patches indicated.  If you are still in a learning mode, you can refresh yourself on the other ASM articles I've published before!

Warning Message:'Cluster target should exist in Enterprise Manager before adding the Cluster ASM target' is generated when Promoting Cluster ASM Target from auto-discovery Results - Note ID 1911635.1
Normal Redundancy :: Storage split caused crash of ASM diskgroup - Note ID 1623400.1
High 'ksv master wait' And 'ASM File Metadata Operation' Waits In Non-Exadata 11g - Note ID 1308282.1

Unmounting ASM disk by querying V$ASM_FILE on 11gR2

The title of My Oracle Support note 1145365.1 "Alert: Querying V$ASM_FILE Gives ORA-15196 After ASM Was Upgraded From 10gR2 to 11gR2 with an AU size > 1M" does not quite tell you the full story because it neglects to tell you what the ORA-15196 message means or what it results in!  The error is for "invalid ASM block header" and it results in the disk going offline if it has less than a certain number of files in the disk group.

I think it would be good to know if a disk will go offline all of a sudden if the GV$ASM_FILE or V$ASM_FILE objects are queried from, so take this as an opportunity to be proactive and run the query in this note to investigate your ASM disk groups before you find out the wrong way that this is an issue!

After SAN upgrade, ASM gives an inconsistent sector size error

While this is an older Alert in My Oracle Support, the article "After SAN Firmware Upgrade, ASM Diskgroups ( Using ASMLIB) Cannot Be Mounted Due To ORA-15085: ASM disk "" has inconsistent sector size" under note ID 1500460.1 may have some relevance for you and I wonder if this may have something to do with an issue we had a while ago with ASM.  Apparently, the logical block sizes and physical block sizes can get out of sync after a SAN upgrade and more importantly the physical block size can be set to a size that ASM does not currently recognize as valid.  Yikes!

Luckily, this article does a VERY through job of explaining the problem, the cause, and the ways that it can be resolved.

ASM Alert: RAC ASM disk offline after rebalance

Just the other day, I was telling you about an outage caused by ASM maintenance so when I was on My Oracle Support and they had an article entitled "ASM Alert! : After A Rebalance Operation, An ASM Disk Goes Offline In A RAC ASM Configuration (External Redundancy Diskgroups)" with note ID 1537143.1 you're darn right I was interested and read it.  I suggest you do the same if you have to deal with ASM on RAC instances because it describes a very real situation where an ASM instance may not realize the disk was dropped on another ASM instance on purpose and then it gets expelled from the first ASM instance.

Why is this so interesting?  It sounds a lot like what I've encountered several times over the past year, and it affects several different database products INCLUDING Exadata on certain releases of the database and Bundle Patches.  Of course we have seen this on non Exadata instances, where the likelihood of us encountering the issue are higher as they might not see large database patch increments which could introduce the fix and have us avoid this altogether.  I'll be asking my DBAs about this tomorrow.  Will you?  Why not?

When is your database in-doubt?

Following up on yesterday's post where I suggest another reason to have alerting in place, today I explain what happened and the errors we saw in our system which may help you down the road.  A scheduled ASM disk maintenance didn't complete as planned since the system never completed (or took) the commands issued, and several hours the database cluster seemed to panic due to ASM with the following message:

ORA-00020: maximum number of processes 100 exceeded
ORA-20 errors will not be written to the alert log for
the next minute. Please look at trace files to see all
the ORA-20 errors.

so the cluster had to be restarted.  What did that mean for us?  Well, this was a remote database cluster to our local database cluster so overnight when the business had the new R12 Create Accounting report scheduled to run during the remote cluster outage this (and several other) report errored with several messages:

ORA-00060: deadlock detected while waiting for resource
ORA-01591: lock held by in-doubt distributed transaction <ID>

This ID and error message were found in our local database log when the transaction became in-doubt, so this is an opportunity to improve some alerting when this happens the next time to avoid extended locking in the application.

Obviously, the next day after the remote database cluster was up while these reports ran they ran successfully.  For the most part.  I'll tell you more about that later.

11gR2 database bug due to ASM defaults

Did you know that ORA-04031 errors can crop up on you, if you have ASM configured with the default memory parameters being too low?  Serious!  Look up MOS Note ID 14292825.8 where a patch is suggested AND an alternative to patching by setting your MEMORY_TARGET value correctly.  You may want to proactively verify that your databases on 11.2.0.3 and up (not including 12.1) have the correct ASM values set to avoid unexpected outages!

Lesson learned: Rebooting a server

Following yesterday's post about encountering an issue with adding space via ASM, we learned a very valuable lesson with rebooting a server: don't leave your servers up for hundreds of days without a reboot.  Why do I say that?  What happens when you have a problem with your server and it needs to be rebooted, but you have had it up for 300, 400, 500, 800, 1000 days?  When you reboot, fstab in Linux needs to check the associated filesystems and when you've left your server up that long the check has more to do but also you've let possible corruption creep into your system without regular reboots and preventative system checks by letting fstab check the filesystems at regularly scheduled times.

So now you're rebooting a server that has a problem, it is taking longer than normal (which needs to be planned for as well in your change plans) and you have no confidence that the filesystem won't be corrupted when it comes back up which can exacerbate an already critical situation.