Corruption in TEMP tablespace on 11.2.0.4 RAC system

There is a bug where if parallelism is turned on you could encounter the scenario described in "ORA-01578 in temp tablespace on RAC" from My Oracle Support document ID 2140124.1 across the nodes so really your only options are to turn parallelism off entirely!

Weekend Learning: Frequent CHECK TIMED OUT listener status on Windows

Want to learn something about RAC configurations?  What about on Windows OS?  Well article "RAC: Frequent CHECK TIMED OUT status of listener and DB resources" from MOS document ID 1608197.1 tells us why these timeouts occur on the Windows platform and how to resolve them as well!

Weekend Learning: Top 5 database/performance RAC issues

If you are running Oracle RAC, check out MOS note "Top 5 Database and/or Instance Performance Issues in RAC Environment" from document ID 1373500.1 to bone up on a few new things this weekend!

Weekend Learning: Sequences in a RAC environment

I posted about some general RAC information previously and today I want to share some more advanced information about "RAC and Sequences" from MOS note ID 853652.1 for those that want to understand more about the Oracle DB which may help you diagnose why transaction sequence numbers just might not be 100% foolproof!

RAC: Getting information about the application or users

I have been working in a RAC environment for so long that the lesson being taught in "How To Get Information on Application / Users in a RAC Environment?" from MOS document ID 2003350.1 is second nature to me, but I think this is really useful information to have so I have decided to share it tonight!

RAC: ORA-245 can be thrown due to controlfile not being shared

Make sure you check out article "ORA-245: In RAC environment from 11.2 onwards Backup Or Snapshot controlfile needs to be in shared location" on My Oracle Support from document ID 1472171.1 to learn a bit about RAC and RMAN!

Hyperion Planning: EPMA Dimension Server does not start using RAC

If you have Hyperion in your environment, how much do you know about Planning?  The EPMA Dimension server?  How about where some of the configuration files are you can run?  Learn about all of them by reading My Oracle Support note "EPMA Dimension Server Not Starting When Oracle RAC is Used for Database" via article ID 1672160.1 to add to your toolbox!

Hardware misconfiguration does not allow CRS to start up

Sometimes the most straightforward MOS articles can tell you so much, and the article "CRS is not starting up when Private Interconnect is disconnected on the other RAC node" from note ID 1981887.1 is a great example of that because there is such a simple explanation at the heart of this article even though it teaches so much.  Did you see this answer coming?  Does it make you rethink anything you have done lately?

11.2.0.3: Instance termination by LMON during node eviction

From the "I wonder if this is from our SR" pages, I have seen firsthand the bug detailed in MOS article "While Evicting One of the Instance, the Remaining instances Terminated by LMON with "LMON is running too slowly and in the middle of reconfiguration"" via note ID 1949505.1 where one instance in a RAC cluster takes the other nodes in the cluster down as well.  Not only is there a bug patch to resolve this, or that it is fixed in 12.2, but there is a database parameter which can help resolve the issue too!

Weekend Learning: RAC patching skipping database defined as ORACLE_HOME

In a continued effort to expand what I might know in a Support role, or what the term even means, I have been keeping my eyes open for nuggets of information that I can add to leverage in the future.  Tonight the My Oracle Support article "Why "opatch auto" not patching RAC database ORACLE_HOME?" via note ID 1479651.1 is one such type of nugget as I typically do not get to look at patching (opatch), ORACLE_HOME, or even GI (Grid Infrastructure) so this is another sneak under the covers of what is happening "at the database level".

ASM connections fail with ORA-01031 if LDAP enabled in OS

While I do not need to connect to an ASM instance, it is interesting to see a lesson that can be applied otherwise to a future problem or project.  If you read article "Connections to the ASM Instance Fail With ORA-01031 When LDAP Is Enabled at OS Level" from document ID 1927489.1 on My Oracle Support you will see there is a very basic lesson to learn about LDAP and how it needs to be setup for RAC High Availability.

11gR2: ORA-600 corruption in rolling back RAC transaction

If you are on several different versions of Oracle's 11gR2 database, you may be susceptible to a known ORA-600 bug detailed in the MOS note "ALERT: Bug 13605839 ORA-600 [ktbsdp1] ORA-600 [kghfrempty:ds] ORA-600 [kdBlkCheckError]. Corruption in Rollback with Clusterwide Global Transactions in RAC" which is found in document ID 1527740.1 but the linked document 13605839.8 has all of the bug details including which versions are affected and which versions have this fixed.

Database Week: RAC notes

Continuing the Database Week feature, I share some documents focused around the Real Application Cluster (RAC) technology with an article about RAC files being shared across nodes, and an article about Solaris evictions while using DISM with the document linking to an in-depth feature on DISM.

RAC database HC_<SID>.dat is used by instance of different Oracle_Home - Note ID 1618161.1
RAC Performance and Eviction Issues on Solaris when DISM in Use - Note ID 1606318.1

Database Week: Data Corruption

This week, I'll highlight a bunch of database related My Oracle Support articles which all have a different slant to them.  If there is one thing that all Support individuals live in fear of, it is corrupted data in the system!  Why?  Because it will show up without any warning and cause problems that probably are not straightforward at all to fix.  Here are some notes to visit and see if you are affected plus maybe get some idea how to be proactive to identify some possible ways to alert about the keywords thrown in the log when data is corrupted!

ALERT: Bug 13605839 ORA-600 [ktbsdp1] ORA-600 [kghfrempty:ds] ORA-600 [kdBlkCheckError]. Corruption in Rollback with Clusterwide Global Transactions in RAC - Note ID 1527740.1

ALERT: Bug 8198906 - Segment header corruption if extent allocation operation is interrupted - Note ID 1229669.1

For Oracle Solaris Cluster, Silent Data Corruption may Occur When Using Buffered Sequential I/O on the Cluster File System - Note ID 1672874.1

Weekend Learning: Steps to stabilize RAC

If you are running a RAC cluster, you should look at MOS note 1344678.1 for "Top 11 Things to do NOW to Stabilize your RAC Cluster Environment".  Not only do you get the top 11, but there are a LOT of links in the article (and at the bottom) which will lead you to even more information to help you on your way to stabilization.  I thought the idea of turning off ADDM in favor of Linux pages was an interesting idea, especially after hearing about performance results on Linux during the past week!

ASM Alert: RAC ASM disk offline after rebalance

Just the other day, I was telling you about an outage caused by ASM maintenance so when I was on My Oracle Support and they had an article entitled "ASM Alert! : After A Rebalance Operation, An ASM Disk Goes Offline In A RAC ASM Configuration (External Redundancy Diskgroups)" with note ID 1537143.1 you're darn right I was interested and read it.  I suggest you do the same if you have to deal with ASM on RAC instances because it describes a very real situation where an ASM instance may not realize the disk was dropped on another ASM instance on purpose and then it gets expelled from the first ASM instance.

Why is this so interesting?  It sounds a lot like what I've encountered several times over the past year, and it affects several different database products INCLUDING Exadata on certain releases of the database and Bundle Patches.  Of course we have seen this on non Exadata instances, where the likelihood of us encountering the issue are higher as they might not see large database patch increments which could introduce the fix and have us avoid this altogether.  I'll be asking my DBAs about this tomorrow.  Will you?  Why not?

Weekend Learning: RAC Attack!

Have you been on social media lately and seen people talking about the RAC Attack when they go to conventions?  (Yury I'm looking at you here!)  Have you wondered exactly what they are doing, and how you can join in on the fun?  Well look no further than this article where it is broken down step by step so you can install Oracle in a RAC configuration on your own computer!

Learning more about DBA_ALERT_HISTORY

In a previous post I indicated how I was going to create an alert based on the table DBA_ALERT_HISTORY because it would show us when a database went down and also come back up.  What I found out the other day was that if a node in our RAC instance becomes unreachable, or can't communicate with the Oracle database for another reason such as a server crash, the table DBA_ALERT_HISTORY may not get logged to until the node becomes available again.

This was interesting because even though the other node went through configuration believing it was the only node available, the remaining node isn't configured to log into this table that it lost track of the other node since it is the responsibility of the offending node to write to the table.  After the server was communicative again 10-20 minutes later, I saw a most interesting series of entries logged into the table indicating that the server is coming up, followed by a notification that the server is down just milliseconds later....even though the instance was still coming up!  This second notification was actually the original notification that should have went out when the instance went down, but it couldn't be logged to the table until the server was operational and communicating with the DB.

I still think I need to create an alert based on this table, but I thought it was very interesting to see the table in a real-time fashion while we had a server basically offline and my expectations didn't match up with what happened at all!

RAC nodes with unknown code differences

As I said several months ago, while going to Exadata we went to a RAC platform and since that time I have yet to encounter any pitfalls with going to this technology stack....until now!  After applying some Exadata related patches we bounced our system as we've done several times since RAC, and just like we did in our QA mock deployment drills, but when the user community got a chance a few days later to put the system through its paces we received various errors reported in both canned and custom functionality.  Drilling into the issue further, we realized it had something to do with our tax solution with "com.evermind.server.http.HttpIOException: Broken pipe" in our application server logs and this in the error logging table:

ORA-29270: too many open HTTP requests
ORA-29273: HTTP request failed
ORA-06512: at "SYS.UTL_HTTP", line 1130
ORA-29270: too many open HTTP requests

It is important to note that one of the patches applied upgraded us from 11.2.0.2.0 to 11.2.0.3.0 and subsequent red herrings from My Oracle Support of 1301699.1 and 1293056.1 had us all assuming the issue resided in new Access Control Lists functionality that might have been introduced with the patching.  Eventually we were able to find that the code being run to get a URL response passed back would run successfully on one of the RAC nodes but not the other and that had nothing to do with ACLs at all.  With this we continued to research and found that the package being called was actually corrupt on one node, but not the other, and had no outwards sign of being INVALID or otherwise corrupted since when it was compiled as a new object on the "bad" node it would run successfully.

This could be the end to our story, if not for the fact that after we recompiled the object we were still receiving the "too many open HTTP requests" error.  At this point, we dug deeper into the system and one item we encountered in our research on My Oracle Support stuck out to us from 961468.1:

What is the reason of ORA-29270?

The database server has a hardcoded limit of 5 open HTTP connections per session. When you attempt to open a 6th http connection, this error is thrown.

Now we hadn't changed the application, or the other database which we're communicating with in this scenario, over the weekend but in some way the database received (or otherwise caused) corrupted communications and held onto these 5 HTTP connections in such a way as to not release them for further communications.  Running lsof and netstat found that on the node in question there were several hundred connections stuck from the other database, and once a kill was issued for them the natural order was restored to the universe when they died off and allowed connections to work again without issue.

This was quite an interesting problem as it was nebulous in finding where the root cause resided, and added a few tools into my toolbox for the next time we have something like this!