Weekend Learning: Exadata Database Machine and Exadata Storage Server versions

Have you ever asked yourself what the latest and greatest versions of Exadata components are?  So have I!  Well we are all in luck, as MOS document 888828.1 for the note "Exadata Database Machine and Exadata Storage Server Supported Versions" has all of the information you would ever need about Exadata versions!!

Exadata: Flash Cache Compression on X3 and X4 Storage Servers causing disk drive failures

Check what version of software you have on your Storage Servers immediately due to the bug listed in "(EX28) High risk of data loss on X3 or X4 Exadata storage when flash compression is enabled and running late 2015 or early 2016 software release" MOS article ID 2115005.1 which currently has NO solution!  There is a workaround listed, but that is turning off the Flash Cache Compression which will likely negatively affect the performance of your Exadata application and Oracle gives some great advice about proceeding with caution if you are planning upgrading!

Exadata: CellCLI "list activerequest" command can produce I/O hang in Storage Server

Do not miss out on applying the patch to resolve the issue described in the MOS article "(EX27) Database IO may hang when CellCLI command "list activerequest" is executed on an Exadata Storage Server" from note ID 2095255.1 so your system is not subject to this if you ever need to run the command in an Exadata machine!

Weekend Learning: Whitepaper on Exadata monitoring with Enterprise Manager

Do not miss this whitepaper on Monitoring Exadata Storage Server Grids with Enterprise Manager from the Maximum Availability Architecture (MAA) team which has the entire step by step process and all the SQL you need to set up all of the conditions you need to alert on!

Exadata: Cellsrv process could crash on Storage Servers during cell-to-cell offload operations

If you are running Exadata platforms, do you know what version of software you have in your Storage Servers or what is running on your Grid Infrastructure?  Find out because the article "(DB35) Cellsrv process on storage servers may crash when performing a cell-to-cell offload operation" on My Oracle Support affects only certain combinations of software and note ID 2074393.1 details both of the patches that are needed to combat this situation!

Exadata disk drives in Storage Servers at risk for increased failures and performance issues

If you have Exadata in your environment I think you need to take a look at this newest MOS alert "(EX25) Disk drives in Exadata storage servers may experience increasing number of failures, unrecoverable media errors, confinement, and higher levels of read data latencies" via note ID 2073916.1 because several different versions of Exadata are affected by this issue with High Performance disk drives!

Exadata: New feature Exadata I/O Latency Capping causing ORA-27626 errors

Tonight we find out in My Oracle Support document 1925661.1 "ORA-27626: Exadata error : 2201 (IO cancelled due to slow/hung disk)" that the new feature Exadata I/O Latency Capping introduced with Exadata Storage Server 11.2.3.3.1 can actually interfere with I/O reads so learn what error messages you can prepare for in the log and trace files!

Learning more about your Oracle installation via IaaS cloud

In this series of articles from the Oracle Systems Blog Austria I think that one can get a better sense of all the parts and pieces involved in making sure your system is up and running 100%.  Sure you could also read these articles to learn more about going to an Infrastructure as a Service with various Oracle services or products, but reading more about servers, storage, OS or virtualization will just broaden perspectives, that we may not have today, for the future.

R12: FND_LOG_MESSAGES

Along the lines of my presentation, and published article, I bring you new details of another vector you need to verify to not bloat your database.  Looking into the space consumption of our database post R12 upgrade, the DBAs found that the table FND_LOG_MESSAGES was taking up 13% of our space in PROD so they found My Oracle Support document 735138.1 which details how to purge this information.  Why is this related to R12 you ask?  Based on some of the data analysis scripts in my article, I substituted in FND_LOG_MESSAGES and found that prior to R12 our monthly total in this table was just under 3.5 million rows added back in 2010.  Yet, the largest volume of data added to this table was in October when we went to R12 of over 320 million rows.  Okay, sure the system churn and processing might be excessive when installed so let's look at the next highest month that's been completed; in November we more than 233 million rows added to FND_LOG_MESSAGES, which is an increase over 67 times between our previous high and the following month after R12 was installed!

I have a few more nuggets of information to share with you about this situation, so stay tuned the next few days if you're interested by this!

EDIT: I realized I forgot to add in the links to my other articles last night!

Data (historical and backups)

I had found this post a little while ago on cleaning up historical data, but I wanted to save blogging about it until I had something real to say about it.  This is that time as it has become very apparent that another one of our Main Tenants should be something to do with data, maybe Data Consumption or Data Usage but SOMETHING because data is just piling up in our systems every day.  Audit data, user data, this data, and that data and how much is being purged, how much can be purged, who has a strategy for keeping track of all of this and where does that fit in with a corporate strategy?

Skip several years later, and the database had grown to be huge for that time. Retrievals were taking longer and longer, and talking to some of the sysadmins late at night around the coffee machine, I learned that the backup was getting too large to even back it up within a week. The application was starting to hit pretty much a ‘brick wall’; it started to fall further and further behind, and something really needed to be done.

Does this sound familiar?  How big is your system?  Has it hit 1 TB?  Are you approaching 5, 10, 50, 100, or more TB of space for your footprint?  How do you approach your backup system?  Do the people that manage your backups tell you when you're having a problem, or do you have to figure it out yourself?  Questions like this are going through my head as I re-read that link because lately in the morning I'm seeing something like this in our new event monitoring alert:

  Waits                                                           Wait Time        Server

 

  Backup: MML create a backup piece           79291.40622    X1

  Backup: sbtbackup                                        3937.101761   X2                  

  Backup: sbtbackup                                      17308.812339   X3

 

This sure seems like our systems are suddenly "wasting" a lot of cycles per hour on backups, especially when these events are from backups that run into our morning several hours and sometimes even into the afternoon.  So begins the investigation!

System Wait Events Part 2

For my first follow up post, I thought it would be appropriate to show why ample alerting can help you to save the day.  It just so happens that on the day that we were closing A/R for the year, we started seeing some very odd things in a new alert we have.

Waits                                                           Wait Time

  gc buffer busy acquire                                     17253.907174

  read by other session                                      16173.301272

  cell single block physical read                             4533.532144

  Disk file operations I/O                                    3961.591694

  gc buffer busy release                                       3783.63651

  RMAN backup & recovery I/O                                  2962.854376

  buffer busy waits                                           1803.283808

  gc current block busy                                       1652.088758

At this time, we see in our DB logs a multitude of these entries:

Tue Sep 04 13:15:04 2012

OSPID: 19311: connect: ossnet: connection failed to server x.x.x.x, result=5 (login: sosstcpreadtry failed) (difftime=1953)

Anecdotal evidence of something going wrong is that at this time I observed that closing a form took 20-30 seconds for an individual, and I had another business user indicate that in the last hour or so he's been getting a "circle mouse" indicating waiting on the form when he tries to move between records on a student account.  So knowing that "something is going on" I start drilling in some more to investigate sessions which are making up some of the time in the above alert buckets and this session breakdown of wait times over 2 seconds shows:

WAIT_EVENT	TOT_WAITS	WAIT_PCT	WAIT_SECONDS	WAIT_MINUTES	TIME_PCT
gc buffer busy acquire	1033	0.09	418.49	6.97	0.03
gc cr block 2-way	31312	2.71	3.21	0.05	0
gc cr block busy	6460	0.56	3.24	0.05	0
SQL*Net message from client	532955	46.14	1526240.71	25437.35	99.96
cell single block physical read	1972	0.17	2.8	0.05	0

I don’t know what the session was doing, or if it was successful, but that doesn’t look like a normal session wait time profile on our system.  Later we saw this as well (but it was related to our DBA trying to kill the RMAN jobs):

Tue Sep 04 14:18:52 2012

ARCH: Possible network disconnect with primary database

So, this is all nice and everything but what happened?

Answer:  Our storage engineer started researching the issue and found that our IB switches and other Exadata components had no issues, but when he checked our 7420 storage array out it was a completely different matter all together.  There were no issues with head 1 of the array, but when he went to check on head 2 he was unable to log in through GUI or ssh with also a simple df command hanging on an NFS mount coming from that head.  When he performed a takeover to move NFS control over to head 1, head 2 complained about lacking resources when attempting to fork something and the only way he could force a takeover was to power down head 2.  What I learned by researching for this article can be found here about what a 7420 head configuration could look like in Illustration 20 and Illustration 21 and the following paragraph as well.