ZFS Storage Appliance on Solaris could have transient write hangs due to snapshot actions

Tonight from My Oracle Support "Solaris 10, Solaris 11 and ZFS Storage Appliance (ZFSSA) Software may Intermittently Experience Transient Write Hangs if a Snapshot Destroy or Replication is in Progress" in document ID 2067174.1 is a pretty straightforward article that you need to be reading if you are on these Solaris platforms as this could happen at any time due to the type of actions being taken which are automatic!

Solaris: ZFS root pools hang system when unavailable

While we are talking about ZFS and not ZFSSA "Systems with a ZFS Root Filesystem Running Certain Solaris 10 patches or Solaris 11 SRUs may Hang if the Disks of the Root Pool Become Temporarily Unavailable" is still a good document to read on My Oracle Support from article ID 2098232.1 so you can learn what symptoms are possible on Solaris just in case this happens to your system!

ZFS Storage Appliance and Solaris can see write performance degradation when filesystem is almost full

Tonight MOS has a surprising article in store for us as "Solaris 10, Solaris 11 and ZFS Storage Appliance (ZFSSA) Software may Experience Write Performance Degradation when ZFS Filesystem is Near Quota" from note ID 1985528.1 as an entire pool can lose performance if the poolset comes within 2GB of being totally used!  You can also find out what commands are used to check this, and if you are affected then see what patchsets are available so you can get patched up to prevent this from happening to you.

Weekend Learning: Password aging on Solaris

Are you supporting Oracle on Solaris, or is it in your environment in any way?  Do you know about the /etc/shadow file and how it contains the password aging, login activity, account validity and failed attempt lock out policy information?  Check out MOS document "Description of "Password Aging"" from note ID 1007235.1 for all the details about this!

Weekend Learning: Solaris Zones

Ever wonder about virtualization or VM emulation?  How about hardware partitioning on a server?  If you answered yes to either of these questions, check out this article from the Oracle Systems Blog Austria about the 10 best features of Solaris Zones!

Weekend Learning: R12: Solaris on SPARC OS updates required

Do you run EBS R12 with the Oracle Solaris OS on a SPARC platform?  If so, do you know about changes coming in December requiring you to be above a specific release of Solaris in order to continue receiving EBS patches?  True story!!  Check out the My Oracle Support note "New Oracle E-Business Suite R12 Operating System and Tools Requirements for the Oracle Solaris on SPARC platform" via document 1912669.1 for all of the details!

ALERT: Bash vulnerability and Oracle products

Coming straight from Oracle Security Alert CVE-2014-7169, is the notice that the GNU Bash shell command line for Unix/Linux has a newly released vulnerability which will allow an unauthenticated user session to run code.  There are many products like Exadata, Solaris and Linux which do have a patch available to fix this (Exadata was not patched earlier this morning, unless I missed something) but there is a long list of Oracle application which do not yet have patches.  Follow the link to find all the details!

Weekend Learning: Solaris 20 years old

Of course those that do not learn from history are doomed to repeat it, but it may be interesting to know that Solaris celebrated 20 years of existence in 2012 and this site was created to tell the story and share some videos about it as well.  What is the big draw here?  The site also looks into the future by sharing a roadmap for the next few years, including when Solaris 12 might be released.

Weekend Learning/Database Week: Solaris and Sun notes

Tonight my post pulls double duty under both the Weekend Learning and Database Week headings because there is just SO much information here for a single heading!  While we might not be running our EBS on a Solaris box, that does not mean there is not valuable information to learn so I do keep track of articles for that platform especially when they deal with the ZFS appliance.  I you had to pick just one to favorite here in My Oracle Support, I would go with 1021776.1 so you can keep on top of all the new items that come out each week!

How to replace a Sun Ray or peripheral device attached to a Sun Ray. - Note ID 1626529.1
Information Center: Sun Server X4-2 - Note ID 1619441.2
Information Center: Sun Server X4-2L - Note ID 1619505.2
Information Center: Sun Blade X4-2B Server Module - Note ID 1619506.2
Information Center: Troubleshooting for V210/V240/V440/V215/V245/V445 Servers - Note ID 1422373.2
Information Center: Patch and Maintenance for V210/V240/V440/V215/V245/V445 Servers - Note ID 1422213.2
Information Center: Overview for V210/V240/V440/V215/V245/V445 Servers - Note ID 1422147.2
Information Center: Patch and Maintenance for T5120/T5220/T5140/T5240/T5440/T6320/T6340 Servers - Note ID 1414972.2
SERVICE ALERT WEEKLY SUMMARY REPORT (2014) - Note ID 1021776.1
Solaris 11 and ZFS Storage Appliance Software (ZFSSA) NFS Servers may Experience a Deadlock Panic in 'nfssrv:rfs4_dbe_lock' - Note ID 1666249.1
Solaris 11 and ZFS Storage Appliance Software (ZFSSA) Using the Network Data Management Protocol (NDMP) Will Fail to Backup Files and Directories With Inodes Greater Than 2^30 - Note ID 1666311.1
Solaris 10 Kernel Patches 141444-09 and 141445-09 May Cause Interface Failure in IP Multipathing (IPMP) - Note ID 1021262.1
Solaris 11 and ZFS Storage Appliance Software (ZFSSA) May Experience a System Panic in dbuf_hash_remove() - Note ID 1640078.1
Why can the results of df(1M) differ for a shared ZFS on the NFS server and client? - Note ID 1670663.1
Issues With Certain Solaris 10 patches and Solaris 11 SRUs May Cause a Process to Become "Unkillable" - Note ID 1641745.1
Certain Solaris 10 Patches and Solaris 11 SRUs may Cause the ldmd(1M) Service to go into Maintenance State, or Cause the fmd(1M) Process to Dump Core During System Boot - Note ID 1645850.1

Database Week: Data Corruption

This week, I'll highlight a bunch of database related My Oracle Support articles which all have a different slant to them.  If there is one thing that all Support individuals live in fear of, it is corrupted data in the system!  Why?  Because it will show up without any warning and cause problems that probably are not straightforward at all to fix.  Here are some notes to visit and see if you are affected plus maybe get some idea how to be proactive to identify some possible ways to alert about the keywords thrown in the log when data is corrupted!

ALERT: Bug 13605839 ORA-600 [ktbsdp1] ORA-600 [kghfrempty:ds] ORA-600 [kdBlkCheckError]. Corruption in Rollback with Clusterwide Global Transactions in RAC - Note ID 1527740.1

ALERT: Bug 8198906 - Segment header corruption if extent allocation operation is interrupted - Note ID 1229669.1

For Oracle Solaris Cluster, Silent Data Corruption may Occur When Using Buffered Sequential I/O on the Cluster File System - Note ID 1672874.1

Concurrent reports not completing for a single person

Sounds like a weird situation right?  That's what we thought too, but we weren't really laughing about it since it happened on our A/R Close night and we still had it happening after the DBAs bounced the Concurrent Manager.  A few days later jobs by this person are still running and nothing is completing but I don't see any locks at the database and the OS_PROCESS_IDs from FND_CONCURRENT_REQUESTS aren't actually alive on the server.  So what's going on?  Well, the log files tell the story.

Upon closer inspection, it appears that the reports are actually completing all of their work before passing off the job for printing as all of the reports have the same last lines since the user is trying to PRINT TO THE SAME PRINTER.  Alright.  We have a Solaris OS machine here which we aren't running CUPS on, so lpstat -lp <printer name> should give us some details about the printer yet when I issue the command nothing happens.  No detail results.  No command prompt returned.  Nothing.  So it appears that the concurrent manager hands off the reports for printing, but if the printer exists and can't complete the request (in this case just hanging out there) the concurrent request will never continue to Completed.

After we get the printer issue resolved, I try to cancel the reports in Running status but the application locks up on me.  Looking at the locks in the system I can see something is holding me up, but I don't think it is the concurrent request session but the concurrent manager itself so I can't kill it but I took the PROCESS number 14517 from GV$SESSION and went to the server:

[me@server ~]$ ps -eaf | grep 14517
 applmgr  9895 14517   0   Aug 06 ?           0:00 /bin/sh -c lp -c -dPRINTERNAME -n1 -t"USERNAME.REQUESTID" /u01/app/applmgr/11i/i
 applmgr 14517  3854   0   Aug 06 ?           0:07 FNDLIBR FND Concurrent_Processor

You can see that this process number corresponded to the requests which were running under the USERNAME (USER_NAME.FND_USER), REQUESTID (REQUEST_ID.FND_CONCURRENT_REQUESTS), and the PRINTERNAME (PRINTER_NAME.FND_PRINTER) so we can look at the other OS processes with that information:

[me@server ~]$ ps -eaf | grep USERNAME
 applmgr  9895 14517   0   Aug 06 ?           0:00 /bin/sh -c lp -c -dPRINTERNAME -n1 -t"USERNAME.REQUESTID" /u01/app/applmgr/11i/i
 applmgr  9896  9895   0   Aug 06 ?           0:00 lp -c -dPRINTERNAME -n1 -tUSERNAME.REQUESTID /u01/app/applmgr/11i/i
 applmgr  7434  3303   0   Aug 02 ?           0:00 /bin/sh -c lp -c -dPRINTERNAME -n1 -t"USERNAME.REQUESTID2" /u01/app/applmgr/11i/i
 applmgr  7435  7434   0   Aug 02 ?           0:00 lp -c -dPRINTERNAME -n1 -tUSERNAME.REQUESTID2 /u01/app/applmgr/11i/i

What I find interesting is that you can see in the first two lines the process ID 9895 is listed twice, and the same with the last two lines and process ID 7434 so we've learned something about Unix here that the printer OS processes are marked as children of the main concurrent request OS process.  Next up, I issued kill -9 <OS process ID> commands for 9896 and 7435 as applmgr and after I kill the first one process 9895 disappears and the application is no longer locked up on me trying to cancel the report.  Why?  Killing the print job that was hanging at the OS level allowed the concurrent OS process to complete believing it was done printing, which then allowed it to report back to the application that it was done and allowed my session to "complete".  Next up, I killed all of the other threads that had the lp command for our user and when I went back to the application to search for their Running reports there were no longer any reports running.

DTrace and Linux

At the OTN Garage, a recent article got me thinking about how far I've come and how far I still have to go.  All of that because I haven't really had the pleasure of running DTrace even on a Solaris system because I haven't had a need to travel far enough down the guts of the Oracle machine.  Doesn't that make you stop and think?  There is just SO much with Oracle products that we can learn, that when we peel back one layer of the onion we're presented with yet another layer which we need to learn about.