Weekend Learning: Proactive Oracle Support Event Calendar

Make sure you take a look at My Oracle Support's "Get Proactive Dynamic Event Calendar" from note ID 125716.1 because there are some very good sessions on the upcoming calendar about Accreditation, Administration, Working Effectively, and Service Requests!

Weekend Learning: Can you void Support Services from Oracle by not applying the solution they offered?

I share a LOT of articles from My Oracle Support so it is appropriate that I share a post ABOUT My Oracle Support for once!  If you have ever wondered "If an Alternate Service Request Solution is Implemented in Lieu of an Oracle Recommended Solution, Does That Void the Support Services from Oracle?" then check out article ID 2008181.1 for the answer because I am glad that I did!

Weekend Learning: Oracle Database Support News via Oracle Premier Support

I am not too sure if document 230.1 on My Oracle Support is always for their Premier Support group, or just for the latest Oracle Database Support News, but I am going to bookmark article "Oracle Premier Support - Oracle Database Support News - Issue June 2015, Volume 52" on the off chance it gets updated each month and then I will be bookmarking "Oracle Premier Support - Oracle Database Support News Archive" from note ID 1284265.1 as well so I can stay up to date on news from Oracle Support groups!

Weekend Learning: Getting the most out of My Oracle Support

Since I review a lot of articles from My Oracle Support and recommend that you add them as favorites, this post from Steven Chan's blog reminds us all about the Hot Topics e-mail that goes out every day and how it can be modified so you are notified when your favorited MOS documents are changed!  Do not miss out on this easy way to stay up to date on MOS articles that are relevant to your current technology stack, or even about where your technology is going in the future!!

Weekend Learning: Improve hardware support from Oracle

Did you know there are ways you can get better hardware support from Oracle via MOS and the Proactive Hardware Services portal?  Visit the page "Improve Your Hardware Support Experience - Part 2" to find out more about this, visit Part 1, ad get the latest blog site for MOS too by visiting the most recent post on this blog site!

Weekend Learning: MOS Conflict Checker tool

A few weeks ago a new tool was released on the My Oracle Support portal so you can upload an OPatch file and have Oracle tell you what patch conflicts exist for the target patch you want to apply to your system.  Visit the page "How to Use the My Oracle Support Conflict Checker Tool [Video]" via document ID 1091294.1 to learn more about the steps for analyzing an OPatch, resolving conflicts, and what known issues are out there right now for this new tool.  Check this out, and let me know about your experiences with it!

Why the wait event "SQL*Net message from dblink" can suddenly increase for no reason

Following yesterday's article about making assumptions post project implementation, are the technical details to what I was talking about!

We launched our 12c upgrade to a major Oracle database in our infrastructure and being one of the first in the company to do so made us (okay, probably just me) very cautious (or leery) about what bugs would shake out of the system after we put it in.  Over the weekend we did the upgrade by installing 12c on new hardware, made sure GoldenGate was up to date before we broke the link between systems, and then linking to the new system.  After the logs were synchronized and everything was tested, not a creature was stirring, not even a mouse.

The next day, we still had no obvious issues but our system wait event alert told me that the activity for the wait event "SQL*Net message from dblink" was starting to trend upwards from where it normally is during the day.  By midday, the activity was measured to be 10-15 TIMES what it was Friday before the deployment and not only that but "TCP Socket (KGAS)" activity was trending higher on the new 12c database.  Of course, 12c is the culprit here, and we have to find out what bug we are encountering right?  Not so fast.  We need to get evidence and prove that out!

First off, is what we were seeing in the database log every few seconds on the new 12c database connecting back out to our main database:

Fatal NI connect error 12170.

  VERSION INFORMATION:
        TNS for Linux: Version 12.1.0.1.0 - Production
        Oracle Bequeath NT Protocol Adapter for Linux: Version 12.1.0.1.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 12.1.0.1.0 - Production
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12535

TNS-12535: TNS:operation timed out
    ns secondary err code: 12560
    nt main err code: 505

TNS-00505: Operation timed out
    nt secondary err code: 110
    nt OS err code: 0

This leads us to My Oracle Support article "ORA-0600 [17099]" in document ID 1325533.1 but why is nobody noticing a problem here that is being reported if "things are not right" to such a degree?  I'm glad you asked, because while I started researching this I noticed that my SQL client tool was hanging for up to 15 minutes before the error "ORA-03113: end-of-file on communication channel" was returned.  That is not right at all.

So, what kind of errors are we seeing in our main database log?  I am glad you asked!

Fatal NI connect error 12170.

  VERSION INFORMATION:
        TNS for Linux: Version 11.2.0.3.0 - Production
        Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.3.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.3.0 - Production
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12535

TNS-12535: TNS:operation timed out
    ns secondary err code: 12560
    nt main err code: 505

TNS-00505: Operation timed out
    nt secondary err code: 110
    nt OS err code: 0
  Client address:
Error 3135 trapped in 2PC on transaction 3283.9.55580. Cleaning up.
Error stack returned to user:
ORA-03135: connection lost contact
ORA-02063: preceding line from 12C_DB

Seeing that most of the error messages are the same between both databases made me do more research on My Oracle Support!  What I found was "Alert Log Errors: 12170 TNS-12535/TNS-00505: Operation Timed Out" in document ID 1628949.1 and again the article was extensive.  Not only in depth, but it had another hint to what the problem was which was found in the first MOS article I shared here.

What is the problem?  The SQLNET.EXPIRE_TIME parameter was not set in the $TNS_ADMIN/sqlnet.ora file on the new 12c platform, the way it was in the original database which we "cloned" over.  After setting this on both RAC nodes and reloading the listeners, the issue went away along with all of the error messages in the log file!

We never had a problem with 12c, even though it was easy to blame it outright at the start!

Weekend Learning: My Oracle Support Accreditation

Do you work with Oracle directly through the My Oracle Support site?  Do you have a few minutes free this weekend to learn something, and prove that you have what it takes to work with Oracle?  If you have answered yes to these questions, then you should go to MOS and look up these articles to start getting accredited:

My Oracle Support Accreditation Series - Level 1 My Oracle Support - 1579751.1
My Oracle Support Accreditation - Frequently Asked Questions (FAQ) - 1585906.1
My Oracle Support Accreditation Series - E-Business Suite - 1580100.1

Yes, there are more series besides the EBS application so go check out the other series available!

R12.1.3: Can a refund from Receivables have a Payables invoice in a different OU?

I am not sure why this feature is requested, but I do not spend time designing business cases so I assume this could be valid in certain scenarios depending on different business processes that some companies may have.  Regardless, it appears that this could have existed prior to R12.1.3 but is not possible after that code release per MOS note 1626484.1, yet if you need it to work this way an Enhancement Request has been opened in bug 18274880.  Oracle suggests that you open an SR, and mention this bug, if you want to be associated with the development of the ER.

Oracle Support at Collaborate14!

The next OAUG Collaborate conference is coming up quick, and MOS note 1525331.1 tracks the sessions that feature Oracle Support during the conference!  Have you ever wanted to know more about Oracle Support, or even meet some of the people you've worked with previously?  Well then, this is your chance!  Take advantage of it!

Weekend Learning: Oracle Data Dictionary

This likely will be old hat to many out there in the community, but I found note 1500058.1 in My Oracle Support that deserves a good viewing if you're new to looking for information related to Oracle database and application technology.  Titled "Master Note: Overview of Oracle Data Dictionary" it goes over the difference between the views that are prefixed with ALL_, DBA_, and USER_ plus teaches us about the view DICTIONARY which lists all views in the data dictionary.

For more advanced individuals, this note also has several interesting tidbits about the O7_DICTIONARY_ACCESSIBILITY parameter as well as a change in 12c to the SELECT ANY DICTIONARY privilege which limits access to several items so they are no longer able to be viewed.

ASM Alert: RAC ASM disk offline after rebalance

Just the other day, I was telling you about an outage caused by ASM maintenance so when I was on My Oracle Support and they had an article entitled "ASM Alert! : After A Rebalance Operation, An ASM Disk Goes Offline In A RAC ASM Configuration (External Redundancy Diskgroups)" with note ID 1537143.1 you're darn right I was interested and read it.  I suggest you do the same if you have to deal with ASM on RAC instances because it describes a very real situation where an ASM instance may not realize the disk was dropped on another ASM instance on purpose and then it gets expelled from the first ASM instance.

Why is this so interesting?  It sounds a lot like what I've encountered several times over the past year, and it affects several different database products INCLUDING Exadata on certain releases of the database and Bundle Patches.  Of course we have seen this on non Exadata instances, where the likelihood of us encountering the issue are higher as they might not see large database patch increments which could introduce the fix and have us avoid this altogether.  I'll be asking my DBAs about this tomorrow.  Will you?  Why not?

Weekend Learning: Adding more security to Oracle Workflow

If you're a regular reader of this blog, you'll know that the Oracle Workflow product is near and dear to my heart for several reasons, so when I stumbled upon MOS note 965413.1 my curiosity was captured.  This note outlines several options for making the Workflow product more secure by limiting the amount of data returned to the user based on the partition that the data resides in, but I think the real benefit of this might be the suggestion you can add your own security sets on these data objects which may yield some positives if you're using the Workflow product quite a bit and need to segment your data returns.  How would you use this type of extra security in the Workflow product?

An odd AME error when submitting expense report

A few days ago we had a user report they had run into an error when they tried to submit their expense report for approval:

Error Name: WFENG_ITEM_ATTR Error Number: 3103 Error Message: 3103: Attribute 'AME_ENABLED' does not exist for item 'APEXP/<expense report ID>'. Error Stack: Wf_Engine.SetItemAttrNumber(APEXP, <expense report ID>, USER_ID, <user ID>) Wf_Engine.SetItemAttrText(APEXP, <expense report ID>, AME_ENABLED, Y

Looking on MOS I found several articles, but none seem to be directly on point as note 401070.1 assumes that a profile option is set wrong (and it isn't for us) and note 317539.1 believes that a tablespace needs to be altered and Apache bounced (neither one needed to be addressed here).  To resolve the issue I had the user withdraw, and then resubmit their expense report, and while I don't know what the root cause for this was I wanted to share it with the community just in case you encounter it in the future.

Weekend Learning: Slow database I/O

If you've always wondered about your database I/O performance, ways to measure and interpret it, or some documentation on the subject you're in luck today!  I found the article "How to Tell if the I/O of the Database is Slow" on My Oracle Support under note 1275596.1 and it starts to help peel back the onion of performance a bit, especially with all the documents linked within the note, so I saved it knowing this was something I wanted to review in depth when I had some time.  I suggest if you're interested in performance on an Oracle system, you do the same this weekend with me.  Maybe we'll compare notes at the end with a quiz.  :}

Weekend Learning: A/R Customer Interface

Yesterday I focused on the A/R module, so it only makes sense to end this long weekend (if you had the day off to celebrate President's Day that is) by sharing some recent MOS articles I've found for A/R!  First off we have note 1493002.1 that goes over the Customer Interface (RACUST) program, terminology, as well as known issues and patches.  Next up, is note 1460677.1 which goes over the meaning of INTERFACE_STATUS error messages which is a useful reference when troubleshooting problems importing or creating records in A/R.  Lastly, we have note 1195997.1 that details out the fields in the table RA_INTERFACE_LINES_ALL including their validation and destination details.

R12: Missing GL CM error messages in FND_MSG_PUB

I've shared a few A/P stories over the last few days, so it's only far to turn to our A/R crowd for another new bug we found in R12!  A few months after we had R12 up and running, all of a sudden a certain type of receipt was not allowing us to create credit memos on the invoice and all we were getting back was the error code "AR_RW_GL_DATE_BEFORE_CM_GL".  At first I thought it was a custom type of error message because it was very rough, and not what I was expecting from Oracle API error codes, so in searching My Oracle Support I was amazed to not only find it but I also found note 1393304.1 detailing how this was one of several error codes ("AR_APPLY_BEFORE_CM" is another) did not have their robust error reporting defined.

This bug is the exclusion of several error descriptions from FND_MSG_PUB.GET, but it could have been found and reported on before we had R12 in place if we had test cases for a bad data scenario like this.  As with several other R12 posts recently, this isn't to highlight Oracle in a negative light, but shine a positive light on the ability to find these items prior to your deployments with stronger test cases and more rigorous testing.

R12: Problem with Expense Report Export Program

Our A/P group reported an issue when a line on the Expense Report Export Program showed up indicating "Expense Report Line and Distribution Totals do not match" while the table AP_EXPENSE_REPORT_HEADERS_ALL showed "EXPENSE LINE DIST MISMATCH" for the REJECT_CODE.  This was news to me as I had not seen it in R11 nor the past three months we had been running on R12 so off to My Oracle Support I went, and I found document 1582981.1 entitled "Expense Report Export Rejection: Expense Report Line and Distribution Totals do not match".  This has the patch or patches which may have caused this, one of which WAS in our system, matches our exact error messages/text, and has additional SQL statements which allow you to verify if this note applies to the data in your tables.

You have two options to fix this, as you do with most issues found in MOS: patch or get a datafix.  Luckily after submitting the information requested in this note via the SR process, the Oracle analyst indicated it was a rather simple datafix and after applied it DID resolve the issue at hand.  I wanted to highlight this problem because the cause of it as detailed in the MOS note needs to be in your testing suite if you're on or thinking of going to R12 since it could be more frequent for your shop or happen a lot sooner after implementation.

Weekend Reading: Oracle Whitepaper on MOAC for custom code in R12

Researching this week, I found the Oracle Whitepaper entitled "Oracle Applications Multiple Organizations Access Control for Custom Code" in MOS note ID 420787.1.  This article contains primarily R12 content, since MOAC was introduced in R12, and while it is not required that you enable it when you install R12 it may be important information for you during your planning if having multiple organizations is a possibility.

R12: Compiling Forms and ORA-04023

A few days ago I shared a new MOS article that caught my eye, and here is another!  Why do you receive ORA-04023 errors when upgrading to R12.1.1 or R12.1.3?  Read note 1503874.1 to find out what the parameter _disable_fast_validate needs to be set at, and why it isn't set the way it is.