Random Oracle MOS Notes - 12/04/13

Today I want to share several random My Oracle Support notes that I've ran across over the past few months, that you may get some benefit out of reviewing yourself.

226738.1 - This note has a 60 page PDF on the batch creation of user accounts, which may come in handy for you if you have a homegrown "self-service" access application like we do.

731190.1 - If the title of this note doesn't get you interested, "Handy SQL for the Oracle Applications", I don't know what else I can say to do the trick!

235053.1 - A manual just shy of 150 pages on the implementation of Oracle SSWA (Self-Service Web Applications).

1516355.1 - This is another MOS note where the title says it all: "E-Business Applications Performance Basic Skills Cookbook".

R12: HR report API error

When we were validating our application after going to R12, we received this curious error message when running a report which would go over database links to the PeopleSoft HR application:

DECLARE
*
ERROR at line 1:
ORA-20000: Error encountered -20000 ORA-20000: Error encountered -20001 ORA-20001: The HR object is invalid
Cause:        The HR object which is attempted to be locked is inconsistent with the corresponding object in the HR Schema.  
Action:        Retry the action with the correct o
ORA-06512: at line 247

Was the problem on our side, in the HR schema locally, or in the PSHR application?  A developer that looked into the problem commented this was a message from the API indicating that an update call validation failed due to the record being updated, as opposed to any actual object invalidation.  While we had not received anything like that in 11i, we were in the brave new world of R12 and anything is possible!

We followed the recommendation to try again, and this time the report did not fail when it was run.  Was the employee data on our side, or PSHR, no longer locked?  Did a DBA recompile invalid objects in the interim?  We will never know, but I want to share this one in the hopes that it may help you down the road when you are upgrading and to get feedback if you have encountered this behavior too.

R12: A Quantum Leap

As a fan of Scott Bakula and the show Quantum Leap, that's the first thing that leapt (pun intended) to mind when we encountered the following issue.  :}

Our application servers started to alert because login transactions were failing off and on, so the DBAs took a look and found that several threads from our BPEL 10g middle-tier application were consuming a lot of resources as they appeared to be in ACTIVE status.  What caught my eye, was there was no real reason for them to be caught up in the API call they were trying to work on, so I helped to take a look into this some more.  Digging into this, the sessions were all marked in gv$session as being in the wait event "resmgr:cpu quantum" from the BPEL 10g machines but I've never EVER seen this wait event displayed.  Neither had our DBAs so an SR was opened and out to MOS and the Internet we went, as the wait event encountered has to do with a feature called Resource Manager yet as the setting resource_manager_plan in GV$PARAMETER showed we weren't even supposed to have it running.

Our research on MOS yielded us a direct hit with note 949033.1, another possibility in note 1530676.1 and bug 15915949 about VKRM processes not running (but they were for us), and several blog posts such as this one and another from the same blog helped us look into this a bit further but we were really stuck.  Based on the first note, our DBAs ran the following scripts:

execute dbms_scheduler.set_attribute('MONDAY_WINDOW','RESOURCE_PLAN','');
execute dbms_scheduler.set_attribute('TUESDAY_WINDOW','RESOURCE_PLAN','');
execute dbms_scheduler.set_attribute('WEDNESDAY_WINDOW','RESOURCE_PLAN','');
execute dbms_scheduler.set_attribute('THURSDAY_WINDOW','RESOURCE_PLAN','');
execute dbms_scheduler.set_attribute('FRIDAY_WINDOW','RESOURCE_PLAN','');
execute dbms_scheduler.set_attribute('SATURDAY_WINDOW','RESOURCE_PLAN','');
execute dbms_scheduler.set_attribute('SUNDAY_WINDOW','RESOURCE_PLAN','');
execute dbms_scheduler.set_attribute('WEEKNIGHT_WINDOW','RESOURCE_PLAN','');
execute dbms_scheduler.set_attribute('WEEKEND_WINDOW','RESOURCE_PLAN','');

What doesn't make a lot of sense is, a. if we don't have Resource Manager set to run, why do we have to set windows for it and b. how was that going to change the session state of the processes coming in from the BPEL application?  All of the threads for VKRM and the BPEL quantum wait state sessions remained in their states after this was run, so what next?  Welllllll.....our DBAs wanted to refresh the BPEL connection pools to see what would happen.  Nothing changed.  Next, Oracle suggested that we should kill off the processes if it was okay but because we couldn't see EXACTLY what they were doing this was somewhat blind and typically killing off sessions will make them go away.  :}

In the end, we're not sure what happened to create this condition and we're not sure that it is avoided entirely so I'm expecting that we'll see a part 2 to this post someday.

Thursday Twitter (Turkey Leftovers) - 12/01/13

I've obviously missed a few weeks of my Twitter updates, so today you get a massive update following the same format as last time, and we'll see how it goes!

Announcements:

https://twitter.com/cwarticki/status/401353337796182017 - Larry Ellison inducted into the Business Hall of Fame, but the real "announcement" here is his tribute to Steve Jobs.

https://twitter.com/OracleBlogs/status/402869160679833600 - What are the recommended quarterly upgrades for the EBS system?  Well, now you can find out!

https://twitter.com/OracleCert/status/404004307579592705 - Last chance to take the Exadata 11g Essentials exam!

https://twitter.com/Oracle/status/405019657796464640 - 25 MORE Twitter feeds that Oracle thinks we should be following, and while I'm not on there right now; I will be, I will be.

https://twitter.com/JonathanGennick/status/405760725391380480 - Going to skip Cyber Monday?  If you like Apress/Alpha books, you might not want to.

https://twitter.com/OracleBlogs/status/405869112456663040 - The new EBS 1099 patches have been released!

Training/Free Stuff:

https://twitter.com/kylehhailey/status/401246600011317250 - Kyle Hailey shares a great way to absorb a lot of TED information in one gulp.

https://twitter.com/Oracle_Edu/status/401434403030581249 - Oracle University wants to help us learn more about EBS end user best practices.

https://twitter.com/oraclebase/status/402811650610188288 - Tim Hall clues us in on the free e-book Multitenant for Dummies from Oracle.

https://twitter.com/OraclePress/status/403563714814951424 - Oracle Press shares a free chapter of the R12 EBS book on Oracle Enterprise Repository.

https://twitter.com/stevefeuerstein/status/403979711354634240 - The Power of Names recorded by Steve Feuerstein for Dell.

https://twitter.com/oraclehardware/status/405003036449583104 - Want an introduction to the Oracle SuperCluster M6-32?  Sign up for this webinar!

https://twitter.com/Oracle_EBS/status/405364087858941952 - EBS team lets us know that there are free webinars for R12.1 and R12.2 on how to upgrade.

https://twitter.com/DBAKevlar/status/405709613837713408 - There is going to be an Oracle Innovation Forum in Denver, which gives you a chance to pick Kellyn Pot'Vin's brain!

https://twitter.com/OracleDatabase/status/405788137156538368 - Did you miss the first DBaaS webinar?  Well there's going to be another one on the 3rd, so go sign up.

https://twitter.com/oraclemagazine/status/405801388137267200 - A free subscription to the Oracle magazine!

General News/Information:

https://twitter.com/Oracle_EBS/status/401346372756312065 - EBS team helps us learn about FSG Signal 11 errors.

https://twitter.com/Oracle_EBS/status/401366502588567553 - The EBS team again comes through with a new analyzer for iProcurement items if you are on R12.

https://twitter.com/bobevansIT/status/401394584233590784 - Bob Evans shares with us that SalesForce is having some issues lately.

https://twitter.com/OCPrep/status/401841556979605504 - Do you know what order to take certifications from Oracle?  Matthew Morris offers us a primer on it!

https://twitter.com/vjswami/status/402117374108639232 - A scary thought.....is Google what Skynet began as?  :}

https://twitter.com/devops/status/402240592752951296 - A great bit on DevOps cycle on the humanity side of things.

https://twitter.com/OTNArchBeat/status/402493505647685632 - Where are you going with your career?  Is an architect role in sight?  This might be worth a read!

https://twitter.com/thatjeffsmith/status/402535834328387584 - All too often Jeff, all too often.

https://twitter.com/devops/status/402648530046951424 - There's a DevOps conference?!?!

https://twitter.com/ericries/status/402967122118443008 - I think this is a bit harsh, but having experienced the waterfall method a bit I can say I see his point.

https://twitter.com/oraclehardware/status/403129516463489024 - What does Oracle think availability best practices are?

https://twitter.com/dba_hba/status/403133092572831745 - Allan Robertson tell us that Facebook's VP of hardware thinks we are close to open source switches.  That's pretty interesting.

https://twitter.com/leight0nn/status/403203127568261120 - Leighton Nelson is interested if you are running Exadata with a ZFS backup device.

https://twitter.com/dba_hba/status/403558860373229569 - Allan has a corrupted LOB to deal with, and found MOS notes 833635.1 &  253131.1 useful in identifying the issue which looked to be a ORA-1555 error initially but was actually an ORA-22924 error in the end.

https://twitter.com/Seth_M_Miller/status/403742348078632960 - Seth has a "problem" with Exadata, and you won't believe what it is!

https://twitter.com/S_Trallori/status/404385548225093632 - Great way to visually show the difference between object and relational databases.

https://twitter.com/fritshoogland/status/404728333466095616 - What does Frits Hoogland think when the wait interface is not good enough?

https://twitter.com/martinberx/status/404939120532672512 - Martin Berger asks a VERY thought provoking question, and with 30+ replies by some of the best in the game it's a good thread to read up on!

https://twitter.com/OAUG1/status/404986743168196608 - OAUG wants to help us learn from some failures on implementing R12.

https://twitter.com/RMOUG_ORG/status/405383027565395970 - This could fall in the announcements category, but I just thought it was interesting that Apple purchased this company.

https://twitter.com/Oracle_EBS/status/405744091746828289 - Did you know there was a Concurrent Analyzer for EBS?  Grab it today!
 

Conference News:

https://twitter.com/UKOUG/status/403179392874319872 - In or near Ireland?  Well the call for papers for OUG Ireland 2014 closes on the 13th!

https://twitter.com/COLLAB_OAUG/status/403227880601042944 - 575 abstracts were selected by OAUG for COLLAB 14 and here https://twitter.com/COLLAB_OAUG/status/403981499386437632 is the lineup!

https://twitter.com/Enkitec/status/403271073052884992 - The E4 call for papers closes at the end of the year!

DOAG:  There was a LOT of information coming out of this conference, so it gets a section all to itself!

https://twitter.com/uwehesse/status/402728736250167296 - Uwe Hesse shares words from Graham Wood about ELAPSED_TIME in v$sql, as does Benedikt Nahlovsky about ASH sample timing https://twitter.com/bnahlovsky/status/402729446752649216.

https://twitter.com/bnahlovsky/status/402778795943292930 - Benedikt also shares what Craig Shallahamer presented for wait events, which is worth some time absorbing.

https://twitter.com/db_inmemory/status/403069574213795840 - Maria Colgan let's us know that she's presenting the In-Memory and Bjoern Rost shows us https://twitter.com/brost/status/403131628106829825 how packed the room was for it!

https://twitter.com/UweHesse/status/403089649318785024 - Uwe tells us that PDB cloning will be available with a future patch, https://twitter.com/UweHesse/status/403095393778823168 a limit on memory consumption hasn't been reached yet, https://twitter.com/UweHesse/status/403096839626719232 that there is some hazy alert.log issues with PDB as well, and https://twitter.com/UweHesse/status/403470105742700544 that creating a PDB clone on ZFS is easy as pie.

https://twitter.com/bnahlovsky/status/403134768948477952 - Benedikt tells us that Maria shared a new column required in Oracle 12c.

https://twitter.com/UweHesse/status/403141180508364800 - Uwe shares with us that an 11g instance running on version 12 needs to be on at least one ASM instance.

https://twitter.com/UweHesse/status/403434903192502272 - Uwe also keys in on the new Backup, Logging and Recovery Appliance and that a full backup is taken only once and then incremental after that.

Social Media:

https://twitter.com/sql_handle/status/401235075200999425 - A way to map out your Twitter followers!

https://twitter.com/ClickTips/status/404959417952522240 - A good tip for bloggers, one I learned about a year in and I wish I had known sooner.

https://twitter.com/Oracle/status/405415164989538304 - Oracle shares The Power of Peer Networks.

https://twitter.com/ClickTips/status/405731473648386049 - Ten steps to successful blogging.

There are a few more days that I could add to this post, but I'll just make a promise instead to be a better blogger and give you a week's worth of Twitter this Thursday!

EDIT: I noticed a few of the hard returns didn't come out exactly right, so I've corrected that.

The future of Oracle Enhancement Requests

Ever want to be on the cutting edge?  Have you thought about seeing inside Oracle's thought process about improving?  Well are you ever in luck today!

Read MOS note 1584210.1 titled "FAQ for the Enhancement Request Community", then come back here.  Back?  Good!  So what did you think?

I'm excited!  This is the future of how Oracle wants to work with their customer base to get Enhancement Requests in the queue, worked on, and ranked by the very customers they are serving.  How cool is that?!?  Also, this is going to be a template for how ERs will be worked for all the other streams so a. get in now, b. make the process better, and c. get ready to work like this for all of the other EBS modules (and maybe beyond).

R12: Bouncing Workflow agent process

I've regaled you with tales of our DBAs being proactive and our Workflow Mailer going down, so when we went to R12 they were put together to continue our story from yesterday as they developed a new alert to tell us when Workflow agent processes went down in order to allow us to provide better support for the users of our application.  Come to find out that during our first week of R12, the new alert is going off at odd and random times causing us and the DBAs to start researching what is going wrong.  Is it the alert?  Nope, we manually check and the Workflow Deferred Agent Listener service component is bouncing sporadically so the DBAs think we need to reference MOS Notes 958178.1 and 953103.1 but before we can run the associated SQL scripts to check what's going on the alert clears as the service component is now Running.  Odd to say the least.

The next time this starts bouncing, we get the chance to review the SQL scripts which tell us there are items in the WF_DEFERRED Advanced Queue but nothing which matches the problems described in the MOS notes so we keep digging.  Referring to MOS note ID 736898.1 it does indicate to use our current re-build notification queue script, but it has an additional step to re-create an index that currently does not exist in our environment.  We run the re-build notification queue script with no changes to the service component.  I keep digging through MOS, and I come across a really good document for Workflow issues (1191125.1) which directed me to the WF_ERROR table and that seems to be saying that the oracle.apps.ar.applications.CashApp.apply business event is causing this because the WF_ERROR table is filled with those entries.  This seems to match what the log file is saying:

oracle.apps.fnd.wf.bes.AgentListenerProcessor.listen()]:WF_EVENT.listen returned processed count: 500, error count: 0

oracle.apps.fnd.wf.bes.AgentListenerProcessor.listen()]:WF_EVENT.listen returned processed count: 223, error count: 10

oracle.apps.fnd.wf.bes.AgentListenerProcessor.read()]:10consecutive errors occurred

Looking in WF_ERROR I can eyeball the errors, and also with this script I was able to see what was being queued and then what errors are associated with that:

select * from wf_deferred a, wf_error b
where a.enq_tid = b.enq_tid
order by a.enq_tid

So what is happening is that the Workflow Deferred Agent Listener service component processes through hundreds of events and then all of a sudden there is a batch of records which has more than 10 errors and it begins the dance of restart death until it gets restarted successfully all by itself when the events get dequeued from WF_DEFERRED and WF_ERROR.

What did we do to address this?  First off, we increased the error count so the service component wouldn't stumble over just 10 errored items, no matter what business event they were from, and then we made sure to turn off the offending business event subscription.

R12: CashApp.apply and Workflow

With the Workflow system pre-R12, we assigned the owner of Workflow to be the SYSADMIN user and my boss entered his e-mail address in the FND_USER account for SYSADMIN so we could receive system e-mails and audit the account properly.  This had an unintended side effect after we went live with 12.1.3, as he promptly received thousands of error notifications for the Workflow event oracle.apps.ar.applications.CashApp.apply which was odd since nothing was wrong in the system since it had passed all of the verification steps in order to release the system.

Looking further into this issue, it appears that the event subscriptions for this were turned on for the Oracle Projects module even though we didn't install it!  You can reference MOS notes 973074.1 and 1313111.1 about how to turn off the subscription for PA_PWP_NOTIFICATION.Receive_BE and patch 8903995 if you really want to "fix" your system by installing the part of Oracle Projects that is looking for this subscription to be on.

Tonight I'm bringing you essentially part one of a series, as this was the start of a much larger issue for us.  I thought after a long absence, this was a good place to start back up with the blog.  :}

Collaborate 14: A Bench Player (and encouraged)

Well it looks like I can't replicate sharing good news about being accepted to OAUG's Collaborate 14 this year.  At least, not yet that is.  You see, I had one presentation declined outright and the other was marked as alternative so it looks like I'm on the bench ready to get called up.  That really is good news as far as I'm concerned!  It means that my abstract was strong enough to be considered with the first round of presenters, but just didn't fit in this year.  So far.  That doesn't mean I won't suggest that you (or your company) join the OAUG or at least go to the Collaborate event as a non-member.  You really should join and go, because it is absolutely worth the investment in yourself, your employees, and your company.

Weekend Reading: Troubleshooting the Linux kernel with Tanel

In this post by Tanel Poder, he lays out very simply how one can use general commands to find out what is going on with a Linux process.  No stack dumps required, no exotic debuggers, or other tools required.  I read this months ago, and flagged it for use later because I'm looking forward to using it.  Yes, I want to have a problem in our environment so I can troubleshoot it with these techniques.  :}

Thursday Twitter - 11/14/13

After yesterday's news, I'm revitalized a bit to share a BUNCH of new things with you but it's Thursday.  You know what that means right?  That's right kids, it's time to visit Twitter!!  All that energy means that I'm sharing almost TWICE as many items as I normally do!

https://twitter.com/footballears/status/397018319657914368 - Frits Hoogland shares the BEST tweet of the week.  The week isn't even over yet, but I'm not going to see anything better than this!

Announcements:

https://twitter.com/oracleblogs/status/401071204314120192 - Oracle announces the Big Data X4-2 Appliance!

https://twitter.com/methodr/status/400352523929677824 - Cary Millsap had released a new book!

https://twitter.com/oraclecloudzone/status/400298064470368256 - Oracle now has 17 Cloud data centers!

Training/Free Stuff:

https://twitter.com/oraclepress/status/399942337427939328 - Get Chapter 8 of the brand new Oracle Database 12c: Install, Configure & Maintain Like a Professional for free!

https://twitter.com/oracle_edu/status/399963239771279360 - Test your knowledge of GoldenGate!

https://twitter.com/csierra_usa/status/400237401269760000 - Carlos Sierra's scripts and presentations.

https://twitter.com/uwehesse/status/400556683526930432 - Uwe Hesse tells us that the Exadata documentation is generally available for our continued learning!

https://twitter.com/oraclepress/status/400708598608977920 - Another free chapter, this time about tuning for EBS R12.

https://twitter.com/stevefeuerstein/status/400973819302539264 - Steve Feuerstein has a session about the power of names in code.  Sounds interesting!

General News/Information:

https://twitter.com/fritshoogland/status/400906989091319808 - Have Oracle Diagnostic Pack installed?  Frits tells you something you need to know!

https://twitter.com/dbakevlar/status/400844582113787904 - Kellyn Pot'Vin shares 10 things Linux admins should know.

https://twitter.com/oraclehardware/status/400578813283016704 - Engineering as a Service?  Really?  News to me!

https://twitter.com/oraclemagazine/status/400372431912574977 - Great snippet about being proactive.

https://twitter.com/mariselsins/status/399932917888983040 - Maris Elsins introduces us to a command line interface for downloading MOS patches.

https://twitter.com/sql_handle/status/398927333614026752 - Another thought-provoking tweet!

Conference News:

https://twitter.com/nocoug/status/401057044561539072 - NoCOUG fall conference details and did you know that first-time attendees are free?  https://twitter.com/nocoug/status/401057204175765504

https://twitter.com/collab_oaug/status/400996072845107201 - OAUG let's us know that Collaborate 14 speakers will be announced on the 19th or 20th.

Hotsos in March!

The Hotsos Symposium this year has been billed as having more than 30 speakers that are global experts in Oracle performance optimization and I can tell you that when I attended in 2012 it certainly lived up to that reputation!  Why am I telling you all this?

I've been accepted to present at this prestigious event!!  I'm so excited and just besides myself with joy!  I even have my own speaker page!!

I hope to see you there in March, or at least have you watch virtually, as the conference is very well priced (especially the already mentioned virtual option) and has some of the best speakers in the Oracle business.  I'm especially excited about the Training Day offered, this year with Tanel Poder!

Thursday Twitter (on Friday) - 11/08/13

Due to some very sketchy wireless on vacation (BlizzCon if you were wondering) I've missed a few days of posting but while I'm in line this morning I wanted to at least share out some of the tweets I've favorited this week!

https://twitter.com/tanelpoder/status/398563340324069376 - The first part of Tanel Poder's troubleshooting articles, focusing on beyond the wait events.

https://twitter.com/oracle_ebs/status/398547938030657536 - I didn't know anything about this new 12.2 feature Flexfield Value Set Security.

https://twitter.com/oracleblogs/status/398517262082379776 - Have Exadata, going to it, or interested in it?  E4 is the place for you!

https://twitter.com/markrittman/status/398373327049617408 - Google helps do SQL injection attacks!

https://twitter.com/oakies_blog/status/398298896126205952 - Oak Table shares a piece by Kyle Hailey on the price of data.

https://twitter.com/cwarticki/status/398200950911172608 - Oracle Support was well represented by Chris Warticki at the EastCoastOracle (ECO) conference.

https://twitter.com/oracle_edu/status/397436989450244096 - Think you know your stuff?  Try this sample exam question for Linux!

https://twitter.com/oratweets/status/397434769534746624 - Iggy Fernandez shares the NoCOUG journal for this month.

https://twitter.com/oracle/status/397422358324973568 - New release of 12c to support DBaaS!

https://twitter.com/thatjeffsmith/status/396628968176361472 - Some tips for your online blogging/tweeting success via THAT Jeff Smith.

Oracle Secure Global Desktop webinar

Off of the R12 path, and more on the webinar path, is a glimpse into a product called Secure Global Desktop (found via Twitter) from Oracle which seems to be a type of "cloud based" (my words, not theirs) desktop tool that lets you remotely (and securely duh) access your applications and desktop from anywhere.  This webinar is being held in just over 2 days away, so register now and let me know what you think about this product.  I think it looks interesting, especially with what looks like support for a lot of different operating systems!

Thursday Twitter (on Saturday) - 11/2/13

Since I'm posting this a few days late, I'm going to give you a LOT of links (especially around free presentations and webinars):

https://twitter.com/yvelik/status/393640724014432256 - Yury gives us a list of Oracle Aces, and then:

https://twitter.com/oracleace/status/394931066865475584 - The Oracle Ace program announces their newest crop of inductees!  Jealous!

This is our presentation/webinar part of the post:

https://twitter.com/uwehesse/status/394437352053276673 - Uwe shares a presentation by Jonathan Lewis and Maria Colgan.

https://twitter.com/dbakevlar/status/395287653349740544 - Kellyn shares the RMOUG FB group for Training Days this year.

https://twitter.com/oaug1/status/395561917483593728 - OAUG members can get some webinars too!
https://twitter.com/enkitec/status/395272871057952768 - Kerry Osborne has a 12c webinar coming up!

https://twitter.com/oakies_blog/status/395580896671436800 - Karen Morton is hosting a webinar also on being an Oracle pro!  I signed up for this one a few minutes after it hit my inbox.  :}

https://twitter.com/otnarchbeat/status/395626178436153344 - ArchBeat shares with us some Oracle ADF development session info about another Virtual Developer Day.

https://twitter.com/oracle_edu/status/396369126354345984 - GoldenGate advanced webinar free from Oracle University.

https://twitter.com/tanelpoder/status/394939333351727104 - Tanel with another performance post, this time showcasing some latch troubleshooting.

https://twitter.com/carymillsap/status/395178513890037760 - Cary shares a mind blowing stat about Amazon code moves!

https://twitter.com/oracle_ebs/status/395574583421853696 - Gathering stats for EBS from the ATG group!  Gold star blog post if there ever was one!

State of the Union (or blog in this case)

Taking a moment to thank everybody following the blog, seeing my updates on LinkedIn, or following me on Twitter!  This month I didn't top last month's view count, but it's my second month in a row of having more than 1,000 hits and more than 44 hits per post during the month.  I've even passed the 5,500 hit mark AND I've posted more than 200 articles for the blog!

Last June I started the blog with no real goal in mind, but now I've started to hit a stride and it's really great to be part of the Oracle community.  Looking forward to hearing back from several conferences that I've submitted abstracts for so I can get out of my "shell" here and participate even more!