Concurrent Manager bottleneck

A continuation of a yesterday's post, we had several "overnight" reports that normally started anywhere between 7 PM and 3 AM actually wait until 4 AM to start running in the Concurrent Manager.  I was even alerted on our on-call phone about a backlog between 3-4 AM on one day, but I figured it was something to do with our user's requests that had been sitting there for a few days and that after cancelling them I assumed the issue would resolve itself.

Good news?  It did, but I was still curious about the why of it so I looked at Autoinvoice Import Program reports, which should have kicked off but didn’t, until I ran into the same report scheduled by a user after 4 PM which didn’t kick off until after 4 AM.  Very curious right?  Why would the system just stop for 12 hours?  Suddenly it hit me.  What did the work shifts look like for the Standard Manager (assuming that’s where they went, and not Quick Manager)?

Our user in question had 7 of 10 slots in the Standard Manager with his stuck reports running before 4 PM and once the shift changed on the first night the process maximum was 7, which didn’t let anything else actually run the following morning until another shift change at 4 AM!  Then later on the second night he had 8 of 10 slots before 4 PM, and again after the shift change nothing was able to process on the third morning until another shift change but that night was a different story because we had his reports complete during the day so there was no lingering backlog to stop up the managers the next morning.

As a byproduct of allowing user reports to sit running for too long, we identified a bottleneck in our system since usually the only time you bother looking at your Concurrent Manager setting is when you have a problem right?  If the system has run for years, and nothing is wrong, you won't go looking for problems since there are none reported!  I find it really interesting that this shift setup is almost completely backwards from our bigger EBS platform where we let the Concurrent Managers ramp up at night until we throttle them back when we have more of a user load on the system, but here we throttle it back at night and then ramp up the processes when the users are in the system.  Maybe this group of users is more report intensive, but I doubt it so off on the hunt for details and a good long term solution I go!  (Sure I can just change the processes here so we avoid that kind of bottleneck in the future, but is that really a solution?)