When the Oracle database archiver log files fills up the /flash disk volume, it has the result of halting the rest of the Hyperion suite of products (in this environment we have FDM, HFM, Planning, Shared Services, WorkSpace). Once the archiver logs were emptied and /flash had available space, everything in Hyperion land started proceeding as if nothing ever happened. Except one oddity. We were able to validate our WorkSpace nodes individually without encountering any issues, yet when we went through the hardware load balancer VIP we would receive this error message:
The startup document specified does not exist in the repository. Select a new startup document on the General preferences tab. com.hyperion.tools.cds.util.HSRpcParseException: The response from the server contained an unexpected content type.
Further troubleshooting puzzled us as we found that this error would occur only when the second node was in the VIP, even though we could hit that node directly without any issues. We restarted all of the services that we could think of which would resolve this, even restarting the WorkSpace servers themselves, to no effect. Opening an SR with Oracle, we were asked to remove the offending node from the software load balancing by going into the Agent configuration yet even this had no positive results. We ended up having to reboot the entire Hyperion environment, with no other changes being made, to restore service.
This makes me believe that there were some persistent connections opened which held either sessions or ports so that once the issue with /flash was resolved some part of Hyperion was not able to communicate back to the rest of the system appropriately. Next time, I'll gather some additional information to try and correlate so we don't have to bounce the entire ecosystem.
No comments:
Post a Comment