We had an issue the other day where we did not know if a new server would be able to reach out to the WSDL that our old server was able to hit. While we were fairly confident that we had a problem, I was introduced to a new OS command: curl. Used by itself with no switches, and just the WSDL link after the command as the first argument, retrieved the WSDL definition on our old server without a problem. On the new server the command was not successful because it was not able to reach the destination!
Reading up on the Wikipedia page linked above it appears that this command will download the source of any webpage offered in the argument, including saving the source code off locally with the right switches! This may be a rare tool that we have in our toolbox, but the one time you need it is the one time you will be glad you have it!
Showing posts with label Linux commands. Show all posts
Showing posts with label Linux commands. Show all posts
Monday, June 9, 2014
Thursday, February 7, 2013
RAC nodes with unknown code differences
As I said several months ago, while going to Exadata we went to a RAC platform and since that time I have yet to encounter any pitfalls with going to this technology stack....until now! After applying some Exadata related patches we bounced our system as we've done several times since RAC, and just like we did in our QA mock deployment drills, but when the user community got a chance a few days later to put the system through its paces we received various errors reported in both canned and custom functionality. Drilling into the issue further, we realized it had something to do with our tax solution with "com.evermind.server.http.HttpIOException: Broken pipe" in our application server logs and this in the error logging table:
ORA-29270: too many open HTTP requests
ORA-29273: HTTP request failed
ORA-06512: at "SYS.UTL_HTTP", line 1130
ORA-29270: too many open HTTP requests
It is important to note that one of the patches applied upgraded us from 11.2.0.2.0 to 11.2.0.3.0 and subsequent red herrings from My Oracle Support of 1301699.1 and 1293056.1 had us all assuming the issue resided in new Access Control Lists functionality that might have been introduced with the patching. Eventually we were able to find that the code being run to get a URL response passed back would run successfully on one of the RAC nodes but not the other and that had nothing to do with ACLs at all. With this we continued to research and found that the package being called was actually corrupt on one node, but not the other, and had no outwards sign of being INVALID or otherwise corrupted since when it was compiled as a new object on the "bad" node it would run successfully.
This could be the end to our story, if not for the fact that after we recompiled the object we were still receiving the "too many open HTTP requests" error. At this point, we dug deeper into the system and one item we encountered in our research on My Oracle Support stuck out to us from 961468.1:
What is the reason of ORA-29270?
The database server has a hardcoded limit of 5 open HTTP connections per session. When you attempt to open a 6th http connection, this error is thrown.
Now we hadn't changed the application, or the other database which we're communicating with in this scenario, over the weekend but in some way the database received (or otherwise caused) corrupted communications and held onto these 5 HTTP connections in such a way as to not release them for further communications. Running lsof and netstat found that on the node in question there were several hundred connections stuck from the other database, and once a kill was issued for them the natural order was restored to the universe when they died off and allowed connections to work again without issue.
This was quite an interesting problem as it was nebulous in finding where the root cause resided, and added a few tools into my toolbox for the next time we have something like this!
ORA-29270: too many open HTTP requests
ORA-29273: HTTP request failed
ORA-06512: at "SYS.UTL_HTTP", line 1130
ORA-29270: too many open HTTP requests
It is important to note that one of the patches applied upgraded us from 11.2.0.2.0 to 11.2.0.3.0 and subsequent red herrings from My Oracle Support of 1301699.1 and 1293056.1 had us all assuming the issue resided in new Access Control Lists functionality that might have been introduced with the patching. Eventually we were able to find that the code being run to get a URL response passed back would run successfully on one of the RAC nodes but not the other and that had nothing to do with ACLs at all. With this we continued to research and found that the package being called was actually corrupt on one node, but not the other, and had no outwards sign of being INVALID or otherwise corrupted since when it was compiled as a new object on the "bad" node it would run successfully.
This could be the end to our story, if not for the fact that after we recompiled the object we were still receiving the "too many open HTTP requests" error. At this point, we dug deeper into the system and one item we encountered in our research on My Oracle Support stuck out to us from 961468.1:
What is the reason of ORA-29270?
The database server has a hardcoded limit of 5 open HTTP connections per session. When you attempt to open a 6th http connection, this error is thrown.
Now we hadn't changed the application, or the other database which we're communicating with in this scenario, over the weekend but in some way the database received (or otherwise caused) corrupted communications and held onto these 5 HTTP connections in such a way as to not release them for further communications. Running lsof and netstat found that on the node in question there were several hundred connections stuck from the other database, and once a kill was issued for them the natural order was restored to the universe when they died off and allowed connections to work again without issue.
This was quite an interesting problem as it was nebulous in finding where the root cause resided, and added a few tools into my toolbox for the next time we have something like this!
Wednesday, October 17, 2012
System Wait Events Part 2
For my first follow up post, I thought it would be appropriate to show why ample alerting can help you to save the day. It just so happens that on the day that we were closing A/R for the year, we started seeing some very odd things in a new alert we have.
At this time, we see in our DB logs a multitude of these entries:
Tue Sep 04 13:15:04 2012
So, this is all nice and everything but what happened?
Answer: Our storage engineer started researching the issue and found that our IB switches and other Exadata components had no issues, but when he checked our 7420 storage array out it was a completely different matter all together. There were no issues with head 1 of the array, but when he went to check on head 2 he was unable to log in through GUI or ssh with also a simple df command hanging on an NFS mount coming from that head. When he performed a takeover to move NFS control over to head 1, head 2 complained about lacking resources when attempting to fork something and the only way he could force a takeover was to power down head 2. What I learned by researching for this article can be found here about what a 7420 head configuration could look like in Illustration 20 and Illustration 21 and the following paragraph as well.
Waits Wait Time
gc buffer busy acquire 17253.907174
read by other session 16173.301272
cell single block physical read 4533.532144
Disk file operations I/O 3961.591694
gc buffer busy release 3783.63651
RMAN backup & recovery I/O 2962.854376
buffer busy waits 1803.283808
gc current block busy 1652.088758
At this time, we see in our DB logs a multitude of these entries:
Tue Sep 04 13:15:04 2012
OSPID: 19311: connect: ossnet: connection failed to server x.x.x.x, result=5 (login: sosstcpreadtry failed) (difftime=1953)
Anecdotal evidence of something going wrong is that at this time I observed that closing a form took 20-30 seconds for an individual, and I had another business user indicate that in the last hour or so he's been getting a "circle mouse" indicating waiting on the form when he tries to move between records on a student account. So knowing that "something is going on" I start drilling in some more to investigate sessions which are making up some of the time in the above alert buckets and this session breakdown of wait times over 2 seconds shows:
Anecdotal evidence of something going wrong is that at this time I observed that closing a form took 20-30 seconds for an individual, and I had another business user indicate that in the last hour or so he's been getting a "circle mouse" indicating waiting on the form when he tries to move between records on a student account. So knowing that "something is going on" I start drilling in some more to investigate sessions which are making up some of the time in the above alert buckets and this session breakdown of wait times over 2 seconds shows:
WAIT_EVENT
|
TOT_WAITS
|
WAIT_PCT
|
WAIT_SECONDS
|
WAIT_MINUTES
|
TIME_PCT
|
gc buffer busy acquire
|
1033
|
0.09
|
418.49
|
6.97
|
0.03
|
gc cr block 2-way
|
31312
|
2.71
|
3.21
|
0.05
|
0
|
gc cr block busy
|
6460
|
0.56
|
3.24
|
0.05
|
0
|
SQL*Net message from client
|
532955
|
46.14
|
1526240.71
|
25437.35
|
99.96
|
cell single block physical read
|
1972
|
0.17
|
2.8
|
0.05
|
0
|
I don’t know what the session was doing, or if it was successful, but that doesn’t look like a normal session wait time profile on our system. Later we saw this as well (but it was related to our DBA trying to kill the RMAN jobs):
Tue Sep 04 14:18:52 2012
ARCH: Possible network disconnect with primary database
So, this is all nice and everything but what happened?
Answer: Our storage engineer started researching the issue and found that our IB switches and other Exadata components had no issues, but when he checked our 7420 storage array out it was a completely different matter all together. There were no issues with head 1 of the array, but when he went to check on head 2 he was unable to log in through GUI or ssh with also a simple df command hanging on an NFS mount coming from that head. When he performed a takeover to move NFS control over to head 1, head 2 complained about lacking resources when attempting to fork something and the only way he could force a takeover was to power down head 2. What I learned by researching for this article can be found here about what a 7420 head configuration could look like in Illustration 20 and Illustration 21 and the following paragraph as well.
Thursday, August 2, 2012
OS Changes - Commands
So we've upgraded from SUN Solaris to RedHat Linux, and I've found some key differences so far. The command prstat no longer really works well to find out how healthy your server/CPUs are, as it needs to be combined with the new top command to get a fuller understanding of how healthy your system is and this site will help you understand why there is a difference between not only the commands but how the OS is operating as well.
To investigate printing issues I grew accustomed to the lpstat command combined with the -lp switches so that a command lpstat -lp PRINTERNAME would give me information as to how the printer was connected to our system and if the printer queue was up or not. I've found out that issuing this command in a RedHat Linux environment doesn't provide that old type of feedback, so we've had to alter our approach and instead use the -a switch so that the entire command looks like lpstat -a PRINTERNAME now.
To investigate printing issues I grew accustomed to the lpstat command combined with the -lp switches so that a command lpstat -lp PRINTERNAME would give me information as to how the printer was connected to our system and if the printer queue was up or not. I've found out that issuing this command in a RedHat Linux environment doesn't provide that old type of feedback, so we've had to alter our approach and instead use the -a switch so that the entire command looks like lpstat -a PRINTERNAME now.
Subscribe to:
Posts (Atom)