While we expect that our Exadata system may struggle with regard to OS load metrics when the free memory on the machine is running low, we weren't expecting to see the same behavior when it is has recovered a good deal of memory as well. After a very busy day fighting against this type of trend, our DBAs went to Oracle asking why they were seeing "kernel: oracle: page allocation failure. order:5, mode: 0xd0" in the system messages. To my surprise, Oracle came back with MOS note ID 1546861.1 titled "[Linux OS] System Hung with Large Numbers of Page Allocation Failures with "order:5" on Exadata Environments" and told us that we needed to change Linux OS parameters (which have been setup this way from the beginning as far as I can tell) to make sure our system is more stable when it gets low on free memory.
This is a very good document to review, even if you aren't having the same type of OS problems, as it has several suggestions for improving performance using Infiniband Interface changes too!