Stopping/Starting RAC instances for a rolling patch

You’ve done a rolling patch for a RAC test environment right?  And some of you have multiple instances running on the same node for testing, development, training, whatever… it can be a pain taking all those instances up and down.

Patching has come a long way with Oracle and the rolling patch is a beautiful thing compared to what it used to be, however, it’s still a pain to take down each instance on a particular node especially when you have more than a few when doing a rolling patch.

So… I’ve created a customizable script for doing just that.  It will take down the instances you want on each node and restart them before moving on to the next node.  This script was built with our PSoft environment in mind.

RAC node startup problems after RHEL patch

ENV: Red Hat Enterprise Linux Server release 5.8 (Tikanga), EE

As most organizations, we have maintenance, and our maintance occurred this last Saturday which usually entails applying RHEL OS patches to our RAC clusters.  We don’t usually encounter problems…. but we did this time.  Now, normally (actually always) we don’t relink any of our oracle/clusterware after applying OS patches… that apparently will get you into trouble.

After applying the latest round of RHEL patches on the 1st node of our test environment, the CRS wouldn’t come up…. the only things you saw running when doing a ps was

ohasd.bin reboot, init.ohasd run, cssdmonitor, & orarootagent.bin

when checking the crs, you’d see this:

Continue reading