RAC node startup problems after RHEL patch

ENV: Red Hat Enterprise Linux Server release 5.8 (Tikanga), EE

As most organizations, we have maintenance, and our maintance occurred this last Saturday which usually entails applying RHEL OS patches to our RAC clusters.  We don’t usually encounter problems…. but we did this time.  Now, normally (actually always) we don’t relink any of our oracle/clusterware after applying OS patches… that apparently will get you into trouble.

After applying the latest round of RHEL patches on the 1st node of our test environment, the CRS wouldn’t come up…. the only things you saw running when doing a ps was

ohasd.bin reboot, init.ohasd run, cssdmonitor, & orarootagent.bin

when checking the crs, you’d see this:

Continue reading