Updated PSoft clone scripts

The scripts for the PeopleSoft clone have been updated and tested. The two scripts are
located in the dba_scripts tab.

The 1st script is run by the user Oracle and the 2nd script is run by the ASM/Clusterware
owner, Grid. The 2nd script is running every 60 seconds in the user Grid’s crontab… it
polls for a file that the 1st script writes which triggers the 2nd script to run.

OakTable World 2012 slides & videos

I had the privilege of attending some of this year’s OakTable World while attending OOW.  Absolutely fantastic!  If you get the chance to attend OOW in the future, biggie size your trip by spending much of your time at OakTable World and skip some of the OOW sessions.  You won’t regret it.  Check out some of the great slide decks and videos at the link below:


ORA-10562 on physical standby

UPDATE: (11/30)  Looks as if the corruption began just after our datacenter cutover to our dark fiber run down to Norman (standby site).  Let’s see what the network guys come up with.

Redo log corruption occuring on the DR site on our physical standby. This is either occuring in flight (and we have no firewalls in place between our primary and standby site) or it’s occuring on the standby storage tier. The reason I say this is because I fixed the issue by re-applying the archive logs from the standby site after figuring out which redo logs in which thread occured before the last good SCN stamp on the physical standby (which can be found in the alert log as seen below).

Below are the notes from yesterdays exercise…

Seriously with the block corruption on PRDFT??????  just in….. hot off the press in the alert log on the DR1 instance:

Tue Nov 27 13:39:42 2012 Errors in file /opt/app/oracle/diag/rdbms/prdftdr/PRDFTDR1/trace/PRDFTDR1_pr05_19715.trc  (incident=81053): ORA-00600: internal error code, arguments: [kdourp_inorder2], [97], [97], [44], [36], [], [], [], [], [], [], [] Incident details in: /opt/app/oracle/diag/rdbms/prdftdr/PRDFTDR1/incident/incdir_81053/PRDFTDR1_pr05_19715_i81053.trc

Continue reading

ORA-00245: control file backup operation failed

We’re writing our snapshot controlfiles to a “non-shared” device in our RAC environment (?/dbs/snapcf_DBNAME.f) and this apparently can be a problem… but only when using RAC. 

Per DocId 1473914.1, “…any instances in the cluster may write to the snapshot / backup controlfile. Hence the Snapshot controlfile need to be visible to all instances.”  This hasn’t been a problem for us in the past, but apparently, since “any” instance can write to the snapshot controlfile, it finally did and threw the error.

An easy fix really, just configure RMAN to write to a shared device, in our case ASM, as seen below (see DocId 1472171.1):


RMAN-08591: invalid archived log deletion policy

Not documented that I could find, however, it appears that when using an RMAN configuration to ensure archivelogs are applied to the standby before deleting them, RMAN relies on the data guard broker to take care of this check.
If you were to turn off the log transport via dgmgrl and run your RMAN backup, you would see the following error at the end of the output when deleting archivelogs:
RMAN-08591: WARNING: invalid archived log deletion policy
Looks as though RMAN depends on DMON when using this policy in the configuration:

RMAN ORA-01008: not all variables bound

Ran into this bug (info here) over the weekend that affects when logging into RMAN:

DBGSQL:     TARGET> select count(*) into :dbstate from v$parameter where lower(name) = ‘_dummy_instance’ and upper(value) = ‘TRUE’

DBGSQL:        sqlcode = 1008

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-00554: initialization of internal recovery manager package failed

RMAN-06003: ORACLE error from target database:

ORA-01008: not all variables bound

The “possible” workaround from Oracle did indeed work:
SQL> alter system flush shared_pool;

Column formatting in a CASE statement

I have a daily RMAN report that I send to myself and I wanted to color code the duration of the backup… for instance, backups taking less than 100 minutes were green, between 100 and 300 minutes, orange, over that red.

Currently, I’m using the following to format the column in the HTML output:

column “B/Up Duration in mins” ON HEADING ‘B/Up Duration!in mins’ format 999.99

Well, that works fine until I put that column into a CASE statement.  It still splits the heading on two lines but no longer gives me the number format I wanted.  Also, I needed to alias the column so the column formatting would take affect… to get around the two problems, here’s what I did.  First, I format the column within the CASE statement and then put the alias at the end of the CASE statement as seen below:


    WHEN elapsed_seconds/60 < 100 then

      ‘<span class=”threshold-ok”>’||rpad(trunc(elapsed_seconds/60,2),4,0)||'</span>’

   WHEN elapsed_seconds/60 between 100 and 300 then

       ‘<span class=”threshold-warning”>’||rpad(trunc(elapsed_seconds/60,2),5,0)||'</span>’


        ‘<span class=”threshold-critical”>’||rpad(trunc(elapsed_seconds/60,2),5,0)||'</span>’

   END as “B/Up Duration in mins”


Stopping/Starting RAC instances for a rolling patch

You’ve done a rolling patch for a RAC test environment right?  And some of you have multiple instances running on the same node for testing, development, training, whatever… it can be a pain taking all those instances up and down.

Patching has come a long way with Oracle and the rolling patch is a beautiful thing compared to what it used to be, however, it’s still a pain to take down each instance on a particular node especially when you have more than a few when doing a rolling patch.

So… I’ve created a customizable script for doing just that.  It will take down the instances you want on each node and restart them before moving on to the next node.  This script was built with our PSoft environment in mind.