UPDATE: (11/30) Looks as if the corruption began just after our datacenter cutover to our dark fiber run down to Norman (standby site). Let’s see what the network guys come up with.
Redo log corruption occuring on the DR site on our physical standby. This is either occuring in flight (and we have no firewalls in place between our primary and standby site) or it’s occuring on the standby storage tier. The reason I say this is because I fixed the issue by re-applying the archive logs from the standby site after figuring out which redo logs in which thread occured before the last good SCN stamp on the physical standby (which can be found in the alert log as seen below).
Below are the notes from yesterdays exercise…
Seriously with the block corruption on PRDFT?????? just in….. hot off the press in the alert log on the DR1 instance:
Tue Nov 27 13:39:42 2012 Errors in file /opt/app/oracle/diag/rdbms/prdftdr/PRDFTDR1/trace/PRDFTDR1_pr05_19715.trc (incident=81053): ORA-00600: internal error code, arguments: [kdourp_inorder2], , , , , , , , , , ,  Incident details in: /opt/app/oracle/diag/rdbms/prdftdr/PRDFTDR1/incident/incdir_81053/PRDFTDR1_pr05_19715_i81053.trc