10g RAC的dataguad异常一则

今天重启一个dataguad的时候,无法启动MRP进程,报错如下

Wed Mar  9 15:32:55 2016ALTER DATABASE recover managed standby database using current logfile disconnectWed Mar  9 15:32:55 2016Attempt to start background Managed Standby Recovery process (ORCL1)MRP0 started with pid=30, OS id=9267Wed Mar  9 15:32:55 2016MRP0: Background Managed Standby Recovery process started (ORCL1)Wed Mar  9 15:33:00 2016Managed Standby Recovery starting Real Time ApplyWed Mar  9 15:33:00 2016MRP0: Some datafile enqueues are still held! Retry recovery...Wed Mar  9 15:33:00 2016Errors in file /home/oracle/oracle/admin/ORCL/bdump/orcl1_mrp0_9267.trc:ORA-01124: Message 1124 not found; No message file for product=RDBMS, facility=ORA; arguments: [1]ORA-01110: Message 1110 not found; No message file for product=RDBMS, facility=ORA; arguments: [1] [+DATA/orcl/system01.dbf]Wed Mar  9 15:33:00 2016Managed Standby Recovery not using Real Time ApplyWed Mar  9 15:33:00 2016Errors in file /home/oracle/oracle/admin/ORCL/bdump/orcl1_mrp0_9267.trc:ORA-01124: Message 1124 not found; No message file for product=RDBMS, facility=ORA; arguments: [1]ORA-01110: Message 1110 not found; No message file for product=RDBMS, facility=ORA; arguments: [1] [+DATA/orcl/system01.dbf]Wed Mar  9 15:33:20 2016MRP0: Background Media Recovery terminated with error 1153Wed Mar  9 15:33:20 2016Errors in file /home/oracle/oracle/admin/ORCL/bdump/orcl1_mrp0_9267.trc:ORA-01153: Message 1153 not found; No message file for product=RDBMS, facility=ORAWed Mar  9 15:33:20 2016Errors in file /home/oracle/oracle/admin/ORCL/bdump/orcl1_mrp0_9267.trc:

这里说一下环境

主库是10g 2节点RAC,备库同样是2个节点10g RAC,但是只有节点1做日志恢复

事故经过:

昨天由于备库的存储故障(有raid,数据没异常),就停了crs让存储工程师维护,维护以后,新磁盘需要同步数据,所以就今天启动dg,间隔大概24小时

今天维护好了,需要启动,2节点分别执行了/etc/init.d/init.crs start

都知道这个命令会启动asm及db的实例,但是我给忽略了

在节点1起来以后,看日志,数据库open到了readonly,然后执行了shutdown,并启动到mount执行revoer,但是这里报错,信息如上

其实现在很清晰的能看到,1节点启动MRP进程准备恢复,但由于2节点也处于readonly模式,数据文件已经被占用了.所以节点1的MRP没法恢复数据文件.就被关闭了

解决办法:

shutdown 2节点db即可