12c rac 实例无法启动之磁盘组空间耗尽

一、环境描述
oracle 12c rac

二、处理过程

1、实例启动失败SYS@orcl1> startupORACLE instance started.Total System Global Area 1241513984 bytesFixed Size                  2923872 bytesVariable Size             620757664 bytesDatabase Buffers          603979776 bytesRedo Buffers               13852672 bytesDatabase mounted.ORA-03113: end-of-file on communication channelProcess ID: 8482Session ID: 1 Serial number: 48907

2、查看日志部分省略...............Starting background process ARC2Starting background process ARC3ARC2 started with pid=44, OS id=8737Unable to create archive log file '+DATADG'Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_8659.trc:ORA-19504: failed to create file "+DATADG"ORA-17502: ksfdcre:4 Failed to create file +DATADGORA-15041: diskgroup "DATADG" space exhaustedARCH: Error 19504 Creating archive log file to '+DATADG'Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_8659.trc:ORA-16038: log 2 sequence# 108 cannot be archivedORA-19504: failed to create file ""ORA-00312: online log 2 thread 1: '+DATADG/ORCL/ONLINELOG/group_2.263.954539539'ARC3 started with pid=45, OS id=8739System state dump requested by (instance=1, osid=8659), summary=[abnormal instance termination].System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_diag_8552_20171231165944.trcUSER (ospid: 8659): terminating the instance due to error 16038Dumping diagnostic data in directory=[cdmp_20171231165944], requested by (instance=1, osid=8659), summary=[abnormal instance termination].Instance terminated by USER, pid = 8659

3、确认磁盘组空间大小$asmcmd -pASMCMD [+] > lsdgState    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  NameMOUNTED  EXTERN  N         512   4096  1048576     10240     5763                0            5763              0             Y  CRSDG/MOUNTED  EXTERN  N         512   4096  1048576      6144       13                0              13              0             N  DATADG/ASMCMD [+] > 

4、处理思路1、使用rm命令清理部分归档日志。2、启动数据库3、使用rman crosscheck archivelog all;4、扩磁盘组

5、注意需要使用rman进行crosscheck ,然后再删除归档日志在控制文件中的记录。RMAN> crosscheck archivelog all;using target database control file instead of recovery catalogallocated channel: ORA_DISK_1channel ORA_DISK_1: SID=66 instance=orcl2 device type=DISKvalidation failed for archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_50.280.962120097 RECID=11 STAMP=962120097validation failed for archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_51.282.962121355 RECID=13 STAMP=962074554validation failed for archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_52.283.962121355 RECID=14 STAMP=962074555validation failed for archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_53.288.962121385 RECID=19 STAMP=962121385validation failed for archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_54.289.962184681 RECID=20 STAMP=962184682validation failed for archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_55.291.962185743 RECID=22 STAMP=962185742validation failed for archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_56.292.962188105 RECID=23 STAMP=962188106validation failed for archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_57.295.962188239 RECID=26 STAMP=962188238validation failed for archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_58.297.962188923 RECID=28 STAMP=962188924部分省略..................................validation succeeded for archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_31/thread_2_seq_109.288.964199163 RECID=132 STAMP=964199165validation succeeded for archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_31/thread_2_seq_110.325.964199595 RECID=135 STAMP=964199601Crosschecked 124 objectsRMAN> RMAN> delete expired archivelog all;released channel: ORA_DISK_1allocated channel: ORA_DISK_1channel ORA_DISK_1: SID=66 instance=orcl2 device type=DISKList of Archived Log Copies for database with db_unique_name ORCL=====================================================================Key     Thrd Seq     S Low Time            ------- ---- ------- - --------------------11      1    50      X 07-DEC-2017 15:24:55        Name: +DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_50.280.96212009713      1    51      X 07-DEC-2017 15:34:57        Name: +DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_51.282.96212135514      1    52      X 07-DEC-2017 02:55:53        Name: +DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_52.283.96212135519      1    53      X 07-DEC-2017 15:56:20部分省略 ...........................................51      2    66      X 08-DEC-2017 07:41:03        Name: +DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_66.319.96222588353      2    67      X 08-DEC-2017 20:57:57        Name: +DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_67.321.96222588557      2    68      X 09-DEC-2017 10:16:05        Name: +DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_68.325.96227378559      2    69      X 09-DEC-2017 10:16:23        Name: +DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_69.327.962274133Do you really want to delete the above objects (enter YES or NO)? yesdeleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_50.280.962120097 RECID=11 STAMP=962120097deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_51.282.962121355 RECID=13 STAMP=962074554deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_52.283.962121355 RECID=14 STAMP=962074555deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_53.288.962121385 RECID=19 STAMP=962121385deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_54.289.962184681 RECID=20 STAMP=962184682deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_55.291.962185743 RECID=22 STAMP=962185742deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_56.292.962188105 RECID=23 STAMP=962188106deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_57.295.962188239 RECID=26 STAMP=962188238deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_58.297.962188923 RECID=28 STAMP=962188924deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_59.298.962206427 RECID=29 STAMP=962206427deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_60.300.962208543 RECID=31 STAMP=962208544deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_61.302.962223761 RECID=48 STAMP=962177843deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_62.305.962223817 RECID=36 STAMP=962223817deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_63.306.962223899 RECID=37 STAMP=962223899deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_64.309.962224067 RECID=40 STAMP=962224066deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_65.310.962224095 RECID=41 STAMP=962224094deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_66.311.962224101 RECID=42 STAMP=962224100deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_67.314.962224111 RECID=45 STAMP=962224111deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_68.316.962224463 RECID=47 STAMP=962224462deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_69.317.962224863 RECID=49 STAMP=962224862deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_70.320.962225885 RECID=52 STAMP=962225884deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_71.322.962273773 RECID=54 STAMP=962273772deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_72.323.962273775 RECID=55 STAMP=962273775deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_73.324.962273779 RECID=56 STAMP=962273783deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_74.326.962273785 RECID=58 STAMP=962273785deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_47.281.962120099 RECID=12 STAMP=962073300deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_48.284.962121379 RECID=15 STAMP=962121380deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_49.285.962121381 RECID=16 STAMP=962121380deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_50.286.962121383 RECID=17 STAMP=962121382部分省略 ....................deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_66.319.962225883 RECID=51 STAMP=962225883deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_67.321.962225885 RECID=53 STAMP=962225884deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_68.325.962273785 RECID=57 STAMP=962273784deleted archived logarchived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_69.327.962274133 RECID=59 STAMP=962274134Deleted 48 EXPIRED objects

最近也是有客户遇到类似的问题，通过操作系统命令删除了部分归档，然后备份脚本备份失败，发现是因为这个原因引起的，优化一下脚本，添加进去crosscheck 命令，以后就不会出现类似问题了。