今天没事重启了一下rac ，原本以为会没啥问题的，没想到rac2，rac3 节点都正常起来了，但rac1 始终起不来，查看日志，如下:
[ctssd(3088)]CRS-2402:The Cluster Time Synchronization Service aborted on host rac1. Details at in /u01/app/11.2.0/grid_1/log/rac1/ctssd/octssd.log. 2017-03-09 16:06:40.950: [ctssd(3106)]CRS-2405:The Cluster Time Synchronization Service on host rac1 is shutdown by user 2017-03-09 16:06:45.255: [ctssd(3114)]CRS-2401:The Cluster Time Synchronization Service started on host rac1. 2017-03-09 16:06:45.256: [ctssd(3114)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac2. 2017-03-09 16:06:48.405: [ctssd(3114)]CRS-2404:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid_1/log/rac1/ctssd/octssd.log. 2017-03-09 16:06:48.980: [ctssd(3114)]CRS-2402:The Cluster Time Synchronization Service aborted on host rac1. Details at in /u01/app/11.2.0/grid_1/log/rac1/ctssd/octssd.log. 2017-03-09 16:06:49.995: [ohasd(2308)]CRS-2807:Resource 'ora.asm' failed to start automatically. 2017-03-09 16:06:49.995: [ohasd(2308)]CRS-2807:Resource 'ora.crsd' failed to start automatically. 2017-03-09 16:06:49.995: [ohasd(2308)]CRS-2807:Resource 'ora.ctssd' failed to start automatically. 2017-03-09 16:06:49.995: [ohasd(2308)]CRS-2807:Resource 'ora.evmd' failed to start automatically.
尝试手动重启crs，发现最后还是起不来，最好在看一下报错信息，发现The Cluster Time Synchronization Service aborted on host rac1。
于是根据报错信息，发现rac1的时间和rac2，rac3 已经不同了，于是赶紧把rac1的时间改回来，
[root@rac1 ~]# date -s '2017-03-05 13:50:12'
[root@rac1 ~]# hwclock -w
然后手动重启
[root@rac1 bin]# ./crsctl stop crs -f CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1' CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1' CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1' CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'rac1' CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.crf' on 'rac1' CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1' CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1' CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed CRS-4133: Oracle High Availability Services has been stopped. [root@rac1 bin]# ./crsctl start crs CRS-4123: Oracle High Availability Services has been started. [root@rac1 bin]# ps -ef |grep pmon grid 4130 1 0 13:53 ? 00:00:00 asm_pmon_+ASM1 root 4323 2970 0 13:53 pts/0 00:00:00 grep pmon [root@rac1 bin]#
