hadoop hbase无法启动 2
今天又碰到的问题：
Not able to place enough replicas
2015-02-08 18:35:43,978 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:web cause:java.io.IOException: File /hbase/aaaa/fbade52c03733ec2aa6d5355052d9c89/recovered.edits/0000000000004181150.temp could only be replicated to 0 nodes, instead of 1
2015-02-08 18:35:43,978 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 8020, call addBlock(/hbase/aaaa/fbade52c03733ec2aa6d5355052d9c89/recovered.edits/0000000000004181150.temp, DFSClient_hb_m_m66,60000,1423391732102, null) from 192.168.0.66:42030: error: java.io.IOException: File /hbase/aaaa/fbade52c03733ec2aa6d5355052d9c89/recovered.edits/0000000000004181150.temp could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /hbase/aaaa/fbade52c03733ec2aa6d5355052d9c89/recovered.edits/0000000000004181150.temp could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)
2015-02-08 18:35:44,014 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place enough replicas, still in need of 3 to reach 3
Not able to place enough replicas
2015-02-08 18:35:44,014 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:web cause:java.io.IOException: File /hbase/.META./1028785192/recovered.edits/0000000000004125376.temp could only be replicated to 0 nodes, instead of 1
2015-02-08 18:35:44,014 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 8020, call addBlock(/hbase/.META./1028785192/recovered.edits/0000000000004125376.temp, DFSClient_hb_m_m66,60000,1423391732102, null) from 192.168.0.66:42030: error: java.io.IOException: File /hbase/.META./1028785192/recovered.edits/0000000000004125376.temp could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /hbase/.META./1028785192/recovered.edits/0000000000004125376.temp could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)
重启了n多边都没有效果。
删除之，发现你启动hmaster后该文件依然存在，删除后又恢复了。
干脆连着目录一起删除，ok
./hadoop fs -rmr /hbase/aaaa/8aab6d49ca2235530d7bf992dcb15e55
./hadoop fs -rmr /hbase/.META./1028785192
再次启动hmaser ok
./hbase-daemon.sh start master
jps查看 ok hmaster启动起来了。
咳，hadoop啊，想不暴力都难！！
该死，原来以为都ok了，同事测试的时候发现问题：
./hbase shell
list
可以看到表
可是扫描表的时候说表不存在。
立马想到是hbase本身出了问题。
果断修复
./hbase hbck
发现存在空洞不连续的情况。
./hbase hbck -fixMeta -fixAssignments
./hbase hbck -repair
修复完成后，再次./hbase hbck 发现还是有问题。
再一次./hbase hbck -repair
这次居然都ok了。
重新启动shell，扫描表。现在ok.了
重启hbase发现，hbase进程无法关闭，关闭客户端进程，ok了。
再次启动，突然发现包其中一台空间满了，晕，果断修复。
最后启动应用，居然调用都正常了。
真是峰回路转，变化万千。 心里暗暗骂一个，shit.
