spark 从HIVE读数据导入hbase中发生空指针(java.lang.NullPointerException)问题的解决

    xiaoxiao2022-06-28  21

    陆续好多人会问,在写入Hbase的时候总是会出现空指针的问题,而检查程序,看起来一点也没有错。

    如报的错误大致如下:

    Error: application failed with exception java.lang.RuntimeException: java.lang.NullPointerException     at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:209)     at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:288)     at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:268)     at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140)     at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:135)     at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:802)     at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:200)     at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:85)     at org.apache.hadoop.hbase.client.MetaScanner.allTableRegions(MetaScanner.java:310)     at org.apache.hadoop.hbase.client.HTable.getRegionLocations(HTable.java:666)     at org.apache.hadoop.hbase.util.RegionSizeCalculator.<init>(RegionSizeCalculator.java:79)     at org.apache.hadoop.hbase.util.RegionSizeCalculator.<init>(RegionSizeCalculator.java:64)     at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:160)     at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:98)     at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:220)     at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:218)     at scala.Option.getOrElse(Option.scala:120)     at org.apache.spark.rdd.RDD.partitions(RDD.scala:218)     at org.apache.spark.SparkContext.runJob(SparkContext.scala:1335)     at org.apache.spark.rdd.RDD.count(RDD.scala:925)     at HBaseTest$.main(HBaseTest.scala:27)     at HBaseTest.main(HBaseTest.scala)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     at java.lang.reflect.Method.invoke(Method.java:606)     at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:367)     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:77)     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.NullPointerException     at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.getMetaReplicaNodes(ZooKeeperWatcher.java:269)     at org.apache.hadoop.hbase.zookeeper.MetaRegionTracker.blockUntilAvailable(MetaRegionTracker.java:241)     at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:62)     at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1213)     at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1174)     at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:294)     at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:130)     at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:55)     at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:201)     ... 28 more

    今天就将这种问题解决方法写成博客,其实解决办法非常简单。其问题主要发生在这行代码当中。

    hbaseContext.bulkPut[(Array[Byte], Array[(Array[Byte], Array[Byte], Array[Byte])])](rdd,       tableName,       (putRecord) => {         val put = new Put(putRecord._1)         putRecord._2.foreach((putValue) => put.add(putValue._1, putValue._2, putValue._3))         put       },       true); }

    这个问题,主要原因在于从HiveContext中访问的DataFrame中,遍历的某些行里面putRecord中的某一个单元值为NULL,所以就会抛出这种异常。

    因此在put.add的时候首先需要进行判断一下。

    如 putRecord.IsNullAt(index),这样进行判断,如果为NULL值,简单设个特定的字符串,马上什么问题全部解决。

    转载请注明原文地址: https://ju.6miu.com/read-1124538.html

    最新回复(0)