####场景
从hadoop2.6传数据到hadoop2.7，遇到了下面问题。
####报错信息
Caused by: java.io.IOException: Couldn't run retriable-command: Copying webhdfs://132.121.97.48:50070/glabel/data/201407/20140710/signal-dwel l/part-00000.bz2 to hdfs://c9/tmp/hive_exportdir/data/201407/20140710/signal-dwell/part-00000.bz2 at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:283) ... 10 more Caused by: java.io.IOException: Check-sum mismatch between webhdfs://132.121.97.48:50070/glabel/data/201407/20140710/signal-dwell/part-00000. bz2 and hdfs://c9/tmp/hive_exportdir/.distcp.tmp.attempt_1492021243934_11314_m_000019_0. at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more
####两种方法
1. 修改hdfs-site.xml
dfs.checksum.type = CRC32
2.加 -skipcrccheck参数
hadoop distcp -update -skipcrccheck hftp://X.X.X.X:50070/user/dd_edw/warehouse/X/dt=2014-04-10 /user/impala/warehouse/1
3.加上-Ddfs.checksum.type=CRC32参数，保持两边一致
hadoop distcp -Dipc.client.fallback-to-simple-auth-allowed=true -Ddfs.checksum.type=CRC32 webhdfs://132.121.97.48:50070/glabel/data /tmp/hive_exportdir/ >glabel_data_distcp.log 2>&1 &
