hive从本地导入数据时出现「Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask」错误

现象

通过load data local导入本地文件时报无法导入的错误

hive> load data local inpath '/home/hadoop/out/mid_test.txt' overwrite into table my_mid.mid_test partition (etl_date=20190101);
Loading data to table my_mid.mid_test partition (etl_date=20190101)
Failed with exception Unable to move source file:/home/hadoop/out/mid_test.txt to destination hdfs://namenode01.my.com/my_etl/dw/mid/mid_test/etl_date=20190101/mid_test.txt
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask

相关操作

第一次导入时时成功的，后来发现源文件行数有问题，重新复制了一个文本文件到本地导入目录，后续再导入时就出现了上述问题。

解决办法

查询hive日志获得更详细的信息，日志的常见位置为/tmp/${user}/hive.log中

2019-02-19 09:38:21,503 ERROR [main]: exec.Task (SessionState.java:printError(960)) - Failed with exception Unable to move source file:/home/hadoop/out/mid_test.txt to destination hdfs://namenode01.my.com/my_etl/dw/mid/mid_test/etl_date=20190101/mid_test.txt
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source file:/home/hadoop/out/mid_test.txt to destination hdfs://namenode01.my.com/my_etl/dw/mid/mid_test/etl_date=20190101/mid_test.txt
        at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2644)
        at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:2911)
        at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1398)
        at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1324)
        at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:438)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.hadoop.fs.ChecksumException: Checksum error: file:/home/hadoop/out/mid_test.txt at 0 exp: -827044509 got: 624370567
        at org.apache.hadoop.fs.FSInputChecker.verifySums(FSInputChecker.java:323)
        at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:279)
        at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
        at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
        at java.io.DataInputStream.read(DataInputStream.java:100)
        at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
        at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
        at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1965)
        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1933)
        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1898)
        at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2637)
        ... 23 more
```shell

> 确定是校验失败，找到实际原因是旧文件crc文件与新拷贝文件的校验值不一致。

## 检查是否有crc文件

```shell
[hadoop@my-17 out]$ls -al
总用量 112720
drwxrwxr-x   2 hadoop hadoop     4096 2月  19 09:31 .
drwx------. 39 hadoop hadoop     4096 2月  19 09:57 ..
-rw-r--r--   1 hadoop hadoop  1595242 2月  19 09:12 mid_test.txt
-rw-r--r--   1 hadoop hadoop     3128 2月  19 08:22 .mid_test.txt.crc

确实有此文件，将crc文件删除即可。

后续导入即成功。