问题描述:
使用hive load hdfs文件时报错:
FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed due to task failures: Cannot obtain block length for LocatedBlock{BP-1984322900-192.168.102.3-1594185446267:blk_1180034904_106295094; getBlockSize()=4179; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[192.168.102.11:9866,DS-cb5a2e07-20e9-45fd-869b-5d8b4ad170a4,DISK], DatanodeInfoWithStorage[192.168.102.9:9866,DS-74706bce-bb23-4aaf-a6eb-ceaa9bdbf38c,DISK], DatanodeInfoWithStorage[192.168.102.5:9866,DS-57f122fb-b6ca-437c-a52e-5f81efdd239c,DISK]]}
22/02/23 16:31:17 ERROR ql.Driver: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed due to task failures: Cannot obtain block length for LocatedBlock{BP-1984322900-192.168.102.3-1594185446267:blk_1180034904_106295094; getBlockSize()=4179; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[192.168.102.11:9866,DS-cb5a2e07-20e9-45fd-869b-5d8b4ad170a4,DISK], DatanodeInfoWithStorage[192.168.102.9:9866,DS-74706bce-bb23-4aaf-a6eb-ceaa9bdbf38c,DISK], DatanodeInfoWithStorage[192.168.102.5:9866,DS-57f122fb-b6ca-437c-a52e-5f81efdd239c,DISK]]}
分析问题:
可得知hdfs文件块出现异常,Cannot obtain block length for LocatedBlock,无法获取块文件长度信息
猜测是因为昨日yarn重启导致hdfs文件未关闭写状态
解决问题:
对hive load hdfs文件的地址执行检查命令
hdfs fsck /flume/ex_trade/date=20220222 -openforwrite
Connecting to namenode via http://master1.zd.prod:9870/fsck?ugi=jay&openforwrite=1&path=%2Fflume%2Fex_trade%2Fdate%3D20220222
FSCK started by jay (auth:SIMPLE) from /192.168.102.11 for path /flume/ex_trade/date=20220222 at Wed Feb 23 16:37:10 CST 2022
/flume/ex_trade/date=20220222/ex_trade_.1645508080614 1494 bytes, replicated: replication=3, 1 block(s), OPENFORWRITE: /flume/ex_trade/date=20220222/ex_trade_.1645519019856 4179 bytes, replicated: replication=3, 1 block(s), OPENFORWRITE:
Status: HEALTHY
Number of data-nodes: 6
Number of racks: 1
Total dirs: 1
Total symlinks: 0
Replicated Blocks:
Total size: 51340251 B
Total files: 7
Total blocks (validated): 7 (avg. block size 7334321 B)
Minimally replicated blocks: 5 (71.42857 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 2.142857
Missing blocks: 0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Blocks queued for replication: 0
Erasure Coded Block Groups:
Total size: 0 B
Total files: 0
Total block groups (validated): 0
Minimally erasure-coded block groups: 0
Over-erasure-coded block groups: 0
Under-erasure-coded block groups: 0
Unsatisfactory placement block groups: 0
Average block group size: 0.0
Missing block groups: 0
Corrupt block groups: 0
Missing internal blocks: 0
Blocks queued for replication: 0
FSCK ended at Wed Feb 23 16:37:10 CST 2022 in 2 milliseconds
The filesystem under path '/flume/ex_trade/date=20220222' is HEALTHY
发现文件 ex_trade.1645508080614和 ex_trade.1645519019856未关闭
执行修复文件命令
hdfs debug recoverLease -path /flume/ex_trade/date=20220222/ex_trade_.1645508080614 -retries 3
hdfs debug recoverLease -path /flume/ex_trade/date=20220222/ex_trade_.1645519019856 -retries 3
重新使用hive load hdfs文件,解决问题。