问题描述:

使用hive load hdfs文件时报错:

FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed due to task failures: Cannot obtain block length for LocatedBlock{BP-1984322900-192.168.102.3-1594185446267:blk_1180034904_106295094; getBlockSize()=4179; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[192.168.102.11:9866,DS-cb5a2e07-20e9-45fd-869b-5d8b4ad170a4,DISK], DatanodeInfoWithStorage[192.168.102.9:9866,DS-74706bce-bb23-4aaf-a6eb-ceaa9bdbf38c,DISK], DatanodeInfoWithStorage[192.168.102.5:9866,DS-57f122fb-b6ca-437c-a52e-5f81efdd239c,DISK]]}
22/02/23 16:31:17 ERROR ql.Driver: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed due to task failures: Cannot obtain block length for LocatedBlock{BP-1984322900-192.168.102.3-1594185446267:blk_1180034904_106295094; getBlockSize()=4179; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[192.168.102.11:9866,DS-cb5a2e07-20e9-45fd-869b-5d8b4ad170a4,DISK], DatanodeInfoWithStorage[192.168.102.9:9866,DS-74706bce-bb23-4aaf-a6eb-ceaa9bdbf38c,DISK], DatanodeInfoWithStorage[192.168.102.5:9866,DS-57f122fb-b6ca-437c-a52e-5f81efdd239c,DISK]]}

分析问题:

可得知hdfs文件块出现异常,Cannot obtain block length for LocatedBlock,无法获取块文件长度信息

猜测是因为昨日yarn重启导致hdfs文件未关闭写状态

解决问题:

对hive load hdfs文件的地址执行检查命令

hdfs fsck /flume/ex_trade/date=20220222 -openforwrite

Connecting to namenode via http://master1.zd.prod:9870/fsck?ugi=jay&openforwrite=1&path=%2Fflume%2Fex_trade%2Fdate%3D20220222
FSCK started by jay (auth:SIMPLE) from /192.168.102.11 for path /flume/ex_trade/date=20220222 at Wed Feb 23 16:37:10 CST 2022
/flume/ex_trade/date=20220222/ex_trade_.1645508080614 1494 bytes, replicated: replication=3, 1 block(s), OPENFORWRITE: /flume/ex_trade/date=20220222/ex_trade_.1645519019856 4179 bytes, replicated: replication=3, 1 block(s), OPENFORWRITE:
Status: HEALTHY
 Number of data-nodes:  6
 Number of racks:               1
 Total dirs:                    1
 Total symlinks:                0

Replicated Blocks:
 Total size:    51340251 B
 Total files:   7
 Total blocks (validated):      7 (avg. block size 7334321 B)
 Minimally replicated blocks:   5 (71.42857 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     2.142857
 Missing blocks:                0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Blocks queued for replication: 0

Erasure Coded Block Groups:
 Total size:    0 B
 Total files:   0
 Total block groups (validated):        0
 Minimally erasure-coded block groups:  0
 Over-erasure-coded block groups:       0
 Under-erasure-coded block groups:      0
 Unsatisfactory placement block groups: 0
 Average block group size:      0.0
 Missing block groups:          0
 Corrupt block groups:          0
 Missing internal blocks:       0
 Blocks queued for replication: 0
FSCK ended at Wed Feb 23 16:37:10 CST 2022 in 2 milliseconds


The filesystem under path '/flume/ex_trade/date=20220222' is HEALTHY

发现文件 ex_trade.1645508080614和 ex_trade.1645519019856未关闭

执行修复文件命令

hdfs debug recoverLease -path /flume/ex_trade/date=20220222/ex_trade_.1645508080614 -retries 3

hdfs debug recoverLease -path /flume/ex_trade/date=20220222/ex_trade_.1645519019856 -retries 3

重新使用hive load hdfs文件,解决问题。