Elasticsearch on LeoChu Space

ES映射hive数据类型date无法解析

Sat, 06 Jul 2024 00:00:00 +0000

在es中数据类型为date：

"addTime": {
  "format": "yyyy-MM-dd HH:mm:ss",
  "type": "date"
}

在hive建映射表

CREATE EXTERNAL TABLE hive_es.cty_test1(
addTime date
)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'cty_test/cty_test',
'es.nodes'='172.16.98.113,172.16.98.149,172.16.98.150,172.16.98.151,172.16.98.152',
'es.port'='9200',
'es.mapping.names'= 'addTime:addTime',
'es.date.format'='yyyy-MM-dd HH:mm:ss',
'es.index.auto.create'='false',
)

查询报错：

更改hive表数据类型为string

CREATE EXTERNAL TABLE hive_es.cty_test5(
addTime string
)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'cty_test/cty_test',
'es.nodes'='172.16.98.113,172.16.98.149,172.16.98.150,172.16.98.151,172.16.98.152',
'es.port'='9200',
'es.mapping.names'= 'addTime:addTime',
'es.date.format'='yyyy-MM-dd HH:mm:ss',
'es.index.auto.create'='false',
)

查询继续报错：

查阅资料：

elasticsearch-hadoop中用于将ES中的日期转换为Hive中的日期格式的类为org.elasticsearch.hadoop.hive.HiveValueReader，通过查看该类的源码，其实现的用户日期转换的方法为：

@Override
protected Object parseDate(String value, boolean richDate) {
	return (richDate ? new TimestampWritable(new Timestamp(DatatypeConverter.parseDateTime(value).getTimeInMillis())) : parseString(value));
}

可以看到它是通过javax.xml.bind.DatatypeConverter.parseDateTime(String)方法将对应的日期字符串转换为日期的，该方法不支持的日期字符串格式为“yyyy-MM-dd HH:mm:ss”的字符串，它支持的日期字符串的格式为“yyyy-MM-ddTHH:mm:ss”这样的。

解决方案：

在建表时设置参数’es.mapping.date.rich’=‘false’，然后hive字段类型设为string。

官方解释：

Whether to create a rich Date like object for Date fields in Elasticsearch or returned them as primitives (String or long). By default this is true. The actual object type is based on the library used; noteable exception being Map/Reduce which provides no built-in Date object and as such LongWritable and Text are returned regardless of this setting.

es操作

Sun, 09 Jul 2023 00:00:00 +0000

ES建表

PUT http://121.46.197.112:9200/cty_test
Content-Type: application/json

{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1,
    "analysis": {
      "analyzer": {
        "greek_lowercase_analyzer": {
          "filter": [
            "lowercase"
          ],
          "type": "custom",
          "tokenizer": "standard"
        }
      }
    }
  },
  "mappings": {
    "cty_test": {
      "properties": {
        "dupId": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "id": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "supplierId": {
          "type": "long"
        },
        "addTime": {
          "format": "yyyy-MM-dd HH:mm:ss",
          "type": "date"
        },
        "scCountry": {
          "type": "keyword"
        },
        "shipName": {
          "type": "text"
        },
        "manifestQuanityDouble": {
          "type": "float"
        },
        "importerName": {
          "analyzer": "greek_lowercase_analyzer",
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "tradeType": {
          "type": "integer"
        },
        "arriveDate": {
          "format": "yyyy-MM-dd",
          "type": "date"
        },
        "modeOfTransportation": {
          "type": "keyword"
        },
        "modeoftransPortation": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "weight": {
          "type": "double"
        }
      }
    }
  }
}

ES写入数据

POST http://121.46.197.112:9200/cty_test/cty_test
Content-Type: application/json

{
  "dupId": "34324",
  "id" : "545435",
  "supplierId" : 43636,
  "addTime" : "2021-02-05 08:08:08",
  "scCountry" : "rrr",
  "shipName" : "uuu",
  "manifestQuanityDouble" : 55.00,
  "importerName" : "ccd",
  "tradeType" : 43,
  "arriveDate" : "2018-09-03",
  "weight" : 33.0
}

查询某列不为空

hive映射es

Sun, 16 Oct 2022 00:00:00 +0000

一 hive导入es

1 创建hive-es映射表

CREATE EXTERNAL TABLE hive_es.re_run_test2(
id STRING
,test STRING)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 're_run_test2/test', 
'es.nodes'='172.16.98.113,172.16.98.149,172.16.98.150,172.16.98.151,172.16.98.152',
'es.port'='9200',
'es.mapping.id' ='id')

注：

1.

es.resource对应es中的index/type

2.

1.es.mapping.names为hive和es字段名映射关系。
2.如果hive表和es表字段名完全一致，可以省略此参数。
3.hive中字段名不区分大小写，元数据寸的全是小写；es中字段大小写敏感，如果es中字段名出现大写，需认真填写。
4.es中_id为自动生成，如若需要覆盖，需加参数'es.mapping.id'='id'

2 先导入es映射表相关jar包

add jar /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hive/auxlib/elasticsearch-hadoop-6.3.0.jar;
add jar /data/jar/httpclient-4.5.5.jar;
add jar /data/jar/org.apache.commons.httpclient.jar;

3 向映射表insert数据

二 es导入hive

1 建hive映射表

CREATE EXTERNAL TABLE hive_es.cty_test5(
addTime string
)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'cty_test/cty_test', 
'es.nodes'='172.16.98.113,172.16.98.149,172.16.98.150,172.16.98.151,172.16.98.152',
'es.port'='9200',
'es.mapping.names'= 'addTime:addTime',
'es.mapping.date.rich'='false',
  'es.index.auto.create'='false',
)

注意，hive表数据类型要和es一致，除了es的date要转成hive的string，同时要加参数’es.mapping.date.rich’=‘false’,否则查询会报错.

2 通过映射表向其他表insert

参考：

https://www.cnblogs.com/koushr/p/9505435.html