HSql执行报错:Hive internal error inside isAssignableFromSettablePrimitiveOI void not supported yet

osc_89275544 发布于 01/27 16:14
阅读 66
收藏 0

软件工程师和算法工程师速来!正在直播:如何使用英特尔oneAPI工具实现PyTorch 优化>>>

这是一段简单的sql,需求是根据通过关联表获取union_account_id并按此字段分组进行计算,只需保留在关联表中能查到union_account_id的记录。

代码如下:

set hive.exec.mode.local.auto=false;
set hive.auto.convert.join=false;
set hive.optimize.skewjoin = true;
set hive.groupby.skewindata = true;
set hive.skewjoin.key = 100000;
set mapreduce.map.memory.mb=4096;
set mapreduce.reduce.memory.mb=4096;
set hive.exec.compress.output=true;
set mapred.output.compress=true;
set mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;
set io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec;
drop table if exists ycf_dwh_mdm.m_act_user_portrait_mid2;
create table if not exists ycf_dwh_mdm.m_act_user_portrait_mid2
as
select
tt2.union_account_id
,concat_ws('|',collect_set(tt1.list_product_and_package)) as list_product_and_package
,concat_ws('|',collect_set(nvl(tt1.pay_type_names,''))) as pay_type_names
,concat_ws('^',collect_set(tt1.order_product_ids)) as order_product_ids
,concat_ws('^',collect_set(tt1.order_product_names)) as order_product_names
,concat_ws('^',collect_set(tt1.order_package_ids)) as order_package_ids
,concat_ws('^',collect_set(tt1.order_package_names)) as order_package_names
,concat_ws(',',collect_set(tt1.label_name)) as label_name
from ycf_dwh_mdm.m_act_user_portrait_mid1 tt1
left join ycf_dwh_fdm.f_usr_mag_user_union_account_mapping_full tt2
on tt1.user_id=tt2.user_id and tt2.row_id=1
where tt2.union_account_id is not null
and (
tt1.list_product_and_package<>''
or tt1.pay_type_names<>''
or tt1.order_product_ids<>''
or tt1.order_product_names<>''
or tt1.order_package_ids<>''
or tt1.order_package_names<>''
or tt1.label_name<>''
)
group by tt2.union_account_id
;

数据量:TT1:1600w ;TT2:80w

Hive版本:hive-common-1.1.0

错误日志:

Query ID = hdfs_20220127155252_844fbc68-7d2d-4b31-9a8f-86a90eb15d0d
Total jobs = 5
Launching Job 1 out of 5
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Job = job_1643268609681_0075, Tracking URL = http://xxx-big1.xxx.com:8088/proxy/application_1643268609681_0075/
Kill Command = /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/hadoop/bin/hadoop job  -kill job_1643268609681_0075
Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 1
2022-01-27 15:52:28,397 Stage-1 map = 0%,  reduce = 0%
2022-01-27 15:52:46,978 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_1643268609681_0075 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1643268609681_0075_m_000001 (and more) from job job_1643268609681_0075

Task with the most failures(4):
-----
Task ID:
  task_1643268609681_0075_m_000000

URL:
  http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1643268609681_0075&tipid=task_1643268609681_0075_m_000000
-----
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:455)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
    ... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
    ... 14 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
    ... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:147)
    ... 22 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Hive internal error inside isAssignableFromSettablePrimitiveOI void not supported yet.
    at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:323)
    at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:333)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:116)
    ... 22 more
Caused by: java.lang.RuntimeException: Hive internal error inside isAssignableFromSettablePrimitiveOI void not supported yet.
    at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettablePrimitiveOI(ObjectInspectorUtils.java:1191)
    at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettableOI(ObjectInspectorUtils.java:1200)
    at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1238)
    at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConvertedOI(ObjectInspectorConverters.java:224)
    at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConvertedOI(ObjectInspectorConverters.java:252)
    at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConvertedOI(ObjectInspectorConverters.java:188)
    at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:305)
    ... 24 more


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 2  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

 

加载中
OSCHINA
登录后可查看更多优质内容
返回顶部
顶部
返回顶部
顶部