当前位置:首页 > 服务端 > Flink出现network.partition.ProducerFailedException: java.lang.NullPointerException

Flink出现network.partition.ProducerFailedException: java.lang.NullPointerException

一、错误日志

org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException: Error at remote task manager 'xx.xxx.xxx.xxx/xxx.xxx.xxx.xxx:34750'.
at org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.decodeMsg(CreditBasedPartitionRequestClientHandler.java:325) ~[flinkjob:?]
at org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.channelRead(CreditBasedPartitionRequestClientHandler.java:214) ~[flinkjob:?]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[flinkjob:?]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[flinkjob:?]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[flinkjob:?]
at org.apache.flink.runtime.io.network.netty.NettyMessageClientDecoderDelegate.channelRead(NettyMessageClientDecoderDelegate.java:112) ~[flinkjob:?]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[flinkjob:?]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[flinkjob:?]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[flinkjob:?]
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[flinkjob:?]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[flinkjob:?]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[flinkjob:?]
at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[flinkjob:?]
at org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) ~[flinkjob:?]
at org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) ~[flinkjob:?]
at org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[flinkjob:?]
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[flinkjob:?]
at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[flinkjob:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
Caused by: org.apache.flink.runtime.io.network.partition.ProducerFailedException: java.lang.NullPointerException
at org.apache.flink.runtime.io.network.netty.PartitionRequestQueue.writeAndFlushNextMessageIfPossible(PartitionRequestQueue.java:224) ~[flinkjob:?]
at org.apache.flink.runtime.io.network.netty.PartitionRequestQueue.enqueueAvailableReader(PartitionRequestQueue.java:110) ~[flinkjob:?]
at org.apache.flink.runtime.io.network.netty.PartitionRequestQueue.userEventTriggered(PartitionRequestQueue.java:173) ~[flinkjob:?]
at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) ~

二、原因分析

  由于启动参数中只设置了一个插槽,而多个Task任务都来共享这个插槽,导致网络出现繁忙、一些Task容器无法使用到插槽,从而出现network.partition.ProducerFailedException

三、解决方案

  发生报错之前,我的启动参数如下:

bin/flink run -d -m yarn-cluster -yjm 2048m -ytm 4096m -p 1 -ys 1 -yD taskmanager.memory.managed.size=265M -c 全类名 flinkjob.jar

  解决方案是:合理设置并行度以及插槽数,在启动参数中指定即可

bin/flink run -d -m yarn-cluster -yjm 2048m -ytm 4096m -p 5 -ys 3 -yD taskmanager.memory.managed.size=265M -c 全类名 flinkjob.jar

参数说明:

  -m:指定yarn集群

  -yjm:指定jobmanager的内存

  -ytm:指定taskmanager的内存

  -p:指定并行度

  -ys:指定插槽的数量

  -yD:后台运行

  taskmanager.memory.managed.size:taskmanager直接管理内存

Flink出现network.partition.ProducerFailedException: java.lang.NullPointerException _ JavaClub全栈架构师技术笔记

作者:宜家数据小哥
来源链接:https://www.cnblogs.com/qq1035807396/p/16626856.html

版权声明:
1、Java侠(https://www.javaxia.com)以学习交流为目的,由作者投稿、网友推荐和小编整理收藏优秀的IT技术及相关内容,包括但不限于文字、图片、音频、视频、软件、程序等,其均来自互联网,本站不享有版权,版权归原作者所有。

2、本站提供的内容仅用于个人学习、研究或欣赏,以及其他非商业性或非盈利性用途,但同时应遵守著作权法及其他相关法律的规定,不得侵犯相关权利人及本网站的合法权利。
3、本网站内容原作者如不愿意在本网站刊登内容,请及时通知本站(javaclubcn@163.com),我们将第一时间核实后及时予以删除。





本文链接:https://www.javaxia.com/server/124807.html

分享给朋友: