scala远程调spark的端口和命令,应该怎么操作?

ios应用猫 发布于 2017/01/04 15:11
阅读 1K+
收藏 0


我试了试这个,但是发现过去不,

spark-submit --supervise --class spark_security.Sockpuppet  --name "testkdl1"  --executor-memory 8192M --num-executors 8 --master yarn-client --driver-java-options "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8888" spark_Security-1.0-SNAPSHOT.jar  logindelaer.properties



代码行报错有:

17/01/04 15:10:57 INFO SparkContext: Running Spark version 1.6.2
17/01/04 15:10:58 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/01/04 15:10:58 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:363)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79)
at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:104)
at org.apache.hadoop.security.Groups.<init>(Groups.java:86)
at org.apache.hadoop.security.Groups.<init>(Groups.java:66)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:280)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:271)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:248)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:763)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:748)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:621)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2198)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2198)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2198)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:322)
at com.pflm.kafka.WordCount$.main(WordCount.scala:16)
at com.pflm.kafka.WordCount.main(WordCount.scala)
17/01/04 15:10:58 INFO SecurityManager: Changing view acls to: Administrator
17/01/04 15:10:58 INFO SecurityManager: Changing modify acls to: Administrator
17/01/04 15:10:58 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(Administrator); users with modify permissions: Set(Administrator)
17/01/04 15:10:58 INFO Utils: Successfully started service 'sparkDriver' on port 63437.
17/01/04 15:10:59 INFO Slf4jLogger: Slf4jLogger started
17/01/04 15:10:59 INFO Remoting: Starting remoting
17/01/04 15:10:59 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.1.32:63451]
17/01/04 15:10:59 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 63451.
17/01/04 15:10:59 INFO SparkEnv: Registering MapOutputTracker
17/01/04 15:10:59 INFO SparkEnv: Registering BlockManagerMaster
17/01/04 15:10:59 INFO DiskBlockManager: Created local directory at C:\Users\Administrator\AppData\Local\Temp\blockmgr-14b10c8e-3126-48da-bcca-b880818144ad
17/01/04 15:10:59 INFO MemoryStore: MemoryStore started with capacity 1122.0 MB
17/01/04 15:10:59 INFO SparkEnv: Registering OutputCommitCoordinator
17/01/04 15:10:59 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/01/04 15:10:59 INFO SparkUI: Started SparkUI at http://192.168.1.32:4040
17/01/04 15:10:59 INFO AppClient$ClientEndpoint: Connecting to master spark://192.168.10.121:7874...
17/01/04 15:10:59 WARN TransportChannelHandler: Exception in connection from /192.168.10.121:7874
java.io.IOException: 远程主机强迫关闭了一个现有的连接。
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(Unknown Source)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
at sun.nio.ch.IOUtil.read(Unknown Source)
at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:313)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:242)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Unknown Source)
17/01/04 15:10:59 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from CentOS121/192.168.10.121:7874 is closed
17/01/04 15:10:59 WARN AppClient$ClientEndpoint: Failed to connect to master 192.168.10.121:7874
java.io.IOException: 远程主机强迫关闭了一个现有的连接。
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(Unknown Source)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
at sun.nio.ch.IOUtil.read(Unknown Source)
at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:313)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:242)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Unknown Source)




加载中
返回顶部
顶部