spark stream JavaKafkaWordCount.java 拉取不到数据

漓江 发布于 2016/10/14 14:43
阅读 563
收藏 0

问题:

   使用spark streaming拉取kafka传递过来的数据,并进行单词统计,但没有获取到任何信息。

现象:

logLevel=ERROR的场合

-------------------------------------------
Time: 1476455248000 ms
-------------------------------------------

-------------------------------------------
Time: 1476455249000 ms
-------------------------------------------

或者就是一堆INFO&WARN信息,如下所示:

logLevel=INFO的场合

Spark assembly has been built with Hive, including Datanucleus jars on classpath
16/10/14 15:42:59 INFO SecurityManager: Changing view acls to: root
16/10/14 15:42:59 INFO SecurityManager: Changing modify acls to: root
16/10/14 15:42:59 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
16/10/14 15:43:00 INFO Slf4jLogger: Slf4jLogger started
16/10/14 15:43:00 INFO Remoting: Starting remoting
16/10/14 15:43:00 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@sv000:50455]
16/10/14 15:43:00 INFO Utils: Successfully started service 'sparkDriver' on port 50455.
16/10/14 15:43:00 INFO SparkEnv: Registering MapOutputTracker
16/10/14 15:43:00 INFO SparkEnv: Registering BlockManagerMaster
16/10/14 15:43:00 INFO DiskBlockManager: Created local directory at /tmp/spark-6b6a1d61-fcbb-4dc8-814b-e669dacb1cf0/spark-5f5aab7b-2fd0-4737-a8da-66c25d6e1dfa
16/10/14 15:43:00 INFO MemoryStore: MemoryStore started with capacity 267.3 MB
16/10/14 15:43:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/10/14 15:43:01 INFO HttpFileServer: HTTP File server directory is /tmp/spark-340aa0b4-c0fe-45aa-b058-610c511a7799/spark-27503e04-ed7c-4421-986e-529b9ab92e9b
16/10/14 15:43:01 INFO HttpServer: Starting HTTP Server
16/10/14 15:43:01 INFO Utils: Successfully started service 'HTTP file server' on port 58905.
16/10/14 15:43:01 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/10/14 15:43:01 INFO SparkUI: Started SparkUI at http://sv000:4040
16/10/14 15:43:02 INFO SparkContext: Added JAR file:/home/myProject/spark-1.2.1-bin-hadoop2.4/lib/spark-examples-1.2.1-hadoop2.4.0.jar at http://172.28.156.200:58905/jars/spark-examples-1.2.1-hadoop2.4.0.jar with timestamp 1476427382046
16/10/14 15:43:02 INFO Executor: Starting executor ID <driver> on host localhost
16/10/14 15:43:02 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@sv000:50455/user/HeartbeatReceiver
16/10/14 15:43:02 INFO NettyBlockTransferService: Server created on 41541
16/10/14 15:43:02 INFO BlockManagerMaster: Trying to register BlockManager
16/10/14 15:43:02 INFO BlockManagerMasterActor: Registering block manager localhost:41541 with 267.3 MB RAM, BlockManagerId(<driver>, localhost, 41541)
16/10/14 15:43:02 INFO BlockManagerMaster: Registered BlockManager
16/10/14 15:43:02 INFO ReceiverTracker: ReceiverTracker started
16/10/14 15:43:02 INFO ForEachDStream: metadataCleanupDelay = -1
16/10/14 15:43:02 INFO ShuffledDStream: metadataCleanupDelay = -1
16/10/14 15:43:02 INFO MappedDStream: metadataCleanupDelay = -1
16/10/14 15:43:02 INFO FlatMappedDStream: metadataCleanupDelay = -1
16/10/14 15:43:02 INFO MappedDStream: metadataCleanupDelay = -1
16/10/14 15:43:02 INFO KafkaInputDStream: metadataCleanupDelay = -1
16/10/14 15:43:02 INFO KafkaInputDStream: Slide time = 2000 ms
16/10/14 15:43:02 INFO KafkaInputDStream: Storage level = StorageLevel(false, false, false, false, 1)
16/10/14 15:43:02 INFO KafkaInputDStream: Checkpoint interval = null
16/10/14 15:43:02 INFO KafkaInputDStream: Remember duration = 2000 ms
16/10/14 15:43:02 INFO KafkaInputDStream: Initialized and validated org.apache.spark.streaming.kafka.KafkaInputDStream@6bb7cce7
16/10/14 15:43:02 INFO MappedDStream: Slide time = 2000 ms
16/10/14 15:43:02 INFO MappedDStream: Storage level = StorageLevel(false, false, false, false, 1)
16/10/14 15:43:02 INFO MappedDStream: Checkpoint interval = null
16/10/14 15:43:02 INFO MappedDStream: Remember duration = 2000 ms
16/10/14 15:43:02 INFO MappedDStream: Initialized and validated org.apache.spark.streaming.dstream.MappedDStream@41c62850
16/10/14 15:43:02 INFO FlatMappedDStream: Slide time = 2000 ms
16/10/14 15:43:02 INFO FlatMappedDStream: Storage level = StorageLevel(false, false, false, false, 1)
16/10/14 15:43:02 INFO FlatMappedDStream: Checkpoint interval = null
16/10/14 15:43:02 INFO FlatMappedDStream: Remember duration = 2000 ms
16/10/14 15:43:02 INFO FlatMappedDStream: Initialized and validated org.apache.spark.streaming.dstream.FlatMappedDStream@6b530eb9
16/10/14 15:43:02 INFO MappedDStream: Slide time = 2000 ms
16/10/14 15:43:02 INFO MappedDStream: Storage level = StorageLevel(false, false, false, false, 1)
16/10/14 15:43:02 INFO MappedDStream: Checkpoint interval = null
16/10/14 15:43:02 INFO MappedDStream: Remember duration = 2000 ms

。。。。


环境信息:

spark:1.2.1

hadoop:2.5.2

Kafka:2.8.0-0.8.1.1

JDK:1.7.06

redhat:6.4


加载中
返回顶部
顶部