ACTIVEMQ 遇到无法解决的问题

李丹 发布于 2012/06/09 23:24
阅读 5K+
收藏 1

【华为云1024程序员节·向云而生】预约直播 抽14件华为电子产品礼包!>>>

最近使用activeMq作为消息总线服务,但是不管如何修改配置也好,总是过不了多久就失去响应,导致前端服务堵死。经检查都是应为JVM在GC的时候僵死导致的,S1区,E区,O区都已经100%占用,但是查看服务端的logs有没有outofmemory异常,找不到这样异常的GC原因,也没有解决办法。amq版本是5.4.3,更新到5.6.0后问题依旧,jdk版本是 "1.6.0_27",消息速率严格控制在60-70/秒

 JVM参数:-server -Xmx3g -Xms3g -Xmn1024m -XX:PermSize=64m -Xss128k -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -Dorg.apache.activemq.UseDedicatedTaskRunner=false

启动命令:

/opt/apache-activemq-5.4.3/bin/activemq start xbean:file:../conf/activemq.test.xml

仅使用了一个topic,然后客户端的话采用的是非事务性的发送机制,非持久化的订阅模式,failover模式:

(ip这里隐去)failover:(tcp://xxx.xxx.xxx.xxx:61618?wireFormat.maxInactivityDuration=10000)&timeout=400&maxReconnectDelay=10000

每次运行一段时间以后JVM HEAP就满了,不停的FGC导致服务端失去响应,使用jmap -histo:live XXX检查内存使用,发现大量的对象如下:
 1:       5203857      666166616  [C
   2:       5110200      245289600  org.apache.activemq.command.ProducerId
   3:       5109099      245236752  org.apache.activemq.command.ProducerInfo
   4:       5109306      204372240  org.apache.activemq.command.SessionId
   5:       5204236      166535552  java.lang.String
   6:       5109522      163504704  java.util.concurrent.ConcurrentHashMap$HashEntry
   7:       5109099      122618376  org.apache.activemq.state.ProducerState
   8:       5109225       81747600  org.apache.activemq.command.ConnectionId
   9:          5824       42077672  [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
  10:         46320        7025536  <constMethodKlass>
  11:         46320        6308096  <methodKlass>
  12:          5313        5534184  [B
  13:          4116        4733048  <constantPoolKlass>
  14:         73509        4319912  <symbolKlass>
  15:          4116        3257576  <instanceKlassKlass>
  16:          3525        2924488  <constantPoolCacheKlass>
  17:         68094        2723760  org.apache.activemq.broker.ProducerBrokerExchange
已经查阅了很多资料,也尝试过很多修改,没有办法解决,特求高人帮忙

如下是mq的启动配置文件

	<!--
		Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file
		distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You
		under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the
		License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by
		applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS,
		WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language
		governing permissions and limitations under the License.
	-->
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:amq="http://activemq.apache.org/schema/core"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
  http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">

	<!-- Allows us to use system properties as variables in this configuration file -->
	<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
		<property name="locations">
			<value>file:${activemq.base}/conf/credentials.properties</value>
		</property>
	</bean>

	<!-- 
        The <broker> element is used to configure the ActiveMQ broker. 
    -->
	<broker xmlns="http://activemq.apache.org/schema/core" brokerName="atmq" dataDirectory="${activemq.base}/data"
		destroyApplicationContextOnStop="true" persistent="false" advisorySupport="false">
		<!--
			For better performances use VM cursor and small memory limit. For more information, see:

			http://activemq.apache.org/message-cursors.html Also, if your producer is "hanging", it's probably due to producer
			flow control. For more information, see: http://activemq.apache.org/producer-flow-control.html
		-->
		<destinationPolicy>
			<policyMap>
				<policyEntries>
					<policyEntry topic=">" producerFlowControl="true" memoryLimit="512mb">
						<pendingSubscriberPolicy>
							<vmCursor />
						</pendingSubscriberPolicy>
					</policyEntry>
					<policyEntry queue=">" producerFlowControl="true" memoryLimit="512mb">
						<!--
							Use VM cursor for better latency For more information, see: http://activemq.apache.org/message-cursors.html

							<pendingQueuePolicy> <vmQueueCursor/> </pendingQueuePolicy>
						-->
					</policyEntry>
				</policyEntries>
			</policyMap>
		</destinationPolicy>


		<!--
			The managementContext is used to configure how ActiveMQ is exposed in JMX. By default, ActiveMQ uses the MBean server
			that is started by the JVM. For more information, see: http://activemq.apache.org/jmx.html
		-->
		<managementContext>
			<managementContext createConnector="false" />
		</managementContext>

		<!--
			Configure message persistence for the broker. The default persistence mechanism is the KahaDB store (identified by
			the kahaDB tag). For more information, see: http://activemq.apache.org/persistence.html
		-->
		<persistenceAdapter>
			<kahaDB directory="${activemq.base}/data/kahadb" />
		</persistenceAdapter>


		<!--
			The systemUsage controls the maximum amount of space the broker will use before slowing down producers. For more
			information, see: http://activemq.apache.org/producer-flow-control.html
		-->
		<systemUsage>
			<systemUsage>
				<memoryUsage>
					<memoryUsage limit="256 mb" />
				</memoryUsage>
				<storeUsage>
					<storeUsage limit="1 gb" name="foo" />
				</storeUsage>
				<tempUsage>
					<tempUsage limit="128 mb" />
				</tempUsage>
			</systemUsage>
		</systemUsage>

		<!--
			The transport connectors expose ActiveMQ over a given protocol to clients and other brokers. For more information,
			see: http://activemq.apache.org/configuring-transports.html
		-->
		<transportConnectors>
			<transportConnector name="openwire" uri="tcp://0.0.0.0:61618" />
		</transportConnectors>

	</broker>

	<!--
		Enable web consoles, REST and Ajax APIs and demos Take a look at activemq-jetty.xml for more details
	-->
	<import resource="jetty.xml" />

</beans>

以下是问题补充:

@李丹:发现一个问题,如果发发送端重启,这时候就能GC掉 (2012/06/11 10:39)
加载中
1
李丹
李丹

引用来自“beata_xing”的答案

请问解决了么,我也遇到同样的问题
已经解决了,查看了activemq的源代码,在session的close方法中,客户端会给服务端发送一条消息,服务端收到该消息后会销毁服务器端内存中与该session关联的producer信息。之前我自己编写的代码中只代理了close、equal这几个方法,但是没有代理producer方法,导致同一个session每次调用都创建了新的producer实例,却没有给服务端发送销毁信息,导致服务端JVM内存溢出。后来代理了createproducer方法后,每次都返回之前创建好的、已经缓存起来的对象,就没有这个问题了
0
mallon
mallon
禁掉消息持久化看看
李丹
李丹
谢谢楼上兄弟的意见,不过我的代码里面已经贴出来了,broker的persistent=false
0
光石头
光石头
去掉jvm的参数,使用默认jvm配置。
李丹
李丹
回复 @屁屁果 : 我是64位的JDK,服务器是8G的内存。后台没有错误日志。看现象就是JVM满了,不停的GC导致应用完全暂停了
光石头
光石头
默认报什么错误?看你jvm配置,如果不是64位jdk,内存分配太高,gc回收可能会出问题
李丹
李丹
谢谢,不过我是使用默认的JVM参数有问题后才换成现在这个参数的
0
DISSECTOR
DISSECTOR

memoryUsage从256兆限制改为1024兆试

去掉producer flowcontrol试试

 

 

 

 

 

李丹
李丹
尝试过了,今天观察了下,发现还是没用
0
DISSECTOR
DISSECTOR

看Heap情况,可能是发送端代码不规范,导致AMQ的producer(连接/线程/对象等)无法释放

(发送)客户端并发连接有多少?是否合理使用了connection pool?

0
SK-J
SK-J

试试

<systemUsage>

075             <systemUsage>
076                 <memoryUsage>
077                     <memoryUsage limit="1024 mb" />
078                 </memoryUsage>
079                 <storeUsage>
080                     <storeUsage limit="1 gb" name="foo" />
081                 </storeUsage>
082                 <tempUsage>
083                     <tempUsage limit="128 mb" />
084                 </tempUsage>
085             </systemUsage>
086         </systemUsage>
0
b
beata_xing
请问解决了么,我也遇到同样的问题
0
b
beata_xing
能把你这段代码贴出来看看吗
返回顶部
顶部