系统跑了几天后 所有http rest接口请求超时

Gannalyo 发布于 08/26 15:48
阅读 787
收藏 1

各位好  帮忙分析个问题呗

某中间件产品分client与server
其中,client与server通过grpc交互
另外,server提供了http rest接口

问题:系统跑了几天后收到http rest接口请求超时的告警信息,经验证所有server的http rest接口均请求超时,但client与server还是可以正常交互的。
tip:期间,服务器未做网络相关任何操作,服务器中其它服务可以正常请求。tracing无相关记录(可能就没有请求到系统中)。

请问  该从何分析起呀?完全懵逼+无头绪

加载中
0
Gannalyo
Gannalyo

收到告警信息前,http rest接口还是可以正常访问的。

0
Kingviker
Kingviker

你问个问题 啥都不说。别人想帮你都没办法。

目前你提供的信息,唯一有效的信息就是:http rest接口均请求超时

既然超时,你看一下是卡到哪个环节超时的, 就继续在这个环节深入排查。

Gannalyo
Gannalyo
额。。。问个问题咋这么难呐? 那。。。大佬 您这面还想要了解哪些信息呀? 我再尽量说点吧: 我有一个test接口,接口内直接return "ok",但是却依然超时。。。。 我不晓得该如何确定卡到哪个环节 您能否再指点下
0
Gannalyo
Gannalyo

tracing都没有记录,所以更能确定是没尽到接口中。

0
@ccny
@ccny

1.如果是通过linux系统布署的, 通过curl命令可以测试一下。配置能了没有。

2.查看网络有没有问题。ping telnet 

Gannalyo
Gannalyo
回复 @@ccny : 目前环境已不存在了,昨晚不知咋搞的,把服务对外的http端口搞没了 我ss -anlt一下 结果我那服务中其它端口是正常在的 但对外的http端口却没了 好刺激
@ccny
@ccny
回复 @Gannalyo : netstat 查看 本机占用端口 情况
Gannalyo
Gannalyo
是linux。 curl ping telnet 都试过 但三者对于我那个服务提供的http端口均不好使 其它服务是可以的
0
不会飞的小龙人

1.查看网络,telent目标机器IP和端口,看是否正常

2.上server服务器上,netstat监听服务端口,是否有连接,连接的状态。另服务本机curl请求rest

3.查看客户端与服务器的安全策略是否有改动,如IP、端口等策略

4.抓客户端出口包和服务端入口包看看

 

Gannalyo
Gannalyo
1.telnet我那个服务 http的ip和port不同 grpc的ip和port是通的 2.netstat服务端口如我服的图片大部分为CLOSE_WAIT状态。本机上curl 127.0.0.1/localhost/ip 加http的端口 也是不行 3.没有改动 4.抓包信息如下。
0
Gannalyo
Gannalyo

贴下端口信息及堆栈信息。

2019-08-28 10:07:10
Full thread dump OpenJDK 64-Bit Server VM (25.222-b10 mixed mode):

"grpc-default-executor-454" #581 daemon prio=5 os_prio=0 tid=0x00007fc1b803c800 nid=0x5c1 waiting on condition [0x00007fc184af6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000093c0a120> (a java.util.concurrent.SynchronousQueue$TransferStack)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
	at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"Attach Listener" #580 daemon prio=9 os_prio=0 tid=0x00007fc250001000 nid=0x5c0 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"grpc-default-executor-453" #579 daemon prio=5 os_prio=0 tid=0x00007fc1b8011800 nid=0x558 waiting on condition [0x00007fc1847f3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000093c0a120> (a java.util.concurrent.SynchronousQueue$TransferStack)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
	at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"grpc-default-executor-449" #574 daemon prio=5 os_prio=0 tid=0x00007fc1b803e000 nid=0x544 waiting on condition [0x00007fc1c55f4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000093c0a120> (a java.util.concurrent.SynchronousQueue$TransferStack)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
	at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"grpc-default-executor-448" #573 daemon prio=5 os_prio=0 tid=0x00007fc1b8025000 nid=0x543 waiting on condition [0x00007fc1c4aeb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000093c0a120> (a java.util.concurrent.SynchronousQueue$TransferStack)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
	at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"grpc-default-worker-ELG-3-6" #105 daemon prio=5 os_prio=0 tid=0x00007fc1d4010800 nid=0x176 runnable [0x00007fc1c5cfb000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000938348e0> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x0000000093834958> (a java.util.Collections$UnmodifiableSet)
	- locked <0x0000000093558030> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

"grpc-default-worker-ELG-3-5" #104 daemon prio=5 os_prio=0 tid=0x00007fc1d400f800 nid=0x172 runnable [0x00007fc2063ce000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x0000000093834760> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000938347d8> (a java.util.Collections$UnmodifiableSet)
	- locked <0x0000000093557fa0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

"grpc-default-worker-ELG-3-4" #103 daemon prio=5 os_prio=0 tid=0x00007fc1d400d800 nid=0x171 runnable [0x00007fc206dfe000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x0000000093834b50> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x0000000093834be0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x0000000093558198> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

"pool-4-thread-5" #75 prio=5 os_prio=0 tid=0x00007fc1e0005800 nid=0x7f39 waiting on condition [0x00007fc1c4fee000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000009252d120> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"pool-4-thread-4" #74 prio=5 os_prio=0 tid=0x00007fc1e0004800 nid=0x7f38 waiting on condition [0x00007fc270217000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000009252d120> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"pool-4-thread-3" #73 prio=5 os_prio=0 tid=0x00007fc1e0003800 nid=0x7f37 waiting on condition [0x00007fc2041c4000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000009252d120> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"pool-4-thread-2" #72 prio=5 os_prio=0 tid=0x00007fc1e0002800 nid=0x7f36 waiting on condition [0x00007fc2068d1000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000009252d120> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"pool-4-thread-1" #71 prio=5 os_prio=0 tid=0x00007fc1e0001800 nid=0x7f32 waiting on condition [0x00007fc1c4eed000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000009252d120> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"DestroyJavaVM" #70 prio=5 os_prio=0 tid=0x00007fc2ca1b4800 nid=0x7ed3 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"grpc-default-worker-ELG-3-3" #66 daemon prio=5 os_prio=0 tid=0x00007fc1d400b800 nid=0x7f2d runnable [0x00007fc204ec9000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x0000000093962508> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000939624f0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x0000000093977778> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

"grpc-default-worker-ELG-3-2" #50 daemon prio=5 os_prio=0 tid=0x00007fc1d4008000 nid=0x7f1d runnable [0x00007fc1c6dfd000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000938da010> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000938da028> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000009376d508> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

"grpc-default-worker-ELG-3-1" #48 daemon prio=5 os_prio=0 tid=0x00007fc1d4006000 nid=0x7f1b runnable [0x00007fc2044c5000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x000000009377c2d0> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x000000009377c2e8> (a java.util.Collections$UnmodifiableSet)
	- locked <0x0000000093925878> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

"grpc-default-boss-ELG-1-1" #47 daemon prio=5 os_prio=0 tid=0x00007fc1dc298000 nid=0x7f1a runnable [0x00007fc2047c6000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x0000000093910a38> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x0000000093910a20> (a java.util.Collections$UnmodifiableSet)
	- locked <0x0000000093972680> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

"ObjectCleanerThread" #46 daemon prio=1 os_prio=0 tid=0x00007fc1dc297800 nid=0x7f19 in Object.wait() [0x00007fc2048c7000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	- locked <0x0000000093962550> (a java.lang.ref.ReferenceQueue$Lock)
	at io.netty.util.internal.ObjectCleaner$1.run(ObjectCleaner.java:54)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

"Thread-9" #44 prio=5 os_prio=0 tid=0x00007fc2c9e5c800 nid=0x7f17 in Object.wait() [0x00007fc204bc8000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x0000000093995070> (a java.lang.Object)
	at java.lang.Object.wait(Object.java:502)
	at io.grpc.internal.ServerImpl.awaitTermination(ServerImpl.java:279)
	- locked <0x0000000093995070> (a java.lang.Object)
	at org.apache.servicecomb.saga.alpha.server.GrpcStartable.start(GrpcStartable.java:80)
	at org.apache.servicecomb.saga.alpha.server.AlphaConfig$$Lambda$30/816288166.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:748)

"Thread-7" #40 prio=5 os_prio=0 tid=0x00007fc2c99bf000 nid=0x7f14 runnable [0x00007fc2056cb000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x0000000092ed9460> (a sun.nio.ch.Util$3)
	- locked <0x0000000092ed9478> (a java.util.Collections$UnmodifiableSet)
	- locked <0x0000000092f084e0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:352)
	at java.lang.Thread.run(Thread.java:748)

"server-timer" #39 daemon prio=5 os_prio=0 tid=0x00007fc2c99bc800 nid=0x7f11 in Object.wait() [0x00007fc2057cc000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.util.TimerThread.mainLoop(Timer.java:552)
	- locked <0x0000000092f67d48> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:505)

"Abandoned connection cleanup thread" #38 daemon prio=5 os_prio=0 tid=0x00007fc2c961e000 nid=0x7f0f in Object.wait() [0x00007fc2062cd000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	- locked <0x0000000091e00d80> (a java.lang.ref.ReferenceQueue$Lock)
	at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:64)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"Log4j2-TF-9-Scheduled-3" #29 daemon prio=5 os_prio=0 tid=0x00007fc2c91fe800 nid=0x7f07 waiting on condition [0x00007fc270518000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000090523a68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"RMI TCP Accept-0" #22 daemon prio=5 os_prio=0 tid=0x00007fc2c8bf0000 nid=0x7efc runnable [0x00007fc270b1a000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
	at java.net.ServerSocket.accept(ServerSocket.java:513)
	at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
	at java.lang.Thread.run(Thread.java:748)

"RMI TCP Accept-1099" #21 daemon prio=5 os_prio=0 tid=0x00007fc2c8bd6800 nid=0x7efb runnable [0x00007fc270c1b000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
	at java.net.ServerSocket.accept(ServerSocket.java:513)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
	at java.lang.Thread.run(Thread.java:748)

"RMI TCP Accept-0" #20 daemon prio=5 os_prio=0 tid=0x00007fc2c8bc3800 nid=0x7efa runnable [0x00007fc270d1c000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
	at java.net.ServerSocket.accept(ServerSocket.java:513)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
	at java.lang.Thread.run(Thread.java:748)

"Service Thread" #18 daemon prio=9 os_prio=0 tid=0x00007fc2c8300800 nid=0x7ef9 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread11" #17 daemon prio=9 os_prio=0 tid=0x00007fc2c82fd800 nid=0x7ef8 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread10" #16 daemon prio=9 os_prio=0 tid=0x00007fc2c82fb800 nid=0x7ef7 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread9" #15 daemon prio=9 os_prio=0 tid=0x00007fc2c82fa000 nid=0x7ef6 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread8" #14 daemon prio=9 os_prio=0 tid=0x00007fc2c82f7000 nid=0x7ef5 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread7" #13 daemon prio=9 os_prio=0 tid=0x00007fc2c82f4800 nid=0x7ef4 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread6" #12 daemon prio=9 os_prio=0 tid=0x00007fc2c82f2800 nid=0x7ef3 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread5" #11 daemon prio=9 os_prio=0 tid=0x00007fc2c82f0800 nid=0x7ef2 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread4" #10 daemon prio=9 os_prio=0 tid=0x00007fc2c82ee800 nid=0x7ef1 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread3" #9 daemon prio=9 os_prio=0 tid=0x00007fc2c82e4800 nid=0x7ef0 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007fc2c82e2000 nid=0x7eef waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007fc2c82e0000 nid=0x7eee waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007fc2c82de000 nid=0x7eed waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fc2c82dc000 nid=0x7eec runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007fc2c82da800 nid=0x7eeb waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fc2c82a5800 nid=0x7eea in Object.wait() [0x00007fc272b23000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	- locked <0x000000009002eed0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fc2c82a3000 nid=0x7ee9 in Object.wait() [0x00007fc272c24000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
	- locked <0x0000000090022b08> (a java.lang.ref.Reference$Lock)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"VM Thread" os_prio=0 tid=0x00007fc2c8299800 nid=0x7ee6 runnable 

"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fc2c8015800 nid=0x7ed4 runnable 

"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fc2c8023000 nid=0x7ed5 runnable 

"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fc2c8024800 nid=0x7ed6 runnable 

"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fc2c8026000 nid=0x7ed7 runnable 

"Gang worker#4 (Parallel GC Threads)" os_prio=0 tid=0x00007fc2c8028000 nid=0x7ed8 runnable 

"Gang worker#5 (Parallel GC Threads)" os_prio=0 tid=0x00007fc2c8029800 nid=0x7ed9 runnable 

"Gang worker#6 (Parallel GC Threads)" os_prio=0 tid=0x00007fc2c802b800 nid=0x7eda runnable 

"Gang worker#7 (Parallel GC Threads)" os_prio=0 tid=0x00007fc2c802d000 nid=0x7edb runnable 

"Gang worker#8 (Parallel GC Threads)" os_prio=0 tid=0x00007fc2c802f000 nid=0x7edc runnable 

"Gang worker#9 (Parallel GC Threads)" os_prio=0 tid=0x00007fc2c8030800 nid=0x7edd runnable 

"Gang worker#10 (Parallel GC Threads)" os_prio=0 tid=0x00007fc2c8032800 nid=0x7ede runnable 

"Gang worker#11 (Parallel GC Threads)" os_prio=0 tid=0x00007fc2c8034000 nid=0x7edf runnable 

"Gang worker#12 (Parallel GC Threads)" os_prio=0 tid=0x00007fc2c8036000 nid=0x7ee0 runnable 

"Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fc2c80e0000 nid=0x7ee5 runnable 

"Gang worker#0 (Parallel CMS Threads)" os_prio=0 tid=0x00007fc2c80d8800 nid=0x7ee1 runnable 

"Gang worker#1 (Parallel CMS Threads)" os_prio=0 tid=0x00007fc2c80da000 nid=0x7ee2 runnable 

"Gang worker#2 (Parallel CMS Threads)" os_prio=0 tid=0x00007fc2c80dc000 nid=0x7ee3 runnable 

"Gang worker#3 (Parallel CMS Threads)" os_prio=0 tid=0x00007fc2c80dd800 nid=0x7ee4 runnable 

"VM Periodic Task Thread" os_prio=0 tid=0x00007fc2c8bf2000 nid=0x7efd waiting on condition 

JNI global references: 316

 

不会飞的小龙人
8090有大量的close_wait,造成这种可能性之一是客户端通知服务器自已要关闭,但服务端的关闭通知没有发送出去,就会处于close_wait关闭等状态下,可以看一上tcp/ip的关闭握手机制。有可能是这个原因造成连接数一直被占用,客户端新的请求处于TIMED_WAITING 中。检查一下客户端与服务端的访问或连接方式;以上只是个人推测;
0
Gannalyo
Gannalyo
18:35:59.303278 IP 10.186.62.36.53514 > 10.186.62.30.8090: Flags [S], seq 1035020563, win 29200, options [mss 1460,sackOK,TS val 3378501853 ecr 0,nop,wscale 7], length 0
18:35:59.303336 IP 10.186.62.30.8090 > 10.186.62.36.53514: Flags [R.], seq 0, ack 1035020564, win 0, length 0
18:36:09.304284 IP 10.186.62.36.53516 > 10.186.62.30.8090: Flags [S], seq 3060660142, win 29200, options [mss 1460,sackOK,TS val 3378511853 ecr 0,nop,wscale 7], length 0
18:36:09.304343 IP 10.186.62.30.8090 > 10.186.62.36.53516: Flags [R.], seq 0, ack 3060660143, win 0, length 0
18:36:19.305543 IP 10.186.62.36.53520 > 10.186.62.30.8090: Flags [S], seq 2723016125, win 29200, options [mss 1460,sackOK,TS val 3378521854 ecr 0,nop,wscale 7], length 0
18:36:19.305607 IP 10.186.62.30.8090 > 10.186.62.36.53520: Flags [R.], seq 0, ack 2723016126, win 0, length 0
18:36:29.307287 IP 10.186.62.36.53522 > 10.186.62.30.8090: Flags [S], seq 30670864, win 29200, options [mss 1460,sackOK,TS val 3378531856 ecr 0,nop,wscale 7], length 0
18:36:29.307361 IP 10.186.62.30.8090 > 10.186.62.36.53522: Flags [R.], seq 0, ack 30670865, win 0, length 0
18:36:39.308461 IP 10.186.62.36.53524 > 10.186.62.30.8090: Flags [S], seq 705223745, win 29200, options [mss 1460,sackOK,TS val 3378541856 ecr 0,nop,wscale 7], length 0
18:36:39.308517 IP 10.186.62.30.8090 > 10.186.62.36.53524: Flags [R.], seq 0, ack 705223746, win 0, length 0
18:36:49.309508 IP 10.186.62.36.53526 > 10.186.62.30.8090: Flags [S], seq 1669864239, win 29200, options [mss 1460,sackOK,TS val 3378551857 ecr 0,nop,wscale 7], length 0
18:36:49.309576 IP 10.186.62.30.8090 > 10.186.62.36.53526: Flags [R.], seq 0, ack 1669864240, win 0, length 0
18:36:59.310724 IP 10.186.62.36.53528 > 10.186.62.30.8090: Flags [S], seq 2735342284, win 29200, options [mss 1460,sackOK,TS val 3378561858 ecr 0,nop,wscale 7], length 0
18:36:59.310799 IP 10.186.62.30.8090 > 10.186.62.36.53528: Flags [R.], seq 0, ack 2735342285, win 0, length 0
18:37:09.311936 IP 10.186.62.36.53530 > 10.186.62.30.8090: Flags [S], seq 822437706, win 29200, options [mss 1460,sackOK,TS val 3378571859 ecr 0,nop,wscale 7], length 0
18:37:09.311994 IP 10.186.62.30.8090 > 10.186.62.36.53530: Flags [R.], seq 0, ack 822437707, win 0, length 0
18:37:19.313079 IP 10.186.62.36.53532 > 10.186.62.30.8090: Flags [S], seq 2359102631, win 29200, options [mss 1460,sackOK,TS val 3378581859 ecr 0,nop,wscale 7], length 0
18:37:19.313398 IP 10.186.62.30.8090 > 10.186.62.36.53532: Flags [R.], seq 0, ack 2359102632, win 0, length 0
18:37:29.315525 IP 10.186.62.36.53534 > 10.186.62.30.8090: Flags [S], seq 1477605647, win 29200, options [mss 1460,sackOK,TS val 3378591861 ecr 0,nop,wscale 7], length 0
18:37:29.315594 IP 10.186.62.30.8090 > 10.186.62.36.53534: Flags [R.], seq 0, ack 1477605648, win 0, length 0
18:37:39.316901 IP 10.186.62.36.53536 > 10.186.62.30.8090: Flags [S], seq 510331887, win 29200, options [mss 1460,sackOK,TS val 3378601862 ecr 0,nop,wscale 7], length 0
18:37:39.316967 IP 10.186.62.30.8090 > 10.186.62.36.53536: Flags [R.], seq 0, ack 510331888, win 0, length 0
18:37:49.317976 IP 10.186.62.36.53538 > 10.186.62.30.8090: Flags [S], seq 1721913897, win 29200, options [mss 1460,sackOK,TS val 3378611863 ecr 0,nop,wscale 7], length 0
18:37:49.318039 IP 10.186.62.30.8090 > 10.186.62.36.53538: Flags [R.], seq 0, ack 1721913898, win 0, length 0
18:37:59.318976 IP 10.186.62.36.53540 > 10.186.62.30.8090: Flags [S], seq 1034833586, win 29200, options [mss 1460,sackOK,TS val 3378621864 ecr 0,nop,wscale 7], length 0
18:37:59.319039 IP 10.186.62.30.8090 > 10.186.62.36.53540: Flags [R.], seq 0, ack 1034833587, win 0, length 0
18:38:09.319964 IP 10.186.62.36.53542 > 10.186.62.30.8090: Flags [S], seq 558161605, win 29200, options [mss 1460,sackOK,TS val 3378631864 ecr 0,nop,wscale 7], length 0
18:38:09.320018 IP 10.186.62.30.8090 > 10.186.62.36.53542: Flags [R.], seq 0, ack 558161606, win 0, length 0
18:38:19.320947 IP 10.186.62.36.53544 > 10.186.62.30.8090: Flags [S], seq 1945459687, win 29200, options [mss 1460,sackOK,TS val 3378641865 ecr 0,nop,wscale 7], length 0
18:38:19.320999 IP 10.186.62.30.8090 > 10.186.62.36.53544: Flags [R.], seq 0, ack 1945459688, win 0, length 0
18:38:29.321913 IP 10.186.62.36.53546 > 10.186.62.30.8090: Flags [S], seq 2201073608, win 29200, options [mss 1460,sackOK,TS val 3378651865 ecr 0,nop,wscale 7], length 0
18:38:29.321976 IP 10.186.62.30.8090 > 10.186.62.36.53546: Flags [R.], seq 0, ack 2201073609, win 0, length 0
18:38:39.322963 IP 10.186.62.36.53548 > 10.186.62.30.8090: Flags [S], seq 3560373512, win 29200, options [mss 1460,sackOK,TS val 3378661866 ecr 0,nop,wscale 7], length 0
18:38:39.323022 IP 10.186.62.30.8090 > 10.186.62.36.53548: Flags [R.], seq 0, ack 3560373513, win 0, length 0
18:38:49.323985 IP 10.186.62.36.53550 > 10.186.62.30.8090: Flags [S], seq 2099246025, win 29200, options [mss 1460,sackOK,TS val 3378671867 ecr 0,nop,wscale 7], length 0
18:38:49.324066 IP 10.186.62.30.8090 > 10.186.62.36.53550: Flags [R.], seq 0, ack 2099246026, win 0, length 0
18:38:59.325334 IP 10.186.62.36.53552 > 10.186.62.30.8090: Flags [S], seq 2452469295, win 29200, options [mss 1460,sackOK,TS val 3378681868 ecr 0,nop,wscale 7], length 0
18:38:59.325400 IP 10.186.62.30.8090 > 10.186.62.36.53552: Flags [R.], seq 0, ack 2452469296, win 0, length 0
18:39:09.326435 IP 10.186.62.36.53554 > 10.186.62.30.8090: Flags [S], seq 716746136, win 29200, options [mss 1460,sackOK,TS val 3378691868 ecr 0,nop,wscale 7], length 0
18:39:09.326504 IP 10.186.62.30.8090 > 10.186.62.36.53554: Flags [R.], seq 0, ack 716746137, win 0, length 0
18:39:19.327551 IP 10.186.62.36.53556 > 10.186.62.30.8090: Flags [S], seq 391417268, win 29200, options [mss 1460,sackOK,TS val 3378701869 ecr 0,nop,wscale 7], length 0
18:39:19.327622 IP 10.186.62.30.8090 > 10.186.62.36.53556: Flags [R.], seq 0, ack 391417269, win 0, length 0
18:39:29.328549 IP 10.186.62.36.53558 > 10.186.62.30.8090: Flags [S], seq 1139945366, win 29200, options [mss 1460,sackOK,TS val 3378711870 ecr 0,nop,wscale 7], length 0
18:39:29.328606 IP 10.186.62.30.8090 > 10.186.62.36.53558: Flags [R.], seq 0, ack 1139945367, win 0, length 0
18:39:39.329520 IP 10.186.62.36.53560 > 10.186.62.30.8090: Flags [S], seq 92703179, win 29200, options [mss 1460,sackOK,TS val 3378721870 ecr 0,nop,wscale 7], length 0
18:39:39.329588 IP 10.186.62.30.8090 > 10.186.62.36.53560: Flags [R.], seq 0, ack 92703180, win 0, length 0
18:39:49.330560 IP 10.186.62.36.53562 > 10.186.62.30.8090: Flags [S], seq 1279627650, win 29200, options [mss 1460,sackOK,TS val 3378731871 ecr 0,nop,wscale 7], length 0
18:39:49.330633 IP 10.186.62.30.8090 > 10.186.62.36.53562: Flags [R.], seq 0, ack 1279627651, win 0, length 0
18:39:59.331731 IP 10.186.62.36.53564 > 10.186.62.30.8090: Flags [S], seq 88565865, win 29200, options [mss 1460,sackOK,TS val 3378741872 ecr 0,nop,wscale 7], length 0
18:39:59.331789 IP 10.186.62.30.8090 > 10.186.62.36.53564: Flags [R.], seq 0, ack 88565866, win 0, length 0
18:40:09.332903 IP 10.186.62.36.53566 > 10.186.62.30.8090: Flags [S], seq 3576850140, win 29200, options [mss 1460,sackOK,TS val 3378751873 ecr 0,nop,wscale 7], length 0
18:40:09.332970 IP 10.186.62.30.8090 > 10.186.62.36.53566: Flags [R.], seq 0, ack 3576850141, win 0, length 0
18:40:19.334073 IP 10.186.62.36.53568 > 10.186.62.30.8090: Flags [S], seq 2679062034, win 29200, options [mss 1460,sackOK,TS val 3378761873 ecr 0,nop,wscale 7], length 0
18:40:19.334135 IP 10.186.62.30.8090 > 10.186.62.36.53568: Flags [R.], seq 0, ack 2679062035, win 0, length 0
18:40:29.335141 IP 10.186.62.36.53570 > 10.186.62.30.8090: Flags [S], seq 4269406086, win 29200, options [mss 1460,sackOK,TS val 3378771874 ecr 0,nop,wscale 7], length 0
18:40:29.335211 IP 10.186.62.30.8090 > 10.186.62.36.53570: Flags [R.], seq 0, ack 4269406087, win 0, length 0
18:40:39.336447 IP 10.186.62.36.53572 > 10.186.62.30.8090: Flags [S], seq 4229523521, win 29200, options [mss 1460,sackOK,TS val 3378781875 ecr 0,nop,wscale 7], length 0
18:40:39.336526 IP 10.186.62.30.8090 > 10.186.62.36.53572: Flags [R.], seq 0, ack 4229523522, win 0, length 0
18:40:49.337534 IP 10.186.62.36.53574 > 10.186.62.30.8090: Flags [S], seq 1368872392, win 29200, options [mss 1460,sackOK,TS val 3378791876 ecr 0,nop,wscale 7], length 0
18:40:49.337598 IP 10.186.62.30.8090 > 10.186.62.36.53574: Flags [R.], seq 0, ack 1368872393, win 0, length 0
18:40:59.338657 IP 10.186.62.36.53576 > 10.186.62.30.8090: Flags [S], seq 1120189370, win 29200, options [mss 1460,sackOK,TS val 3378801876 ecr 0,nop,wscale 7], length 0
18:40:59.338728 IP 10.186.62.30.8090 > 10.186.62.36.53576: Flags [R.], seq 0, ack 1120189371, win 0, length 0
18:41:09.339700 IP 10.186.62.36.53578 > 10.186.62.30.8090: Flags [S], seq 2500468514, win 29200, options [mss 1460,sackOK,TS val 3378811877 ecr 0,nop,wscale 7], length 0
18:41:09.339748 IP 10.186.62.30.8090 > 10.186.62.36.53578: Flags [R.], seq 0, ack 2500468515, win 0, length 0
18:41:19.340731 IP 10.186.62.36.53580 > 10.186.62.30.8090: Flags [S], seq 1427684258, win 29200, options [mss 1460,sackOK,TS val 3378821878 ecr 0,nop,wscale 7], length 0
18:41:19.340793 IP 10.186.62.30.8090 > 10.186.62.36.53580: Flags [R.], seq 0, ack 1427684259, win 0, length 0
18:41:29.341793 IP 10.186.62.36.53582 > 10.186.62.30.8090: Flags [S], seq 2982299141, win 29200, options [mss 1460,sackOK,TS val 3378831878 ecr 0,nop,wscale 7], length 0
18:41:29.341871 IP 10.186.62.30.8090 > 10.186.62.36.53582: Flags [R.], seq 0, ack 2982299142, win 0, length 0
18:41:39.343009 IP 10.186.62.36.53584 > 10.186.62.30.8090: Flags [S], seq 4086577122, win 29200, options [mss 1460,sackOK,TS val 3378841879 ecr 0,nop,wscale 7], length 0
18:41:39.343066 IP 10.186.62.30.8090 > 10.186.62.36.53584: Flags [R.], seq 0, ack 4086577123, win 0, length 0
18:41:49.344106 IP 10.186.62.36.53586 > 10.186.62.30.8090: Flags [S], seq 2882697460, win 29200, options [mss 1460,sackOK,TS val 3378851880 ecr 0,nop,wscale 7], length 0
18:41:49.344176 IP 10.186.62.30.8090 > 10.186.62.36.53586: Flags [R.], seq 0, ack 2882697461, win 0, length 0

 

不会飞的小龙人
回复 @Gannalyo : 看一下有没有第三方的服务或组件通过HttpClient等客户端访问,没有及时合理的释放;
@ccny
@ccny
回复 @Gannalyo : 看一下 ,端口占用数量。可能是端口占用过多。 close_wait 是关闭连接中,频烦的http会这样 ,如果用的 容器,可以考虑加 端口重用。 或者一个 nginx
Gannalyo
Gannalyo
看此图的包HTTP请求没有回复 不知道是不是因为什么图中8090端口为CLOSE_WAIT状态的缘故
0
竹竿先生
竹竿先生

打印GC日志看下. 配置下tomcat的连接池

0
s
stormh

产生了类似半链接攻击,看看代码里面有没有发送关闭处理。另外可以设置一下linux的参数去优化,可以百度一下linux close_wait解决方法,主要配置sysctl.conf下面的参数。

根本解决办法还是要看看你请求rest接口的代码,该关闭的要及时关闭,或者用连接池方式。

0
唐伯虎117

是否存在端口拥堵情况,如果接口都使用一个端口如80端口请求,会造成请求拥堵超时,但是浏览器访问又无问题。

返回顶部
顶部