iSCSI 存储设备突然报错误 detected conn error (1020)

最近新采购了一台大容量存储服务器，在进行测试的时候突然报连接错误，然后设备进入只读模式。

错误信息如下：

DST=124.202.141.148 LEN=40 TOS=0x00 PREC=0x00 TTL=238 ID=54321 PROTO=TCP SPT=40614 DPT=11211 WINDOW=65535 RES=0x00 SYN URGP=0
Dec 24 06:35:08 git-web-1 kernel: [120121.373697] [UFW BLOCK] IN=em1 OUT= MAC=c8:1f:66:c7:41:47:00:25:b4:c1:2b:c5:08:00 SRC=198.74.110.164 DST=124.202.141.148 LEN=40 TOS=0x00 PREC=0x00 TTL=236 ID=2645 DF PROTO=TCP SPT=17849 DPT=9000 WINDOW=512 RES=0x00 SYN URGP=0
Dec 24 06:35:38 git-web-1 kernel: [120151.389800]  connection1:0: detected conn error (1020)
Dec 24 06:35:39 git-web-1 iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Dec 24 06:35:42 git-web-1 iscsid: connection1:0 is operational after recovery (1 attempts)
Dec 24 06:35:56 git-web-1 kernel: [120169.511075] [UFW BLOCK] IN=em1 OUT= MAC=c8:1f:66:c7:41:47:00:25:b4:c1:2b:c5:08:00 SRC=218.77.79.57 DST=124.202.141.148 LEN=40 TOS=0x00 PREC=0x00 TTL=239 ID=54321 PROTO=TCP SPT=50892 DPT=1186 WINDOW=65535 RES=0x00 SYN URGP=0
Dec 24 06:36:17 git-web-1 kernel: [120190.782293]  connection1:0: detected conn error (1020)
Dec 24 06:36:18 git-web-1 kernel: [120191.032966]  connection1:0: detected conn error (1021)
Dec 24 06:36:19 git-web-1 iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Dec 24 06:36:19 git-web-1 iscsid: Kernel reported iSCSI connection 1:0 error (1021 - ISCSI_ERR_SCSI_EH_SESSION_RST: Session was dropped as a result of SCSI error recovery) state (1)
Dec 24 06:36:21 git-web-1 iscsid: connection1:0 is operational after recovery (1 attempts)
Dec 24 06:36:31 git-web-1 kernel: [120204.534811]  connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4324917565, last ping 4324918815, now 4324920068
"syslog" 4051L, 454714C                                                                                                  1,1           Top
Dec 24 06:37:00 git-web-1 iscsid: Kernel reported iSCSI connection 1:0 error (1011 - ISCSI_ERR_CONN_FAILED: iSCSI connection failed) state (3)
Dec 24 06:37:03 git-web-1 iscsid: connection1:0 is operational after recovery (1 attempts)
Dec 24 06:37:38 git-web-1 kernel: [120271.203587]  connection1:0: detected conn error (1020)
Dec 24 06:37:39 git-web-1 iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Dec 24 06:37:41 git-web-1 kernel: [120274.210675] sd 7:0:0:0: [sde] Medium access timeout failure. Offlining disk!
Dec 24 06:37:41 git-web-1 kernel: [120274.212853] sd 7:0:0:0: Device offlined - not ready after error recovery
Dec 24 06:37:41 git-web-1 kernel: [120274.212863] sd 7:0:0:0: [sde] Unhandled error code
Dec 24 06:37:41 git-web-1 kernel: [120274.212866] sd 7:0:0:0: [sde]
Dec 24 06:37:41 git-web-1 kernel: [120274.212868] Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK
Dec 24 06:37:41 git-web-1 kernel: [120274.212871] sd 7:0:0:0: [sde] CDB:
Dec 24 06:37:41 git-web-1 kernel: [120274.212873] Write(16): 8a 00 00 00 00 00 7e 78 c4 22 00 00 04 00 00 00
Dec 24 06:37:41 git-web-1 kernel: [120274.212889] end_request: I/O error, dev sde, sector 2121843746
Dec 24 06:37:41 git-web-1 kernel: [120274.214697] EXT4-fs warning (device sde1): ext4_end_bio:317: I/O error -5 writing to inode 56525453 (offset 58720256 size 8388608 starting block 265230596)
Dec 24 06:37:41 git-web-1 kernel: [120274.214701] Buffer I/O error on device sde1, logical block 265230464
Dec 24 06:37:41 git-web-1 kernel: [120274.216648] Buffer I/O error on device sde1, logical block 265230465
Dec 24 06:37:41 git-web-1 kernel: [120274.218601] Buffer I/O error on device sde1, logical block 265230466
Dec 24 06:37:41 git-web-1 kernel: [120274.220547] Buffer I/O error on device sde1, logical block 265230467
Dec 24 06:37:41 git-web-1 kernel: [120274.222500] Buffer I/O error on device sde1, logical block 265230468
Dec 24 06:37:41 git-web-1 kernel: [120274.224446] Buffer I/O error on device sde1, logical block 265230469
Dec 24 06:37:41 git-web-1 kernel: [120274.226397] Buffer I/O error on device sde1, logical block 265230470
Dec 24 06:37:41 git-web-1 kernel: [120274.228343] Buffer I/O error on device sde1, logical block 265230471
Dec 24 06:37:41 git-web-1 kernel: [120274.230293] Buffer I/O error on device sde1, logical block 265230472
Dec 24 06:37:41 git-web-1 kernel: [120274.232238] Buffer I/O error on device sde1, logical block 265230473
                                                                                                                         37,1           0%
Dec 24 06:37:42 git-web-1 kernel: [120274.600677] sd 7:0:0:0: rejecting I/O to offline device
Dec 24 06:37:42 git-web-1 kernel: [120274.606862] EXT4-fs warning (device sde1): ext4_end_bio:317: I/O error -5 writing to inode 56525454 (offset 50331648 size 8388608 starting block 265234180)
Dec 24 06:37:42 git-web-1 kernel: [120274.606962] sd 7:0:0:0: rejecting I/O to offline device
Dec 24 06:37:42 git-web-1 kernel: [120274.613121] EXT4-fs warning (device sde1): ext4_end_bio:317: I/O error -5 writing to inode 56525454 (offset 50331648 size 8388608 starting block 265234308)
Dec 24 06:37:42 git-web-1 kernel: [120274.613221] sd 7:0:0:0: rejecting I/O to offline device
Dec 24 06:37:42 git-web-1 kernel: [120274.619501] sd 7:0:0:0: rejecting I/O to offline device
Dec 24 06:37:42 git-web-1 kernel: [120274.625639] sd 7:0:0:0: rejecting I/O to offline device

重新 logout iSCSI 设备再 login 后就恢复访问。

一般这种错误会是什么原因引起呢，现在故障没法重现出来。

我们做的测试是大量往设备里同步数据，已经同步了两天了，是今天早上突然数据写入失败。

操作系统是 Ubuntu 14.04，现在存储设备的技术支持也搞不清楚什么原因，他们怀疑会不会有网络的临时中断导致的问题。而通过设备的管理控制台看到的存储设备状态一切正常。