推 ZFS
推 Block mode iSCSI
推 thick mode iSCSI at Filesystem
不推 thin mode iSCSI at btrfs Filesystem
觀察 每一個 datastore寫入延遲有多大？
讀取延遲有多大？
如果有超過100ms，就要考慮換架構了！
root@w26:~# zpool status
pool: rpool
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-9P
scan: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
sda2 ONLINE 0 0 0
sdb2 ONLINE 0 0 0
sdc2 ONLINE 0 0 8
errors: No known data errors
root@w26:~#
Dec 6 11:30:01 w26 systemd[1]: Started Proxmox VE replication runner.
Dec 6 11:30:40 w26 kernel: [167258.110376] ata3.00: exception Emask 0x10 SAct 0x40000001 SErr 0x400100 action 0x6 frozen
Dec 6 11:30:40 w26 kernel: [167258.111069] ata3.00: irq_stat 0x08000000, interface fatal error
Dec 6 11:30:40 w26 kernel: [167258.111733] ata3: SError: { UnrecovData Handshk }
Dec 6 11:30:40 w26 kernel: [167258.112380] ata3.00: failed command: WRITE FPDMA QUEUED
Dec 6 11:30:40 w26 kernel: [167258.113044] ata3.00: cmd 61/08:00:c8:1c:16/00:00:10:00:00/40 tag 0 ncq dma 4096 out
Dec 6 11:30:40 w26 kernel: [167258.113044] res 40/00:04:c8:1c:16/00:00:10:00:00/40 Emask 0x10 (ATA bus error)
Dec 6 11:30:40 w26 kernel: [167258.114363] ata3.00: status: { DRDY }
Dec 6 11:30:40 w26 kernel: [167258.115056] ata3.00: failed command: WRITE FPDMA QUEUED
Dec 6 11:30:40 w26 kernel: [167258.115816] ata3.00: cmd 61/38:f0:98:bd:5d/00:00:95:00:00/40 tag 30 ncq dma 28672 out
Dec 6 11:30:40 w26 kernel: [167258.115816] res 40/00:04:c8:1c:16/00:00:10:00:00/40 Emask 0x10 (ATA bus error)
Dec 6 11:30:40 w26 kernel: [167258.117115] ata3.00: status: { DRDY }
Dec 6 11:30:40 w26 kernel: [167258.117756] ata3: hard resetting link
Dec 6 11:30:41 w26 kernel: [167258.434432] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Dec 6 11:30:41 w26 kernel: [167258.446878] ata3.00: configured for UDMA/33
Dec 6 11:30:41 w26 kernel: [167258.446882] sd 2:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 6 11:30:41 w26 kernel: [167258.446884] sd 2:0:0:0: [sdc] tag#0 Sense Key : Illegal Request [current]
Dec 6 11:30:41 w26 kernel: [167258.446884] sd 2:0:0:0: [sdc] tag#0 Add. Sense: Unaligned write command
@
Dec 6 11:30:41 w26 kernel: [167258.447552] sd 2:0:0:0: [sdc] tag#30 CDB: Write(16) 8a 00 00 00 00 00 95 5d bd 98 00 00 00 38 00 00
Dec 6 11:30:41 w26 kernel: [167258.447552] print_req_error: I/O error, dev sdc, sector 2505948568
Dec 6 11:30:41 w26 kernel: [167258.448217] ata3: EH complete
Dec 6 11:31:00 w26 systemd[1]: Starting Proxmox VE replication runner...
Dec 6 11:31:01 w26 systemd[1]: Started Proxmox VE replication runner.
Dec 6 11:31:31 w26 kernel: [167309.314491] ata3.00: exception Emask 0x10 SAct 0x6 SErr 0x400100 action 0x6 frozen
Dec 6 11:31:31 w26 kernel: [167309.315184] ata3.00: irq_stat 0x08000000, interface fatal error
Dec 6 11:31:31 w26 kernel: [167309.315853] ata3: SError: { UnrecovData Handshk }
Dec 6 11:31:31 w26 kernel: [167309.316509] ata3.00: failed command: WRITE FPDMA QUEUED
Dec 6 11:31:31 w26 kernel: [167309.317170] ata3.00: cmd 61/38:08:b8:56:bc/00:00:52:00:00/40 tag 1 ncq dma 28672 out
Dec 6 11:31:31 w26 kernel: [167309.317170] res 40/00:14:78:c0:5d/00:00:95:00:00/40 Emask 0x10 (ATA bus error)
Dec 6 11:31:31 w26 kernel: [167309.318569] ata3.00: status: { DRDY }
Dec 6 11:31:31 w26 kernel: [167309.319308] ata3.00: failed command: WRITE FPDMA QUEUED
Dec 6 11:31:31 w26 kernel: [167309.319961] ata3.00: cmd 61/38:10:78:c0:5d/00:00:95:00:00/40 tag 2 ncq dma 28672 out
Dec 6 11:31:31 w26 kernel: [167309.319961] res 40/00:14:78:c0:5d/00:00:95:00:00/40 Emask 0x10 (ATA bus error)
Dec 6 11:31:31 w26 kernel: [167309.321316] ata3.00: status: { DRDY }
Dec 6 11:31:31 w26 kernel: [167309.321973] ata3: hard resetting link
Dec 6 11:31:32 w26 kernel: [167309.638379] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Dec 6 11:31:32 w26 kernel: [167309.650802] ata3.00: configured for UDMA/33
Dec 6 11:31:32 w26 kernel: [167309.650807] sd 2:0:0:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 6 11:31:32 w26 kernel: [167309.650808] sd 2:0:0:0: [sdc] tag#1 Sense Key : Illegal Request [current]
Dec 6 11:31:32 w26 kernel: [167309.650809] sd 2:0:0:0: [sdc] tag#1 Add. Sense: Unaligned write command
Dec 6 11:31:32 w26 kernel: [167309.650810] sd 2:0:0:0: [sdc] tag#1 CDB: Write(16) 8a 00 00 00 00 00 52 bc 56 b8 00 00 00 38 00 00
Dec 6 11:31:32 w26 kernel: [167309.650810] print_req_error: I/O error, dev sdc, sector 1388074680
Dec 6 11:31:32 w26 kernel: [167309.651481] sd 2:0:0:0: [sdc] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 6 11:31:32 w26 kernel: [167309.651481] sd 2:0:0:0: [sdc] tag#2 Sense Key : Illegal Request [current]
Dec 6 11:31:32 w26 kernel: [167309.651482] sd 2:0:0:0: [sdc] tag#2 Add. Sense: Unaligned write command
Dec 6 11:31:32 w26 kernel: [167309.651483] sd 2:0:0:0: [sdc] tag#2 CDB: Write(16) 8a 00 00 00 00 00 95 5d c0 78 00 00 00 38 00 00
Dec 6 11:31:32 w26 kernel: [167309.651483] print_req_error: I/O error, dev sdc, sector 2505949304
Dec 6 11:31:32 w26 kernel: [167309.652176] ata3: EH complete
Dec 6 11:32:00 w26 systemd[1]: Starting Proxmox VE replication runner...
Dec 6 11:32:01 w26 systemd[1]: Started Proxmox VE replication runner.
Dec 6 11:33:00 w26 systemd[1]: Starting Proxmox VE replication runner...
Dec 6 11:33:01 w26 systemd[1]: Started Proxmox VE replication runner.
Dec 6 11:33:19 w26 smartd[3087]: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 65 to 64
Dec 6 11:33:19 w26 smartd[3087]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 35 to 36
Dec 6 11:33:29 w26 kernel: [167427.074173] ata3.00: exception Emask 0x10 SAct 0x180000 SErr 0x400100 action 0x6 frozen
Dec 6 11:33:29 w26 kernel: [167427.075032] ata3.00: irq_stat 0x08000000, interface fatal error
Dec 6 11:33:29 w26 kernel: [167427.075807] ata3: SError: { UnrecovData Handshk }
Dec 6 11:33:29 w26 kernel: [167427.076587] ata3.00: failed command: WRITE FPDMA QUEUED
Dec 6 11:33:29 w26 kernel: [167427.077363] ata3.00: cmd 61/30:98:f8:62:bc/00:00:52:00:00/40 tag 19 ncq dma 24576 out
Dec 6 11:33:29 w26 kernel: [167427.077363] res 40/00:9c:f8:62:bc/00:00:52:00:00/40 Emask 0x10 (ATA bus error)
Dec 6 11:33:29 w26 kernel: [167427.078982] ata3.00: status: { DRDY }
Dec 6 11:33:29 w26 kernel: [167427.079741] ata3.00: failed command: WRITE FPDMA QUEUED
Dec 6 11:33:29 w26 kernel: [167427.080390] ata3.00: cmd 61/38:a0:38:c6:5d/00:00:95:00:00/40 tag 20 ncq dma 28672 out
Dec 6 11:33:29 w26 kernel: [167427.080390] res 40/00:9c:f8:62:bc/00:00:52:00:00/40 Emask 0x10 (ATA bus error)
Dec 6 11:33:29 w26 kernel: [167427.081694] ata3.00: status: { DRDY }
Dec 6 11:33:29 w26 kernel: [167427.082422] ata3: hard resetting link
Dec 6 11:33:29 w26 kernel: [167427.397998] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Dec 6 11:33:29 w26 kernel: [167427.409993] ata3.00: configured for UDMA/33
Dec 6 11:33:29 w26 kernel: [167427.409998] sd 2:0:0:0: [sdc] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 6 11:33:29 w26 kernel: [167427.409999] sd 2:0:0:0: [sdc] tag#19 Sense Key : Illegal Request [current]
Dec 6 11:33:29 w26 kernel: [167427.410000] sd 2:0:0:0: [sdc] tag#19 Add. Sense: Unaligned write command
Dec 6 11:33:29 w26 kernel: [167427.410001] sd 2:0:0:0: [sdc] tag#19 CDB: Write(16) 8a 00 00 00 00 00 52 bc 62 f8 00 00 00 30 00 00
Dec 6 11:33:29 w26 kernel: [167427.410002] print_req_error: I/O error, dev sdc, sector 1388077816
Dec 6 11:33:29 w26 kernel: [167427.410711] sd 2:0:0:0: [sdc] tag#20 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 6 11:33:29 w26 kernel: [167427.410712] sd 2:0:0:0: [sdc] tag#20 Sense Key : Illegal Request [current]
Dec 6 11:33:29 w26 kernel: [167427.410713] sd 2:0:0:0: [sdc] tag#20 Add. Sense: Unaligned write command
Dec 6 11:33:29 w26 kernel: [167427.410714] sd 2:0:0:0: [sdc] tag#20 CDB: Write(16) 8a 00 00 00 00 00 95 5d c6 38 00 00 00 38 00 00
Dec 6 11:33:29 w26 kernel: [167427.410716] print_req_error: I/O error, dev sdc, sector 2505950776
Dec 6 11:33:29 w26 kernel: [167427.411467] ata3: EH completeThis message was generated by the smartd daemon running on:
host name: w28
DNS domain: wcjs.tc.edu.tw
The following warning/error was logged by the smartd daemon:
Device: /dev/sdb [SAT], 204 Currently unreadable (pending) sectors
Device info:
WDC WD30EZRX-00AZ6B0, S/N:WD-WMC070076511, WWN:5-0014ee-25c29c3c6, FW:80.00A80, 3.00 TB
For details see host's SYSLOG.
You can also use the smartctl utility for further investigation.
The original message about this issue was sent at Thu Dec 7 13:33:38 2017 CST
Another message will be sent in 24 hours if the problem persists.
host name: w28
DNS domain: wcjs.tc.edu.tw
The following warning/error was logged by the smartd daemon:
Device: /dev/sdb [SAT], 204 Currently unreadable (pending) sectors
Device info:
WDC WD30EZRX-00AZ6B0, S/N:WD-WMC070076511, WWN:5-0014ee-25c29c3c6, FW:80.00A80, 3.00 TB
For details see host's SYSLOG.
You can also use the smartctl utility for further investigation.
The original message about this issue was sent at Thu Dec 7 13:33:38 2017 CST
Another message will be sent in 24 hours if the problem persists.
沒有留言:
張貼留言