[Savannah-hackers-public] Re: [Savannah-hackers-private] frontend and vcs-noshell down

[Sylvain Beucler]

Ward replaced the disk.

Michael: consider using savannah-hackers-public ?

- Sylvain

On Sun, Dec 05, 2010 at 10:44:07AM -0500, Michael J. Flickinger wrote:
> Investigating issue, apparently the frontend and xen guests are down.
> 
> colonialone:~# ssh frontend.in.sv.gnu.org
> ssh_exchange_identification: Connection closed by remote host
> colonialone:~# ssh vcs-noshell.in.sv.gnu.org
> ssh_exchange_identification: Connection closed by remote host
> 
> I checked the console of the internal xen instance and this is what
> was spewing out:
> 
> [366400.520192]  =======================
> [366400.520197] INFO: task exim4:11450 blocked for more than 120 seconds.
> [366400.520204] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [366400.520211] exim4         D 13a5e000     0 11450  11448
> [366400.520220]        e8270a40 00000282 f57f8000 13a5e000 00000001
> e8270bc8 c115db40 00000000
> [366400.520236]        e8122040 c1a47120 0548538a 00000001 c1a47120
> c0112c2b c1a47140 00000001
> [366400.520250]        c0152edc f57f8000 00000002 00000000 00000000
> 001280d2 ed052338 ed052340
> [366400.520267] Call Trace:
> [366400.520274]  [<c0112c2b>] kunmap_atomic+0x35/0x4a
> [366400.520282]  [<c0152edc>] get_page_from_freelist+0x387/0x466
> [366400.520290]  [<c02cb376>] __mutex_lock_slowpath+0x67/0xa9
> [366400.520298]  [<c02cb1a2>] mutex_lock+0xa/0xb
> [366400.520307]  [<c014fc17>] generic_file_aio_write+0x41/0xa9
> [366400.520317]  [<ee072015>] ext3_file_write+0x19/0x83 [ext3]
> [366400.520330]  [<c0170052>] do_sync_write+0xbf/0x100
> [366400.520338]  [<c012eddc>] autoremove_wake_function+0x0/0x2d
> [366400.520348]  [<c010eaf3>] do_page_fault+0x605/0xb2e
> [366400.520357]  [<c010eb59>] do_page_fault+0x66b/0xb2e
> [366400.520364]  [<c01b9731>] security_file_permission+0xc/0xd
> [366400.520374]  [<c016ff93>] do_sync_write+0x0/0x100
> [366400.520383]  [<c01707c4>] vfs_write+0x83/0x120
> [366400.520390]  [<c0170d96>] sys_write+0x3c/0x63
> [366400.520399]  [<c0103f76>] syscall_call+0x7/0xb
> [366400.520406]  =======================
> [366419.819681] INFO: task syslogd:1066 blocked for more than 120 seconds.
> [366419.819700] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [366419.819709] syslogd       D 84505170     0  1066      1
> [366419.819722]        ecc13ac0 00000282 c115db40 84505170 00014279
> ecc13c48 c115db40 00000000
> [366419.819738]        ed743580 c03f3220 c03f3220 00002488 00000000
> 00000005 00137334 00000001
> [366419.819753]        00000000 c0116913 00004a4a ed79dedc ec82dc50
> c012eeeb ec82dc00 ed79dedc
> [366419.819769] Call Trace:
> [366419.819780]  [<c0116913>] __wake_up+0x29/0x39
> [366419.819793]  [<c012eeeb>] prepare_to_wait+0x12/0x4d
> [366419.819801]  [<ee0436e9>] log_wait_commit+0x9f/0xfc [jbd]
> [366419.819817]  [<c012eddc>] autoremove_wake_function+0x0/0x2d
> [366419.819825]  [<ee03f49a>] journal_stop+0x15d/0x1ad [jbd]
> [366419.819838]  [<c018a1c0>] __writeback_single_inode+0x193/0x28a
> [366419.819847]  [<c0153dc1>] generic_writepages+0x1a/0x21
> [366419.819859]  [<c0153df1>] do_writepages+0x29/0x30
> [366419.819866]  [<c014ed5f>] __filemap_fdatawrite_range+0x65/0x70
> [366419.819876]  [<c018aa7a>] sync_inode+0x19/0x43
> [366419.819883]  [<ee072164>] ext3_sync_file+0x88/0x9c [ext3]
> [366419.819900]  [<c018ccdd>] do_fsync+0x41/0x83
> [366419.819907]  [<c018cd3c>] __do_fsync+0x1d/0x2b
> [366419.819916]  [<c0103f76>] syscall_call+0x7/0xb
> [366419.819924]  =======================
> 
> 
> 
> [366424.120090]  [<c0103f76>] syscall_call+0x7/0xb
> [366424.120100]  =======================
> [366471.320257] INFO: task mysqld:11181 blocked for more than 120 seconds.
> [366471.320277] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [366471.320285] mysqld        D e827abac     0 11181   6470
> [366471.320294]        e823c180 00000286 c1ae5444 e827abac c010621f
> e823c308 c115db40 00000000
> [366471.320312]        ecb78900 32f5f402 05479c85 32f3ce08 0001761b
> e827abac 022d535a c1ae5444
> [366471.320329]        e827abac c0133348 022d535a e827abac 022d535a
> c1ae5444 c115db40 00d9e000
> [366471.320344] Call Trace:
> [366471.320352]  [<c010621f>] xen_clocksource_read+0xc/0x164
> [366471.320366]  [<c0133348>] getnstimeofday+0x37/0xbc
> [366471.320377]  [<c014fdf0>] sync_page_killable+0x0/0x2a
> [366471.320386]  [<c02caf1e>] io_schedule+0x49/0x80
> [366471.320395]  [<c014e627>] sync_page+0x33/0x36
> [366471.320403]  [<c014fdf5>] sync_page_killable+0x5/0x2a
> [366471.320410]  [<c02cb04a>] __wait_on_bit_lock+0x2a/0x52
> [366471.320419]  [<c014e592>] __lock_page_killable+0x51/0x57
> [366471.320426]  [<c012ee09>] wake_bit_function+0x0/0x3c
> [366471.320434]  [<c01502be>] generic_file_aio_read+0x331/0x4e1
> [366471.320442]  [<c0170152>] do_sync_read+0xbf/0xfe
> [366471.320453]  [<c012eddc>] autoremove_wake_function+0x0/0x2d
> [366471.320463]  [<c01b9731>] security_file_permission+0xc/0xd
> [366471.320473]  [<c0170093>] do_sync_read+0x0/0xfe
> [366471.320480]  [<c01708e2>] vfs_read+0x81/0x11e
> [366471.320487]  [<c0170d33>] sys_read+0x3c/0x63
> [366471.320496]  [<c0103f76>] syscall_call+0x7/0xb
> [366471.320506]  =======================