Re: [Qemu-devel] [network performance question] only ~2Gbpsthroughputbetweentwo linux guests which are running on the same host vianetperf-tTCP_STREAM -m 1400, but xen can ac

[Michael S. Tsirkin]

I see that RX checksumming is still off for you on virtio, this is
likely what's contribution to the problem.

Here's how it looks for me:
ethtool -k eth1
        Offload parameters for eth1:
        rx-checksumming: on
        tx-checksumming: on
        scatter-gather: on
        tcp-segmentation-offload: on
        udp-fragmentation-offload: on
        generic-segmentation-offload: on
        generic-receive-offload: on
        large-receive-offload: off

you don't supply kernel versions for host or guest kernels,
so it's hard to judge what's going on exactly.

Bridge configuration also plays a huge role.
Things like ebtables might affect performance as well,
sometimes even if they are only loaded, not even enabled.

Also, some old scheduler versions didn't put VMs on different
CPUs aggressively enough, this resulted in conflicts
when VMs compete for the same CPU.
On numa systems, some older host kernels would split VM memory
across NUMA nodes, this might lead to bad performance.



On Sat, Jun 07, 2014 at 11:07:10AM +0800, Zhang Haoyu wrote:
> After updating the qemu from 1.4 to 2.0, the inter-vm throughput can achieve 
> ~5Gbps via netper -t TCP_STREAM -m 1400,
> the performance gap(~2Gbps) between kvm and xen still exist.
> 
> Thanks,
> Zhang Haoyu
> 
> ------------------                             
> Zhang Haoyu
> 2014-06-07
> 
> -----Original Message-----
> From: Zhang Haoyu
> Sent: 2014-06-07 09:27:16
> To: Venkateswara Rao Nandigam; kvm; qemu-devel
> Cc: Gleb Natapov; Paolo Bonzini; Michael S.Tsirkin; yewudi
> Subject: Re: [network performance question] only ~2Gbpsthroughputbetweentwo 
> linux guests which are running on the same host vianetperf-tTCP_STREAM -m 
> 1400, but xen can ac
> 
> > Doesn't that answer your original question about performance gap!
> Sorry, do you mean it's the offloadings cause the performance gap?
> But even OFF the checksum-offload, tso, gro, .etc, the performance gap still 
> exist,
> if I understand correctly, kvm should have better performance than xen from 
> the angle of implementation, because of shorter path, and fewer 
> context-switches,
> especially inter-vm communication.
> 
> And, why the performance gap is so big(~2G vs ~7G) when checksum-offload, 
> tso, gro, .etc is on for both hypervisors?
> Why the packes' size can be so big(65160) and stable on xen, but most 
> packets' size is 1448, only a few part is ~65000 on kvm, when netperf -t 
> TCP_STREAM -m 1400 ?
> Does some TCP configurations have buissness with this? Or some virtio-net 
> configurations?
> 
> Thanks,
> Zhang Haoyu
> 
> -----Original Message-----
> From: address@hidden [mailto:address@hidden On Behalf Of Zhang Haoyu
> Sent: Friday, June 06, 2014 3:44 PM
> To: Venkateswara Rao Nandigam; kvm; qemu-devel
> Cc: Gleb Natapov; Paolo Bonzini; Michael S.Tsirkin; yewudi
> Subject: Re: RE: [network performance question] only ~2Gbps 
> throughputbetweentwo linux guests which are running on the same host via 
> netperf-tTCP_STREAM -m 1400, but xen can ac
> 
> > >> Try Rx/Tx checksum offload on the all the concerned guests of  both 
> > >> Hypervisors.
> > >> 
> > >Already ON on both hypervisors, so some other offloadings(e.g. tso, gso) 
> > >can be supported.
> > 
> > Try Rx/Tx checksum offload "OFF" on the all the concerned guests of  
> > both Hypervisors
> > 
> Off  Rx/Tx checksum offload on XEN guest, 1.6Gbps achived, tcpdump result on 
> backend vif netdeivce shown that packets' size is 1448, stable.
> Off Rx/Tx checksum offload on KVM guest, only ~1Gbps ahchived, tcpdump result 
> on backend tap netdevice shown that packets' size is 1448, stable.
> 
> > And While launching the VM in KVM, in command line of virtio interface, you 
> > can specify TSO, LRO, RxMergebuf. Try this instead of ethtool interface.
> The cuurent qemu command shown as below, and I will change the virtio-net 
> configuration later as your advise, /usr/bin/kvm -id 8572667846472 -chardev 
> socket,id=qmp,path=/var/run/qemu-server/8572667846472.qmp,server,nowait -mon 
> chardev=qmp,mode=control -vnc :0,websocket,to=200,x509,password -pidfile 
> /var/run/qemu-server/8572667846472.pid -daemonize -name centos6-196.5.5.72 
> -smp sockets=1,cores=2 -cpu core2duo -nodefaults -vga cirrus -no-hpet -k 
> en-us -boot menu=on,splash-time=8000 -m 4096 -usb -drive 
> file=/sf/data/local/iso/vmtools/virtio_auto_install.iso,if=none,id=drive-ide0,media=cdrom,aio=threads,forecast=disable
>  -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=200 
> -drive 
> file=/sf/data/local/images/host-f8bc123b3e74/32f49b646d1e/centos6-196.5.5.72.vm/vm-disk-1.qcow2,if=none,id=drive-ide2,cache=directsync,aio=threads,forecast=disable
>  -device ide-hd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=100 
> -netdev type=tap,id=net0,ifname=857266784647200,scrip
>  t=/sf/etc/kvm/vtp-bridge,vhost=on,vhostforce=on -device 
> virtio-net-pci,mac=FE:FC:FE:95:EC:A7,netdev=net0,bus=p
>  ci.0,addr=0x12,id=net0,bootindex=300 -rtc driftfix=slew,clock=rt -global 
> kvm-pit.lost_tick_policy=discard -global PIIX4_PM.disable_s3=1 -global 
> PIIX4_PM.disable_s4=1
>  
> -----Original Message-----
> From: Zhang Haoyu [mailto:address@hidden
> Sent: Friday, June 06, 2014 1:26 PM
> To: Venkateswara Rao Nandigam; kvm; qemu-devel
> Cc: Gleb Natapov; Paolo Bonzini; Michael S.Tsirkin; yewudi
> Subject: RE: [network performance question] only ~2Gbps throughput betweentwo 
> linux guests which are running on the same host via netperf -tTCP_STREAM -m 
> 1400, but xen can ac
> 
> Thanks for reply.
> > >>> And, vhost enabled, tx zero-copy enabled, virtio TSO enabled on kvm.
> > 
> > Try lro "ON" on client side. This would require mergable Rx buffers to be 
> > ON.
> > 
> current setttings for gro and lro,
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> 
> > And  Xen netfront to KVM virtio are not apples to apples because of their 
> > implementation details. 
> > 
> You are right, I just want to make network performance comparison between the 
> two virtualization platform from the view of user.
> 
> > Try Rx/Tx checksum offload on the all the concerned guests of  both 
> > Hypervisors.
> > 
> Already ON on both hypervisors, so some other offloadings(e.g. tso, gso) can 
> be supported.
> 
> kvm virtio-net nic:
> ethtool -k eth0
> Features for eth0:
> rx-checksumming: off [fixed]
> tx-checksumming: on
>     tx-checksum-ipv4: off [fixed]\
>       tx-checksum-ip-generic: on
>       tx-checksum-ipv6: off [fixed]
>       tx-checksum-fcoe-crc: off [fixed]
>       tx-checksum-sctp: off [fixed]
> scatter-gather: on
>     tx-scatter-gather: on
>       scatter-gather-fraglist: on
> tcp-segmentation-offload: on
>     tx-tcp-segmentation: on
>       tx-tcp-ecn-segmentation: on
>       tx-tcp6-segmentation: on
> udp-fragmentation-offload: on
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: off [fixed]
> tx-vlan-offload: off [fixed]
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: on [fixed]
> rx-vlan-filter: on [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: off [fixed]
> tx-udp_tnl-segmentation: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: on
> loopback: off [fixed]
> rx-fcs: off [fixed]
> rx-all: off [fixed]
> tx-vlan-stag-hw-insert: off [fixed]
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> 
> xen netfront nic:
> ethtool -k eth0
> Offload features for eth0:
> rx-checksumming: on
> tx-checksumming: on
> scatter-gather: on
> tcp-segmentation-offload: on
> udp-fragmentation-offload: off
> generic-segmentation-offload: on
> generic-receive-offload: off
> large-receive-offload: off
> 
> <piece of tcpdump result on xen backend vif netdevice >
> 15:46:41.279954 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
> 1193138968:1193204128, ack 1, win 115, options [nop,nop,TS val 102307210 ecr 
> 102291188], length 65160
> 15:46:41.279971 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
> 1193204128:1193269288, ack 1, win 115, options [nop,nop,TS val 102307210 ecr 
> 102291188], length 65160
> 15:46:41.279987 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
> 1193269288:1193334448, ack 1, win 115, options [nop,nop,TS val 102307210 ecr 
> 102291188], length 65160
> 15:46:41.280003 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
> 1193334448:1193399608, ack 1, win 115, options [nop,nop,TS val 102307210 ecr 
> 102291188], length 65160
> 15:46:41.280020 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
> 1193399608:1193464768, ack 1, win 115, options [nop,nop,TS val 102307210 ecr 
> 102291188], length 65160
> 15:46:41.280213 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
> 1193464768:1193529928, ack 1, win 115, options [nop,nop,TS val 102307211 ecr 
> 102291189], length 65160
> 15:46:41.280233 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
> 1193529928:1193595088, ack 1, win 115, options [nop,nop,TS val 102307211 ecr 
> 102291189], length 65160
> 15:46:41.280250 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
> 1193595088:1193660248, ack 1, win 115, options [nop,nop,TS val 102307211 ecr 
> 102291189], length 65160
> 15:46:41.280239 IP 196.6.6.71.53622 > 196.6.6.72.53507: Flags [.], ack 
> 1193138968, win 22399, options [nop,nop,TS val 102291190 ecr 102307210], 
> length 0
> 15:46:41.280267 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
> 1193660248:1193725408, ack 1, win 115, options [nop,nop,TS val 102307211 ecr 
> 102291189], length 65160
> 15:46:41.280284 IP 196.6.6.72.53507 > 196.6.6.71.53622: Flags [.], seq 
> 1193725408:1193790568, ack 1, win 115, options [nop,nop,TS val 102307211 ecr 
> 102291189], length 65160
> 
> Packets' size is very stable, 65160 Bytes.
> 
> Thanks,
> Zhang Haoyu
> 
> -----Original Message-----
> From: address@hidden [mailto:address@hidden On Behalf Of Zhang Haoyu
> Sent: Friday, June 06, 2014 9:01 AM
> To: kvm; qemu-devel
> Cc: Gleb Natapov; Paolo Bonzini; Michael S.Tsirkin; yewudi
> Subject: [network performance question] only ~2Gbps throughput between two 
> linux guests which are running on the same host via netperf -t TCP_STREAM -m 
> 1400, but xen can achieve ~7Gbps
> 
> Hi, all
> 
> I ran two linux guest on the same kvm host, then start the netserver on one 
> vm, start netperf on the other one,  netperf command and test result shown as 
> below, netperf -H 196.5.5.71 -t TCP_STREAM -l 60 -- -m 1400 -M 1400 MIGRATED 
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 () port 0 
> AF_INET : nodelay
> Recv   Send    Send                          
> Socket Socket  Message  Elapsed              
> Size   Size    Size     Time     Throughput  
> bytes  bytes   bytes    secs.    10^6bits/sec  
> 
>  87380  16384   1400    60.01    2355.45   
> 
> but I ran two linux guest on the same xen hypervisor, ~7Gbps throughput 
> achived, netperf command and test result shown as below, netperf -H 
> 196.5.5.71 -t TCP_STREAM -l 60 -- -m 1400 -M 1400 MIGRATED TCP STREAM TEST 
> from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 () port 0 AF_INET
> Recv   Send    Send                          
> Socket Socket  Message  Elapsed              
> Size   Size    Size     Time     Throughput  
> bytes  bytes   bytes    secs.    10^6bits/sec  
> 
>  87380  16384   1400    60.01    2349.82
> 
> many times test performed, the result is similar as above.
> 
> When I tcpdump backend tap netdevice, found that most packets' size is 
> 1448bytes on kvm, and few packets are ~60000Bytes, but  I tcpdump backend vif 
> netdevice, found that  most packets' size is >60000bytes on xen.
> Test result of netperf -t TCP_STREAM -m 64 is similar, more larger packets on 
> xen than kvm. 
> 
> And, vhost enabled, tx zero-copy enabled, virtio TSO enabled on kvm.
> 
> Any ideas?
> 
> Thanks,
> Zhang Haoyu
> 
> <environment>
> guest: linux-3.10, or centos-6.3
> host: linux-3.10(kvm), xenserver-6.2
> 
> <test tool>
> netperf-2.6
> 
> <test result details>
> 1. KVM
> netperf -H 196.5.5.71 -t TCP_STREAM -l 60 -- -m 65000 -M 65000 MIGRATED TCP 
> STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 () port 0 
> AF_INET
> Recv   Send    Send                          
> Socket Socket  Message  Elapsed              
> Size   Size    Size     Time     Throughput  
> bytes  bytes   bytes    secs.    10^6bits/sec  
> 
>  87380  16384  65000    60.00    13743.44 
> 
> netperf -H 196.5.5.71 -t TCP_STREAM -l 60 -- -m 1400 -M 1400 MIGRATED TCP 
> STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 () port 0 
> AF_INET
> Recv   Send    Send                          
> Socket Socket  Message  Elapsed              
> Size   Size    Size     Time     Throughput  
> bytes  bytes   bytes    secs.    10^6bits/sec  
> 
>  87380  16384   1400    60.01    2349.82   
> 
> netperf -H 196.5.5.71 -t TCP_STREAM -l 60 -- -m 64 -M 64 MIGRATED TCP STREAM 
> TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.5.5.71 () port 0 AF_INET
> Recv   Send    Send                          
> Socket Socket  Message  Elapsed              
> Size   Size    Size     Time     Throughput  
> bytes  bytes   bytes    secs.    10^6bits/sec  
> 
>  87380  16384     64    60.01     294.70  
> 
> 2. XEN
> netperf -H 196.6.6.71 -t TCP_STREAM -l 60 -- -m 65000 -M 65000 MIGRATED TCP 
> STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.6.6.71 () port 0 
> AF_INET
> Recv   Send    Send                          
> Socket Socket  Message  Elapsed              
> Size   Size    Size     Time     Throughput  
> bytes  bytes   bytes    secs.    10^6bits/sec  
> 
>  87380  16384  65000    60.01    6327.46 
> 
> netperf -H 196.6.6.71 -t TCP_STREAM -l 60 -- -m 1400 -M 1400 MIGRATED TCP 
> STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.6.6.71 () port 0 
> AF_INET
> Recv   Send    Send                          
> Socket Socket  Message  Elapsed              
> Size   Size    Size     Time     Throughput  
> bytes  bytes   bytes    secs.    10^6bits/sec  
> 
>  87380  16384   1400    60.00    7250.63  
> 
> netperf -H 196.6.6.71 -t TCP_STREAM -l 60 -- -m 64 -M 64 MIGRATED TCP STREAM 
> TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 196.6.6.71 () port 0 AF_INET
> Recv   Send    Send                          
> Socket Socket  Message  Elapsed              
> Size   Size    Size     Time     Throughput  
> bytes  bytes   bytes    secs.    10^6bits/sec  
> 
>  87380  16384     64    60.00     668.25  
> 
> <piece of tcpdump result on backend tap netdevice>
> 11:26:28.907130 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5421192:5422640, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907146 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5422640:5424088, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907161 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 5419328, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
> 11:26:28.907164 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5424088:5425536, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907183 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5426984:5428432, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907203 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 5421192, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
> 11:26:28.907217 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5428432:5429880, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907225 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5429880:5431328, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907238 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5431328:5432776, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907256 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5432776:5434224, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907268 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 5426984, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
> 11:26:28.907294 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [P.], seq 
> 5434224:5434692, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 468
> 11:26:28.907311 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 5429880, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
> 11:26:28.907319 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 5432776, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
> 11:26:28.907345 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5434692:5436140, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907363 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5436140:5437588, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907373 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 5434692, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
> 11:26:28.907385 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5437588:5439036, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907411 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5439036:5440484, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907427 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [P.], seq 
> 5440484:5440692, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 208
> 11:26:28.907427 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 5437588, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
> 11:26:28.907445 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5440692:5442140, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907453 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5442140:5443588, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907468 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5443588:5445036, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907494 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5445036:5446484, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907496 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 5440484, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
> 11:26:28.907506 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5446484:5447932, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907539 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5447932:5449380, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907551 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 5445036, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
> 11:26:28.907555 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5449380:5450828, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907566 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [P.], seq 
> 5450828:5451192, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 364
> 11:26:28.907605 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5451192:5452640, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907623 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5452640:5454088, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907631 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5454088:5455536, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907646 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5455536:5456984, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907662 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5456984:5458432, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907670 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5458432:5459880, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907690 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5459880:5461328, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907699 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5461328:5462776, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907704 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 5450828, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
> 11:26:28.907708 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5462776:5464224, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907721 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5464224:5465672, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907739 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5465672:5467120, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907764 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 5468568:5470016, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 1448
> 11:26:28.907793 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [P.], seq 
> 5470016:5470692, ack 1, win 115, options [nop,nop,TS val 497408 ecr 497985], 
> length 676
> 11:26:28.907796 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 5455536, win 24576, options [nop,nop,TS val 497985 ecr 497408], length 0
> 11:26:28.937289 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 7273240:7274688, ack 1, win 115, options [nop,nop,TS val 497416 ecr 497988], 
> length 1448
> 11:26:28.941798 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6259640, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941815 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6262536, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941820 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6265432, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941825 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6268328, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941829 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6271224, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941834 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6274120, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941839 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6277016, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941843 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6279912, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941848 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6282808, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941852 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6285704, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941856 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6288600, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941873 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6291496, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941888 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6294392, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941893 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6297288, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941904 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6300184, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941909 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6303080, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941919 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6305976, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941924 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6308872, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941934 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6311768, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941939 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6314664, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941950 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6317560, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941955 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6320456, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941961 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6323352, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941966 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6326248, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941967 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 7274688:7276136, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497993], 
> length 1448
> 11:26:28.941970 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6329144, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941975 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6332040, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941980 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6334936, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941985 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6337832, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941991 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6342176, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.941995 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6345072, win 24576, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.942001 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6362448, win 24441, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.942118 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 7276136:7338400, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497993], 
> length 62264
> 11:26:28.942220 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6579648, win 22745, options [nop,nop,TS val 497993 ecr 497411], length 0
> 11:26:28.942456 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 7338400:7403560, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497993], 
> length 65160
> 11:26:28.942501 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 7403560:7468720, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497993], 
> length 65160
> 11:26:28.942547 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 7468720:7533880, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497993], 
> length 65160
> 11:26:28.942689 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6640464, win 22964, options [nop,nop,TS val 497994 ecr 497412], length 0
> 11:26:28.942701 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6720104, win 23145, options [nop,nop,TS val 497994 ecr 497412], length 0
> 11:26:28.942726 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6743272, win 23833, options [nop,nop,TS val 497994 ecr 497412], length 0
> 11:26:28.942864 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 7533880:7599040, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
> length 65160
> 11:26:28.942886 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6811328, win 24159, options [nop,nop,TS val 497994 ecr 497412], length 0
> 11:26:28.942918 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 7599040:7664200, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
> length 65160
> 11:26:28.942971 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 7664200:7729360, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
> length 65160
> 11:26:28.943019 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 7729360:7794520, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
> length 65160
> 11:26:28.943041 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6882280, win 24268, options [nop,nop,TS val 497994 ecr 497412], length 0
> 11:26:28.943187 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 7794520:7859680, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
> length 65160
> 11:26:28.943404 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6928616, win 24576, options [nop,nop,TS val 497994 ecr 497412], length 0
> 11:26:28.943415 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6935856, win 24576, options [nop,nop,TS val 497994 ecr 497412], length 0
> 11:26:28.943420 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6938752, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943427 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6941648, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943433 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6944544, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943438 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6947440, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943444 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6950336, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943452 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6953232, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943458 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6956128, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943463 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6959024, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943468 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6961920, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943473 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6964816, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943479 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6969160, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943487 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6972056, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943493 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6974952, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943500 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6977848, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943505 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6980744, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943511 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6983640, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943519 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6986536, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943525 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6989432, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943533 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6992328, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943542 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6995224, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943546 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 6998120, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943550 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 7001016, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943554 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 7003912, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943558 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 7006808, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943562 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 7011152, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943566 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 7014048, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943570 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 7859680:7924840, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
> length 65160
> 11:26:28.943607 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 7077760, win 24079, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943673 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 7924840:7990000, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
> length 65160
> 11:26:28.943711 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 7096584, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943731 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 7099480, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943750 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 7102376, win 24576, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943777 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 7990000:8055160, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
> length 65160
> 11:26:28.943822 IP 196.5.5.81.40337 > 196.5.5.82.50445: Flags [.], ack 
> 7177672, win 23988, options [nop,nop,TS val 497994 ecr 497413], length 0
> 11:26:28.943833 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 8055160:8115976, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
> length 60816
> 11:26:28.943954 IP 196.5.5.82.50445 > 196.5.5.81.40337: Flags [.], seq 
> 8115976:8120320, ack 1, win 115, options [nop,nop,TS val 497417 ecr 497994], 
> length 4344
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in the body of 
> a message to address@hidden More majordomo info at  
> http://vger.kernel.org/majordomo-info.html
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to address@hidden
> More majordomo info at  http://vger.kernel.org/majordomo-info.html