Re: Python 2 and test/vm/netbsd

[Eduardo Habkost]

On Thu, Oct 17, 2019 at 07:05:41PM -0300, Eduardo Habkost wrote:
> On Wed, Oct 16, 2019 at 07:41:24PM -0300, Eduardo Habkost wrote:
> > On Wed, Oct 16, 2019 at 08:11:57AM +0200, Thomas Huth wrote:
> > > On 16/10/2019 05.00, Eduardo Habkost wrote:
> > > > On Tue, Sep 17, 2019 at 08:31:40PM -0300, Eduardo Habkost wrote:
> > > >> On Mon, Jul 01, 2019 at 07:25:27PM -0300, Eduardo Habkost wrote:
> > > >>> On Mon, Jun 10, 2019 at 01:58:50PM +0100, Peter Maydell wrote:
> > > > [...]
> > > >>>> The configure check also spits out deprecation warnings for
> > > >>>> the NetBSD/FreeBSD/OpenBSD tests/vm configurations. It would be nice
> > > >>>> to get those updated.
> > > >>>
> > > >>> CCing the test/vm maintainers.
> > > >>>
> > > >>> Fam, Alex, are you able to fix this and create new BSD VM images
> > > >>> with Python 3 available?  I thought the VM image configurations
> > > >>> were stored in the source tree, but they are downloaded from
> > > >>> download.patchew.org.
> > > >>
> > > >> Fam, Alex, can you help us on this?  Python 2 won't be supported
> > > >> anymore, so we need the VM images to be updated.
> > > > 
> > > > Anyone?
> > > > 
> > > > I'm about to submit patches to remove Python 2 support, and this
> > > > will break tests/vm/netbsd.
> > > > 
> > > > I'm powerless to fix this issue, because the netbsd image is
> > > > hosted at download.patchew.org.
> > > 
> > > Gerd had a patch to convert the netbsd VM script to ad hoc image
> > > creation, too:
> > > 
> > > https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg04459.html
> > > 
> > > But there was a regression with the serial port between QEMU v3.0 and
> > > v4.x, so it was not included:
> > > 
> > > https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg06784.html
> > 
> > The URL above has this error:
> > 
> >   con recv:  x: Exitqqqqqqqqqqqqqqqqqqqqqqqqqj
> >   con recv: To be able to use the network, we need answers to the
> >   following:Network media type
> >   con send: <enter>
> >   con recv: : qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk Perform autoconfiguration?
> >      >a: Yes b: Noqqqqqqqqqqqqqqqqq
> >   console: *** read timeout ***
> >   console: waiting for: 'a: Yes'
> >   console: line buffer:
> >   
> >   con recv: qqqqqqqqqqqqqqj
> > 
> > I believe that problem was solved in v4, because v4 was reading
> > the serial output 1 byte at a time.
> > 
> > The issue that caused the netbsd patch to be dropped was:
> > https://lore.kernel.org/qemu-devel/address@hidden/
> > 
> > Possibly this is the same issue we saw at:
> > https://lore.kernel.org/qemu-devel/address@hidden/
> > 
> > The test script must either close the console socket, or keep
> > reading from it.  Otherwise, the QEMU VCPU threads might get
> > stuck waiting for the chardev to be writeable.
> 
> It doesn't seem to be the same issue.  Even if the console socket is closed,
> I'm seeing results similar to the ones reported by Peter (the "pkgin -y
> install" step is unreasonably slow).
> 
> Running with V=1, I see packages being downloaded at reasonable speeds, but
> there's a huge interval (of various minutes) between each package download.

I've found the cause for the slowness I'm seeing: for each file
being downloaded, the guest spents at least 75 seconds trying to
connect to the IPv6 address of ftp.NetBSD.org, before trying
IPv4.  I don't know if this is a NetBSD bug, or a slirp bug.

Output of `strace -e trace=network` below:

1571352260.348566 recvfrom(30, 
"~[\201\200\0\1\0\1\0\0\0\0\3ftp\6NetBSD\3org\0\0\1\0\1"..., 1500, 0, 
{sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.5.30.160")}, 
[128->16]) = 48 <0.000016>
1571352260.349030 socket(AF_INET6, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_IP) = 31 
<0.000041>
1571352260.349142 setsockopt(31, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 
<0.000009>
1571352260.349179 setsockopt(31, SOL_SOCKET, SO_OOBINLINE, [1], 4) = 0 
<0.000007>
1571352260.349207 setsockopt(31, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.000008>
1571352260.349239 connect(31, {sa_family=AF_INET6, sin6_port=htons(80), 
sin6_flowinfo=htonl(67108864), inet_pton(AF_INET6, "2001:470:a085:999::21", 
&sin6_addr), sin6_scope_id=377348480}, 28) = -1 ENETUNREACH (Network is 
unreachable) <0.000
021>
1571352266.350112 socket(AF_INET6, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_IP) = 31 
<0.000131>
1571352266.350603 setsockopt(31, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 
<0.000183>
1571352266.350946 setsockopt(31, SOL_SOCKET, SO_OOBINLINE, [1], 4) = 0 
<0.000047>
1571352266.351109 setsockopt(31, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.000043>
1571352266.351260 connect(31, {sa_family=AF_INET6, sin6_port=htons(80), 
sin6_flowinfo=htonl(67108864), inet_pton(AF_INET6, "2001:470:a085:999::21", 
&sin6_addr), sin6_scope_id=377348480}, 28) = -1 ENETUNREACH (Network is 
unreachable) <0.000070>
1571352278.357962 socket(AF_INET6, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_IP) = 29 
<0.000166>
1571352278.358524 setsockopt(29, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 
<0.000046>
1571352278.358757 setsockopt(29, SOL_SOCKET, SO_OOBINLINE, [1], 4) = 0 
<0.000046>
1571352278.358950 setsockopt(29, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.000039>
1571352278.359103 connect(29, {sa_family=AF_INET6, sin6_port=htons(80), 
sin6_flowinfo=htonl(67108864), inet_pton(AF_INET6, "2001:470:a085:999::21", 
&sin6_addr), sin6_scope_id=377348480}, 28) = -1 ENETUNREACH (Network is 
unreachable) <0.000065>
1571352302.373056 socket(AF_INET6, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_IP) = 29 
<0.000323>
1571352302.373909 setsockopt(29, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 
<0.000162>
1571352302.374331 setsockopt(29, SOL_SOCKET, SO_OOBINLINE, [1], 4) = 0 
<0.000159>
1571352302.374626 setsockopt(29, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.000050>
1571352302.374857 connect(29, {sa_family=AF_INET6, sin6_port=htons(80), 
sin6_flowinfo=htonl(67108864), inet_pton(AF_INET6, "2001:470:a085:999::21", 
&sin6_addr), sin6_scope_id=377348480}, 28) = -1 ENETUNREACH (Network is 
unreachable) <0.000070>
1571352335.394568 socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_IP) = 29 
<0.000091>
1571352335.394796 setsockopt(29, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 
<0.000011>
1571352335.394848 setsockopt(29, SOL_SOCKET, SO_OOBINLINE, [1], 4) = 0 
<0.000009>
1571352335.394883 setsockopt(29, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.000008>
1571352335.394913 connect(29, {sa_family=AF_INET, sin_port=htons(80), 
sin_addr=inet_addr("199.233.217.201")}, 16) = -1 EINPROGRESS (Operation now in 
progress) <0.000055>
1571352335.587395 sendto(29, "", 0, 0, NULL, 0) = 0 <0.000220>
1571352335.589650 sendto(29, "GET /pub/pkgsrc/packages/NetBSD/"..., 81, 0, 
NULL, 0) = 81 <0.000088>

-- 
Eduardo