Hello community,
I am working on a gnuradio application that synchronously retrieves data from a usrp_source, does some DSP on each of the streams and selects the signal to send on a usrp_sink based on some metric computation. The sources and sinks represent 3 USRP X310 devices, each with 2 UBX-160 daughterboards, and are synchronized using an Octoclock-G. Thus, creating a 6x1 MISO system. The USRP synchronization is done using the Python API, as well as the flowgraph. The DSP blocks are coded in C++. The flowgraph would be something like:
self.connect((usrp_source, 0), my_dsp_0, (my_switch, 0))
self.connect((usrp_source, 1), my_dsp_1, (my_switch, 1))
self.connect((usrp_source, 2), my_dsp_2, (my_switch, 2))
self.connect((usrp_source, 3), my_dsp_3, (my_switch, 3))
self.connect(my_switch, another_dsp_0, (usrp_sink, 0))
The application runs for a while until UHD outputs a message saying that the poke32 has timed out on one of the USRP. After that, a misalignment error on the receiver streams is shown and the green lights on the USRP go off.
We are under the impression that the DSP blocks working independently may be requesting samples from the source at different times, causing the misalignment. Could this be the case? Is there an issue with my network connection?
I have tried to get rid of the misalignment error by increasing the threshold in the file
{uhd_prefix}/host/lib/transport/super_recv_packet_handler.hpp, through the function set_alignment_failure_threshold. That got rid of the misalignment error, but the failure coming from the poke function is still there.
Interestingly, the exact same behavior arises when I just connect file_sinks to the outputs from the usrp_source and a simple analog signal to the usrp_sink. Below the flowgraph:
self.connect((usrp_source, 0), file_sink_0)
self.connect((usrp_source, 1), file_sink_1)
self.connect((usrp_source, 2), file_sink_2)
self.connect((usrp_source, 3), file_sink_3)
self.connect(analog_sig_c, (usrp_sink, 0))
I'd appreciate any help on this since I'm running out of ideas here. Please, find below more details.
The error:
[ERROR] [X300] 192.168.100.2: x300 fw communication failure #1
EnvironmentError: IOError: x300 fw poke32 - reply timed out [ERROR] [X300] 192.168.100.2: x300 fw communication failure #1
EnvironmentError: IOError: x300 fw poke32 - reply timed out [ERROR] [X300] 192.168.100.2: x300 fw communication failure #3
EnvironmentError: IOError: x300 fw poke32 - reply timed out [ERROR] [UHD] An unexpected exception was caught in a task loop.The task loop will now exit, things may not work.EnvironmentError: IOError:
192.168.100.2: x300 fw communication failure #3
[ERROR] [STREAMER] The receive packet handler failed to time-align packets.
1002 received packets were processed by the handler.
However, a timestamp match could not be determined.
Here are some details of my network configuration:
- UHD built from source, tag v3.14.1.1.
- Gnuradio built from source, tag 3.7.13.5.
- X310 USRP, each equipped with two UBX-160 daughterboards.
- The communication is over Ethernet using the 1Gb port on the X310.
- Each USRP is connected to a dedicated NIC.
- The MTU is set to 1500 for the all interfaces.
- The USRPs are synchronized using an Octoclock-G and the parameters tuned in the Python script. The procedure is similar to the one followed in benchmark_rate (uhd), for the usrp_source and usrp_sink.
- The NIC ring buffers are set to the maximum for both TX and RX, to 4096, using ethtool.
- The OS ring buffers are set as suggested bu Ettus, using the following commands:
>> sudo sysctl -w net.core.rmem_max=33554432
>> sudo sysctl -w net.core.wmem_max=33554432
>> sudo sysctl -w net.core.rmem_default=33554432
>> sudo sysctl -w net.core.wmem_default=33554432