Re: Suggestion about error control

[Scott Dwyer]

To be honest I don???t think I ever used any T10 documentation for the 
SCSI passthrough. It is needed for ATA passthrough, but there is plenty 
of other documentation and open source code for the SCSI passthrough, 
and I know for sure everything I found was free. And from what I can 
tell, the SCSI passthrough is still processed by the kernel, and the 
kernel deals with the inconsistencies of devices, so your concerns about 
the ???zillion existing exceptions??? is still well handled by the kernel.

You only need five SCSI commands:

1) INQUIRY

2) READ CAPACITY (10)

3) READ CAPACITY (16)

4) READ (10)

5) READ (16)

Originally I was using the host_status as one way to tell if a drive was 
offline, but some devices cause this status to be bad for no reason. So 
after every read error perform an inquiry, if it fails then the device 
is no longer responding. Also perform a read capacity command and verify 
the capacity is still reported as the same size, if not then the drive 
is no longer responding properly. It is really that simple, once you get 
past the somewhat complicated part of actually performing and processing 
the SCSI passthrough. Other than the host_status issue, the only other 
issue I have seen is that normally if a device is large enough to 
require READ CAPACITY (16) it is supposed to report a block capacity of 
0xffffffff with the READ CAPACITY (10) command, so you would know to use 
size 16 commands. I don???t remember exactly why or what the conditions 
were, but I found it better to try a READ CAPACITY (16) command first, 
and if it fails for invalid command then stick to size 10 commands.

One other thing that must be followed is there is a buffer limit for 
every connected device when using passthrough mode. The limit is stored 
at /sys/block/DEVICE/queue/max_sectors_kb, where "DEVICE" is the device 
you are reading (example "/sys/block/sda/queue/max_sectors_kb"). The 
number stored here is referenced in KB, and the default for a hard drive 
is usually 512 (meaning 512KB). This number is usually smaller for a USB 
connected drive (120KB). This size limit must not be exceeded when 
reading, or bad things will happen.

You may find those issues to be a reason to say something like ???See, 
there are things that are inconsistent and that is not safe???. But I can 
say that following those basic rules has been rock solid for me with the 
SCSI passthrough. As for the ???zillion existing exceptions???, I have 
stepped into the realm of direct packet communication with USB devices, 
and at that level it does get very messy. It makes one aware of how much 
the kernel does deal with the inconsistencies of the devices so that we 
don???t see the chaos.

Regards,
Scott


On 6/3/2020 5:18 PM, Antonio Diaz Diaz wrote:
> Scott Dwyer wrote:
> > No, you have spent much time on an excellent program, the only one of
> > its kind in the open source world, and I bet with little financial 
> > return.
>
> Thanks. You are right about the "little financial return". I have 
> received about 20 euros in donations in the last three months. (6.67 
> eur/month).
>
> > My intention was to reply to the suggestion of error control that
> > ddrescue doesn't do like other programs. You must go deeper to
> > accomplish this, at a minimum SCSI passthrough. I do it in Linux, and
> > the other program can also do it in Windows I believe. Both are specific
> > and non-portable, due to the nature of what needs to be done at a lower
> > level. It is obviously more complicated, but when done correctly it is
> > no more dangerous than what the kernel does.
>
> How can one be sure that it is done correctly given the zillion 
> existing exceptions? You know. Some drive does not implement some SCSI 
> command. Some other implements it in a funny way. Some other has a bug 
> in the implementation... I mean, the kernel already does it badly 
> enough (specially for USB drives).
>
> See for example this note from http://sg.danny.cz/sg/
> "The term SCSI has several meaning depending on the context. This 
> leads to confusion. One practical way of defining it today is 
> everything that the T10 INCITS committee controls, see www.t10.org . 
> Probably the most succinct overview is this standards architecture 
> page . For practical purposes a "SCSI device" in Linux is any device 
> that uses the Linux SCSI subsystem and this often includes SATA disks."
>
> Moreover, SCSI standards are not freely accesible[1]. If I can't find 
> a free copy, I'll need that someone donates one for the development of 
> ddrescue.
>
> [1] http://www.t10.org/t10_access.htm
>
> > And FYI the kernel does NOT know best when it comes to a failing drive,
> > it will thrash it more than needed in Linux, and Windows is even worse.
>
> I believe you. But at least if linux gets any bug related to a failing 
> drive, say returning wrong data for good sectors near a bad sector, I 
> expect it to be discovered faster than if I make the same mistake in 
> (the much less used) ddrescue, for example.
>
> > Then maybe someone can come up with the SCSI passthrough code for
> > ddrescue (hint to programmers out there that want to, I have produced
> > open source Linux patches for this in the past that would be a good
> > starting point, look into the old ddrutility stuff).
>
> Thank you for the patches. I keep them and I plan to use them at least 
> to compare them with my own code as a way to find possible errors in 
> my code.
>
> IIRC, the main reason why I have never used your SCSI passthrough 
> patch is that its main feature is increasing the read performance, 
> which I think should be done by the kernel when --idirect is used. I 
> do not consider that reading data through the SCSI passthrough 
> interface is safe enough for ddrescue. The readme file for your patch 
> tends to confirm this[2].
>
> [2] 
> http://sourceforge.net/projects/ddrutility/files/ddrescue%20patches/passthrough%20patch/
>
> You have done a good work, but I plan to keep the risks low and limit 
> ddrescue's use of the SCSI passthrough interface to the improvement of 
> the detection of error conditions in the input device.
>
> > > IMO every piece of software should either publish the full source code
> > > (so that users can decide if they trust it) or offer an unlimited
> > > warranty in case of misbehavior of the code.
> >
> > If this were the case, then all software would be open source or open to
> > incredible liability. Without the hope of financial gain (or having the
> > fear of great loss), there would be much less effort, and many good
> > programs would not exist.
>
> It does not need to be "open source" in the sense of "free software", 
> only in the sense of "the users may verify it, even if they aren't 
> allowed to redistribute it". This surely would increase the safety of 
> the software by removing lots of crappy non-free software from the 
> market.
>
> > Maybe I should remove myself from the list so I don't see the emails, 
> > and
> > therefore not tempted to reply. I might just do that...
>
> Please, don't. Your contributions are valuable and appreciated. It is 
> just that writing about non-free software (specially to promote it) is 
> off-topic in GNU lists.
>
> Best regards,
> Antonio.