Re: [Fab-user] status of "occasional droped lines of output" bug 32

[Steve Steiner (listsin)]

On Aug 7, 2009, at 6:46 PM, Jeff Forcier wrote:

> On Fri, Aug 7, 2009 at 6:36 PM, Steve Steiner
> (listsin)<address@hidden> wrote:
>
> > We could waste a  _lot_ of time trying to create a mock
> > object at the right level and we might find ourselves doing it more  
> > than
> > once.  The advantage of a live server is that it actually exercises  
> > the
> > whole chain, not just up to the mocked object.
>
> I've definitely considered this; I'll probably do a "best effort" at
> getting network level testing without having a live setup, but if that
> fails I probably will just use a restricted user on a specific system,
> or (possibly better) have the tests ssh to "localhost" and use
> non-destructive commands when testing.

sshing to localhost has a whole host (pardon the pun) of other issues  
regarding 'whoami', where's "home", etc. on Windows vs *nix that I've  
never even bothered to look at since I don't use Windows.

Cool idea, though.  Finding "the fabric installation dir where I'm to  
run my dir/ls command" is probably not going to be that much fun ;-0.

> The benefit of this latter approach is that anyone could run the tests
> -- I'd be sad if I had to have a test suite that only works when I run
> it, or can't be run offline :) Downside is that it might be harder to
> know what the output of e.g. "ls -l $HOME" would be, i.e. every system
> is different. Offhand I'm thinking of having the test suite find out
> where on the filesystem it is, and to do stuff like "ls -l
> /path/to/where/fabric/seems/to/be" which might solve that problem.

I agree, it would be great for everyone to be able to run the whole  
test suite.

Then we get the best real-world Python Version/OS matrix.

I think running against a live server gives the best chance  
(approaching 100%) of everyone being able to run the tests whereas  
mock objects leaves us always fighting against OS/Python Version edge  
cases.

Having something fail on myBestLinuxDistribution, only on Python 2.5.1  
where it's running against a local ssh session  gives an infinite  
number of points of failure.  Is it the ssh server on the host, the  
way it's authenticated, some oddball local encoding etc.?

Having it fail on myBestLinuxDistribution on Python 2.5.1 against a  
known server, gives us the ability to turn on some debug information  
in Fabric to see where it's falling down without worrying about the  
ssh server's capabilities, whether it can find things, etc.

IOW, a test against a known host on the Internet lets us test the  
whole stack, against a known quantity, and gives the fewest variables  
in all tests.

Writing mock objects for easily created real-life objects has never,  
in my experience, been worth the effort.  This is especially true now  
when real-life resources are so cheap and easy to set up.

Just my two cents; I've been down this road a few times before and  
always found that live-server testing was cheapest in man-hours and  
most accurate in tracking down the actual causes of failures.

Thanks,

S