Re: Release 4.4.0 and statistics package

[Rik]

On 12/28/2017 03:02 AM, Olaf Till wrote:
> On Wed, Dec 27, 2017 at 03:46:41PM -0800, Rik wrote: >> On 12/27/2017 02:38 PM, John W. Eaton wrote: >>> On 12/27/2017 11:35 AM, Rik wrote: >>> >>>> One of the items in the Release Checklist is to incorporate GSOC project >>>> code and patches which look good from the patch tracker. It is best to >>>> do that early as that will de-stabilize things for a while and we want >>>> to get some testing on the new code before sending it out in to the world. >>> >>> I will try to make a 4.2.2 release ASAP. >>> >>> I'm afraid that if we incorporate GSoC code and the spreadsheet functions >>> from the I/O package now, then we will not have a new release for at >>> least another 6 months, probably even longer. >>> >>> Rather than falling into the trap of trying to include and fix everything >>> before the next release I would prefer to move in the direction of more >>> frequent releases with major new features only being added near the >>> beginning of a release cycle. So I propose that we not add any large new >>> chunks of code at this time but focus on fixing what we can for a 4.4.0 >>> release by the end of February. >>> >>> jwe >> >> Arno, >> >> You are listed as the package maintainer for the statistics package. I'm >> not sure if you are subscribed to the Octave Maintainer's list, but a >> significant new 4.4.0 release is expected early next year (see quoted >> e-mail above). As part of that release, a lot of the statistics functions >> are set to be removed from core Octave and transitioned to the statistics >> package. How can this be coordinated? >> >> The base problem being addressed is that many of the functions, for example >> kolmogorov_smirnov_test_2.m, are not of sufficient general utility to be >> included in the core, but are still useful to practitioners of statistics. >> >> Cheers, >> Rik > > Is there a list of these functions, or can you make one?
The scripts statistics directory contains 4 sub-directories.  Everything in tests/, models/, distributions/ can go to the statistics package.

Within the base/ directory I would kick out:

cloglog.m
crosstab.m
ppplot.m
qqplot.m

These functions with base/ directory should be moved somewhere else within core

ols.m
gls.m
lscov.m  [ maybe placed in statistics package ]

After that, see what needs to be added back in.  I think quantile may rely on the empirical distribution which would require keeping that around in core.

> > > ML statistics toolbox has 'kstest' and 'kstest2', the latter probably > corresponding to the above 'kolmogorov_smirnov_test_2'. Is 'kstest' > of more general utility than 'kstest2'?
Octave also has a kolmogorov_smirnov_test.m which probably corresponds to the first function kstest.m

> How does the fact that there > is an ML-pendant in the statistics toolbox influence your decision to > move the code from Octave to Octave Forge?
No influence, not even sure what the ML-pendant thing is.

> If it is moved, should we > rename the functions to their ML name, if appropriate?
Yes, this makes sense to me.  You could always do as we do in core and keep a deprecated version of the function around for 2 release cycles.  For example, rename kolmogov_smirnov_test_2 to kstest2 and then create a new file kolmogov_smirnov_test_2 which just issues a warning and then calls kstest2

--- Code ---
function [pval, ks, d] = kolmogorov_smirnov_test_2 (x, y, alt)

  persistent warned = false;
  if (! warned)
    warned = true;
    warning ("Octave:deprecated-function",
             "kolmogorov_smirnov_test_2 is obsolete and will be removed from a future version of Octave, please use kstest2 instead");
  endif

 [pval, ks, d] = kstest2 (x, y, alt)

endfunction
--- End Code ---

--Rik