Re: pkg.m status and question

[Philip Nienhuis]

Mike Miller wrote:
> On Mon, Jan 06, 2020 at 04:26:21 -0600, PhilipNienhuis wrote:
> > My motive for asking is that this setup of package info makes for needlessly
> > inefficient and complicated code. For each tiny action the relevant structs
> > have to be unwrapped first. IMO dealing with a struct array rather than cell
> > array of structs would make the code much more transparent and much more
> > efficient as well.
>
> I agree with this motivation, having seen this before too. I can't speak
> to why it was originally done as a cell array of scalar structs.
>
> I do have a speculation. The current design reads each pkg DESCRIPTION
> file into a scalar struct, which is then added to a cell array of all
> pkgs. A problem arises if you have pkg A which has a "Url" field in its
> DESCRIPTION, and pkg B which has a "SystemRequirements" field in its
> DESCRIPTION. How do you easily combine two structs with unlike fields
> into a single struct array with a superset of all fields?
>
> By way of example,
>
>     a.Name = "first";
>     a.Description = "A package";
>     a.Url = "https://www.example.org/";;
>     b.Name = "second";
>     b.Description = "Another package";
>     b.SystemRequirements = "fftw3";
>     all = [a, b];
>
> doesn't work. But the current cell array implementation does.

Thanks Mike, that's one thing I figured (and mentioned as first reason). 
But I'm not so impressed. At my work we have complete input data sets 
for our groundwater models in nested Matlab structs (some up to 100's of 
MBytes on disk in compressed v7.3 format, in perhaps 5-10 nesting 
levels); such heterogeneity as you mention is vital and a feature we 
have quite a bit of experience with.

Probably too much detail now, but here goes:
We need a basic struct with all required fields that are of the same 
type for all packages.
All fields that are heterogeneous (not required for each package and/or 
not used by the package manager) go in substructs or cell arrays in a 
dedicated field of the main struct. That way we move heterogeneity from 
the outside wrapper into a much less prominent location inside, out of 
the way of regular package manager operations.
Upgrading an old package database to a new format can be done with a 
dedicated function; such a function can't be too complicated and as it 
runs just twice (global and local) doesn't need much optimization (read: 
developer time). Maybe that function sniffs what the database looks like 
and acts if it's the old cell type. Saving and reading the package 
database is otherwise unaffected.
When installing a new package a check could be due for new fields in the 
"Description" substruct/cell array. Maybe it isn't even necessary.

But read on, another step is needed as well:

Carné rightly mentions backward compatibility. I surmise that is because 
the global octave_packages file by default lives in the share/octave/ 
directory, and as a consequence it is shared between octave releases 
(and theoretically between OS-es if it served to e.g., remote clients). 
(Even the local octave-packages file is/can be shared between Octave 
releases, but I ignore that for now.)
IMO that is a wrong place to begin with as octave_packages has "hard" 
references to arch-dependent subdirs as well; each package in the 
package database has just one reference to an arch-dependent subdir. 
That implies that OF packages with binary modules cannot be shared 
reliably between Octave releases anyway (I've been bitten very often by 
that). Nor can packages be shared between different OS-es. It can be 
argued that even the arch-independent directory cannot be shared between 
Octave releases as .m-files can invoke functions or constructs that are 
new or dropped or renamed in different releases.
All in all, IMO a somewhat better place for the package database would 
be in share/octave/<release>/ and the packages themselves as well.
Or am I wrong? please enlighten me then.

That said, I wonder how much practical situations exist where OF 
packages are shared between Octave releases on networks, or even clients 
with different OS-es.

My impression is that here's a quite a bit of legacy in the way, 
hindering simplification of package management.

Philip