[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Conda environments and reproducibility
From: |
Simon Tournier |
Subject: |
Re: Conda environments and reproducibility |
Date: |
Tue, 29 Nov 2022 15:25:07 +0100 |
Hi Thibault,
On Tue, 29 Nov 2022 at 10:41, Thibault Lestang <t.lestang@imperial.ac.uk> wrote:
> I think the tweet above is about reproducing an enviroment after
> effectively freezing constitutive packages and their dependenies as you
> describe. They probably used something like
>
> conda env export
>
> Which outputs something similar to (trimmed)
>
> name: justnumpy
> channels:
> - defaults
> dependencies:
[...]
> - ncurses=6.3=h5eee18b_3
> - numpy=1.23.4=py310hd5efca6_0
> - numpy-base=1.23.4=py310h8e6c178_0
> - ...
Do you list all the dependencies? Other said, dependencies of
dependencies? Is it only run-time dependencies?
Konrad pointed, (it = Conda)
it claims that it cannot find a
combination of package known to work together and available in the
archive.
and from my understanding, I think it is because the solver (SAT or
else). Well, for instance,
Theorem 1 Checking whether a single package P can be installed, given a
repository R, is NP-complete.
https://www.mancoosi.org/edos/algorithmic/#toc15
Here (conda env export), you generated the Conda requirements using the
repository in the state R. Then, later the repository becomes R’
(somehow it increases the number of combinations) and it does not matter
if the constraints are foo <= 1.23 or are foo=1.2.3 or are
foo=1.2.3=abcd456.
Maybe I am wrong, from my understanding, Conda builds the graph of
dependencies by resolving a combinatorial problem. When you run,
conda env create -f environment.yml
then Conda relies on a “dependency” solver documented here [1]. And,
IMHO, it is where it fails. Well, if instead of ’conda env export’ you
run,
conda list --explicit > spec-file.txt
then later and elsewhere,
conda create --name myenv --file spec-file.txt
it should bypass the solver. But the documentation [1] reads,
Since the solver is not involved, the dependencies of the
explicit package(s) are not processed at all. This can leave the
environment in an inconsistent state, which can be fixed by
running conda update --all, for example.
Done. :-) Conda environments are hard, if not impossible, to reproduce
when time is flying. It is by design, IMHO.
1: <https://conda.io/projects/conda/en/latest/dev-guide/deep-dives/solvers.html>
2:
<https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html>
Cheers,
simon