Re: [Pan-users] ssl problems, not working on ports 443 or 563

[Duncan]

Joe Zeff posted on Mon, 02 Apr 2012 12:03:24 -0700 as excerpted:

> On 04/02/2012 11:46 AM, Duncan wrote:
>> On gentoo I build from sources, but I've not had dependency hell since
>> my Mandrake days, as there's usually at least a masked version of the
>> ebuild that I can unmask somewhere, that has all the deps listed.
> 
> I know that apt-get, the Debian/Ubuntu package manager, will resolve
> deps and bring them in as needed, as I've seen it happen on my sister's
> Ubuntu box.  And, yum, the Fedora equivalent, does the same.  I'm not
> familiar with gentoo, so I have to ask: doesn't your package manager
> take care of such things, or are you talking about building packages
> that are not from the gentoo repos?

Umm... somewhere along the line, one or both of us got off track.  I 
/thought/ what I wrote was perfectly clear, but obviously not, at least 
not to you, or your question either wouldn't be needed or would itself be 
clear to me as well.

So I'll try backing up a bit and do a retrace.  Let's see if that helps.

1) Bob is trying to build pan from current git sources (so presumably he 
has a pan package and its deps are installed, but any new deps in the git 
version are not).  It builds fine without SSL support (thus the original 
question, since he didn't know the ssl support wasn't built), but won't 
build with it as his gnutls is too old (that's a new dep, since the ssl 
support is new, no released version with it, yet).

2) So he needs a new gnutls to build the new pan with ssl support.  
Again, the package he has would take care of deps as of the old gnutls, 
but he needs a newer one for pan, with newer deps.

3) As it happens, one of those newer gnutls deps is nettle, and in gentoo 
(what I use here) as well, the "stable" gnutls was too old for pan.  They 
hadn't stabilized the newer gnutls due to API changes related to its new 
nettle dependencies.

I suspect that we're all clear to that point.

4) This is where I think the confusion may have started so I'll include a 
bit more detail this time around.  I don't know what distro bob is on but 
I'm presuming it's not the Gentoo ~arch (aka testing) that I run.  
Whatever it is, tho, they seem to be in about the same position Gentoo 
itself was a couple months ago when I went thru the gnutls upgrade in 
ordered to satisfy pan's new deps, myself.  Originally I left out a bit 
of gentoo detail as I doubted it mattered to him, but eh... throw it in 
this time and see if it clears things up.

5) On gentoo the in-tree available versions of gnutls and their stability 
level has changed since March 16:

2.10.x (2.10.5 currently, might have been 2.10.4 or whatever then):
No nettle required, stable for quite some time.  (This is what Dave S 
says is in Ubuntu latest, also.)

2.12.x (2.12.18 added in gentoo on March 16): Requires nettle AND p11-
kit.  In theory, it's still 2.x so should maintain API, but apparently 
there were some issues, as gentoo, and now it seems bob's distro... and 
dave shocket's (which he said is Ubuntu latest)... stuck at 2.10.x stable 
for quite some time.

Since March 16's gentoo addition of 2.12.18, various archs appear to be 
stabilizing on it, with about a third of them (including the big ones, 
amd64 and x86) now stable on 2.12.18, while the rest are still stable on 
2.10.5.

3.0.x (3.0.16 also gentoo-added on March 16): 3.x major version.  Gentoo 
has this available in-tree, but it's not stable yet, only ~arch 
(testing).  Before March 16's 3.0.16, 3.x was in-gentoo-tree but I 
believe hard-masked, not visible for installation even for ~arch users, 
unless they queried for it specifically, and deliberately unmasked it.

>From what I can see both from gentoo, and now from this thread I see the 
same in other distros, the 2.12 version is where a lot of the BIG changes 
occurred, including nettle and p11-kit dependencies, and really SHOULD 
have been 3.0.  The 2.x series should have remained stable at 2.10.x, 
without the nettle and p11-kit deps, which only should have been added 
for the 3.x series.

Oh, well, upstreams decide, distros (and end users advanced enough to 
care) have to cope with it.

6) Now, it should be mentioned that gentoo is both automated-from-source 
(end-user builds, but using an automated dependency tracking and build-
script system) and rolling distribution (no real distro version upgrades 
after installation, individual package upgrades are introduced and 
unmasked to ~arch, then to stable, over time).  Additionally, unlike a 
lot of distros but as can be seen from #4 above, it's the norm for 
multiple versions of a particular package to be in-tree at the same time, 
with various users on various versions, depending on whether they're 
~arch or stable (or even have unmasked hard-masked versions), their 
upgrade schedule and whether they religiously deep-upgrade (as I do), or 
only pull in dependency upgrades when some leaf package they've installed 
ups its dependency to require it, or the old versions get pruned from the 
tree.

7) On gentoo, ~arch (testing) is generally defined as upstream-stable, 
but gentoo-package testing, tho in practice, there's a bit of variance on 
that per individual package.  So with some exceptions, unless upstream 
defines it as stable, it's not even a candidate for ~arch and later 
stable.  Upstream development versions, betas and rcs are often available 
in-tree, but they normally remain hard-masked, only visible to those who 
deliberately unmask them for their own reasons.  Additionally, in many 
cases especially for major projects such as kde/gnome/xorg/mozilla, 
upstream unstable/development versions are not added to the main tree, 
but can be found in various overlays.  Often, the overlays have live-vcs-
version builds available as well, for the REALLY adventurous.  There's a 
few of them (hard-masked, of course) in the main tree, but not many.


***>>>
8) Dependency-wise, gentoo's ebuilds (the build-scripts for individual 
packages) specify all the usual dependency information, which is then 
automatically handled by the package manager.  That's why I said I'd not 
seen dependency hell since back when I was on Mandrake.  That's I guess 
the short and direct answer to your question, but it's not complete.
***<<<


9) Here, I normally run ~arch, but in addition to the main tree, have the 
kde, x11/xorg and mozilla (for firefox) overlays added, and routinely 
unmask various packages before they're unmasked in-tree.

For example, due to gentoo's build-from source, gcc is a critical system 
package, and it doesn't tend to be unmasked even to ~arch until it can 
build nearly everything at that stability level (so ~arch for me).  gcc 
4.6 still remains hard-masked, for instance, due to the fact that gentoo's 
still stable on grub-legacy (0.97 with a whole slew of patches), with 
grub2 packages in-tree but hard-masked, and there's apparently some 
blocking boot-bugs when grub-legacy is built with gcc-4.6.

But I unmasked gcc 4.6.2 a few months ago here and did a full rebuild.  
Everything I'm running (I've unmasked grub2 here) builds and appears to 
run fine with gcc 4.6.2.

Similarly, I unmasked and ran the kde 4.8 prereleases, betas 1 and 2, and 
rcs 1 and 2. (FWIW, kde 4.8.0 was released at the end of January, with 
4.8.1 a month later.  4.8.2 should be out this week.  So the pre-releases 
were late last fall and early winter.)  They were never in-tree at all as 
they're pre-releases and weren't stable candidates, but I'm running the 
kde overlay, which had them available.

***>>> (to end)
10) Because I'm always running at least ~arch, which by definition is 
gentoo testing level ebuilds for upstream stable, and am often 
deliberately unmasking the even less mature hard-masked versions, not 
even ~arch, while gentoo's ebuilds DO specify dependency information, at 
the immaturity level I run them, that dependency information isn't always 
100% current.

Thus, a more complete answer to your dependency question is yes, gentoo's 
package management deals with it in general, but at the immature package 
testing level I routinely run, those dependencies aren't 100% guaranteed 
to be current and upto date.  So I do very occasionally see too-low-a-
version errors when I try to build something.

However, because I DO religiously run deep upgrades (I don't ever NOT do 
deep upgrades and my own helper scripts all assume deep), in practice, I 
very seldom see such errors, because generally I've long since done that 
upgrade anyway.  Instead, I much more often see errors because I'm trying 
to run versions too new as I've unmasked something that's hard-masked 
precisely for that reason, an incompatible API change or the like, and 
either the deps are specified in the ebuild and I have to try to figure 
out how to get everything playing on either the old or new version, or 
they're not yet high-side version limited or patches applied to build 
with the new versions (I unmasked the still-masked version, not yet 
expected by the currently specified dependencies, after all), and I have 
to go looking for patches or unmasking sometimes live-vcs versions of the 
depending packages, in ordered to get everything working with the new 
versions.

But that's not dependency hell, since (1) it's almost always due to a 
deliberate unmask on my part, and (2) it's almost always only a single 
level of dependency changes I need to figure out.  Beyond that, generally 
I'm already updated, and/or the existing dependency specs are correct, so 
I don't get into the sort of recursive "but I can't build this because it 
depends on that, and I can't build that because it depends on something 
else, N levels deep" sort of situation that is normally referred to as 
"dependency hell".

11) Another contrast between gentoo and most distributions that helps 
with this sort of thing, is that since gentoo assumes build-from-source 
at the user end, it really doesn't make sense at all to try to split 
packages into run-time and development sub-packages.  IOW, there's no 
libXXXX-dev packages on gentoo, there's only the libXXXX package, 
including both the binary library and the related headers, *.pc files, 
etc, that are normally only required if you're building a package 
depending on that library.  Since in gentoo, EVERY package is assumed to 
be built from source, (almost, servant-ware binary-only packages are an 
exception) ANY package depending on that library would require the build-
time bits to build and install anyway, and why would someone only install 
the library, if they weren't going to be using it as a dependency for 
some other package.  On a source-based distro, it really doesn't make 
sense to split up packages into runtime and -dev sub-packages, so gentoo 
doesn't do so.

That alone seriously diminishes dependency hell issues, since it roughly 
halves the number of package dependencies required. No more having to 
track down libXXX-dev, even when one *IS* doing manual builds outside the 
package system, because if the library is installed, so are the bits of 
it normally found in the -dev package on binary-based distros. =:^)

12) So much for the general gentoo case, now we're back to the specific 
case at hand, pan from live-git-repo sources. =:^)

As mentioned above, only git-pan requires gnutls at all, and that only 
when the --with-gnutls configure option is enabled in ordered to handle 
secure connections.  FWIW, gentoo does have a live-git pan ebuild 
(pan-9999, in gentoo, -9999 versions are used for live-vcs packages) 
available, but it doesn't have a gentoo developer following things 
closely enough to keep up with deps like this.  As a result, I've 
customized my pan-9999, keeping it in my own personal overlay, where I 
can manage the deps changes myself.  I'd submit the changes back to 
gentoo, with optional deps changes hooked up to gentoo USE flags as 
necessary, and do occasionally do so (usually in relation to a release 
version bump), but generally only test the side of the option I'm 
interested in (in this case, --with-gnutls), not both sides, and 
sometimes don't even know how to properly define the mapping for both 
sides, so I'm afraid I'm not ready to be a proper proxy-maintainer of the 
gentoo pan-9999 ebuild yet -- my fixes only help on the one side, and 
require real gentoo developer help to get them setup for the other.

So of course, when git-pan's deps change, I'm making the changes to my 
ebuild to reflect them, myself.  The package management system obviously 
can't do that by itself, tho as with most distros, there's various 
maintainer tools available to help with the process.

But a lot of the time, I'm not even aware of minimum version requirement 
changes, because as I said, I routinely do deep upgrades myself, and with 
occasional exceptions like this gnutls thing, generally have well above 
minimal versions long since installed.  So usually, the only time I 
actually bump the in-ebuild deps is when Heinrich or someone mentions 
current deps on list, and I go back and compare what's in the ebuild 
against them, or when I sync to changes made in the gentoo ebuild since I 
last checked it, sometimes in preparation for submitting a gentoo pan 
package version-bump request, etc.

Of course right now, with gnutls being such a new dep, when the new pan 
version comes out this summer (assuming a continuation of the annual bump 
pattern, on or about August 1), if I file the version-bump request with 
gentoo, I'll probably have to say I don't know exactly how to specify the 
new deps and how to hook them up to the relevant SSL USE flag (there's no 
old one yet to pattern after).  Meanwhile, I haven't modified my ebuild 
at all for that, but instead, have a bashrc file setup that portage 
(gentoo's main PM and the one I use) will source before reading the 
ebuild, that adds the --with-gnutls option via an available user 
extension syntax.  ( EXTRA_ECONF='--with-gnutls' )  So the ebuild doesn't 
specify gnutls dependencies at all, I use the extension to inject the 
configure script option, and with that the configure script simply picks 
up the gnutls (3.0.16) I've manually unmasked and upgraded to.

13) But of course, most of that, being gentoo specific, was irrelevant to 
bob and dave, as dave's running ubuntu and bob's presumably not running 
gentoo either.  So I had skipped all that, apparently leaving you with 
the impression that gentoo's ebuilds don't handle deps automatically, 
even tho I specifically stated that I hadn't experienced dependency hell 
since my mandrake era, before I switched to gentoo.  I still don't quite 
understand how you got from that specific statement that I had NOT had 
dependency hell on gentoo, that it didn't handle the deps well, but since 
I had skipped the details as irrelevant in that context, I guess it must 
have been due to omitted information.

I trust that my usual information overload approach has corrected that 
problem now?  =:^)

14) Fully back to the thread topic, now.  Based on data from multiple 
distros now, it seems that gnutls 2.10.x has been where many stabilized, 
because 2.12.x and above require nettle and p11-kit.  Meanwhile, pan's 
new secure-connections feature requires the --with-gnutls configure 
option, which requires gnutls 2.12+ as earlier versions apparently didn't 
work so well in Heinrich's testing.

So it is that users trying that feature in git-pan are having to build a 
newer gnutls from sources, since their distros are still stabilized on 
2.10, and end up running into both the nettle and p11-kit recursive 
dependency issues when they do so.  Those in turn may recurse into 
further necessary updates, thus triggering "dependency hell", the seeming 
unending series of manual updates required just to build what was 
originally thought to be a single package update.

Luckily for me, most of the work was already done for me in gentoo and 
because I run quite new versions of everything anyway; I only had a 
couple unmasks to do, plus of course rebuilding a couple other packages 
that depended on gnutls after the upgrade, due to the ABI issues.  But 
I'd say about 95% of the complexity was already handled, and what wasn't, 
was a level of issue that I'm used to dealing with relatively routinely 
anyway, in my normal course of early unmaskings, running kde betas, etc.

People on the more ordinary binary distros, however, will be far more 
likely to have to deal with at least 20% of the issues manually (unless 
they're running redhat 5 or something, the packaging system will still 
deal with the majority of the problem, but it'll still look like 
dependency hell, especially to the uninitiated), where I had to deal with 
only perhaps 5%, and on average, they'll be less accustomed to dealing 
with that level of dependency resolution issue in the first place.  So 
they'll have perhaps 4X the exposed problem, but only be perhaps a third 
as well prepared to deal with it, resulting in an experience that's 
likely 4*3=12 times more difficult for them to deal with!

But I've seen suitably determined people bring even a redhat 5 system up 
to current kde, taking days of recursive dependency builds to even get 
kdelibs to build, but getting it done.  If a suitably determined person 
can do that, then surely, a suitably determined person can do the much 
less difficult as it's far closer to current, upgrade from pan 0.133+ to 
git-pan, and with it from gnutls 2.10 to at minimum 2.12, along with its 
recursive dependencies in turn.  There, there's only the two of them, 
nettle and p11-kit, the others are likely to be upto date, instead of the 
several dozen just to build kdelibs, that the poor guy upgrading that 
redhat system was dealing with!

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman