Re: [GNUnet-developers] Proposal: Make GNUnet Great Again?

[Schanzenbach, Martin]

> On 9. Feb 2019, at 12:18, Christian Grothoff <address@hidden> wrote:
> 
> Signed PGP part
> Hi Martin,
> 
> Let's go over the subsystems in your list specifically:
> 
> - Secushare + social + psyc + psycstore + multicast
>  IMO these should go together.  Nothing but
>  Secushare uses the other 4 yet, and they are all
>  similarly experimental.  As even the SecuShare team
>  seems unsure about the code base's viability, splitting
>  this off into its own Git makes some sense to me.
> 
> - voting + secretsharing: voting is already in gnunet-java
>  and separate because the Java build system is completely
>  different; secretsharing is overall pretty generic
>  and dependency-wise unproblematic. I really don't see
>  a good reason for pulling it out of core. If we did,
>  rps, scalarproduct are in the same category (basically
>  cryptographic building blocks that are not yet in real
>  use). But for me, part of the GNUnet approach is that we
>  need to raise the abstraction of the networking API and
>  provide better cryptographic building blocks at the heart
>  of our infrastructure.

Yeah. I was not sure if the cut should be made at secretsharing as I do not 
really know what is does.
But from the same it could go both ways.

> 
> - Conversation + speaker + microphone / fs (+datastore)
>  How do you propose to build gnunet-gtk with this being
>  moved out of core?
>  Do you really want anyone working on this to first
>  build gnunet-core, then gnunet-conversation, then
>  gnunet-gtk-core, and then gnunet-conversation-gtk?
>  If we merged gnunet-gtk-core into gnunet-core, that
>  would at least be semi-sane, but as proposed, this seems
>  just insane.

If course I expect somebody to build it in this order!

> 
> - pt (not 100% sure what it does maybe it can stay in code
>  depending on how niche the API is)
>  Well, PT (protocol translation, IPv4/IPv6) works with
>  DNS, VPN and EXIT to translate DNS replies (think DNS6to4).
>  As GNS needs VPN for some record types, it is not so trivial
>  to unbundle VPN/EXIT/PT from core.
>  This is a good example why it is not easy to unbundle.

I agree. But this is just because I did not understand pt. When you look into it
even superficially, it is obvious, no?

> 
> - Gtk UI (already in separate repo)
>  If we atomize gnunet.git as you propose, either the gnunet-gtk
>  configure.ac must increase drastically in terms of complexity
>  to test which of the gnunet-atoms is available (welcome to 500
>  character configure invocations) or we atomize this as well,
>  which then means you can write an novel for the
>  installation-from-source handbook on the dependency graph.

That should not be the case. Either GNUnet gtk+ is pluggable or it is a 
monolithic nightmare like GNUnet.git is now including a billion configure 
switches.

> 
> - rest
>  This confuses me the most of your proposal. We have json/,
>  which requires libjansson, not just rest. And there are DB
>  adapters (pq/my/sq) which also can (or maybe ought to) handle
>  JSON data.
>  More importantly, IMO the various rest-plugins probably
>  belong with their respective subsystems, and
>  some of those are in core, like for IDENTITY, the DHT or GNS.
>  So if REST isn't in core, then we get build hierarchies like this:
> 
>  gnunet-core (with GNS)
>    gnunet-gtk
>    gnunet-rest
>      gnunet-dht-rest
>      gnunet-identity-rest
>    gnunet-conversation
>      gnunet-conversation-rest
>      gnunet-conversation-gtk
>      gnunet-fs-rest
>      gnunet-fs-gtk
> 
>  Is this really what you believe will attract new developers!?
>  Dozens of repositories with crazy dependency chains?

Obviously the dependency chains must be reasonable, and if rest (as it is an 
API) makes more sense in the core then so be it.
So core rest plugins and the rest API must stay in code.
But I have absolutely _no_ issue with having the rest plugins in the reclaim 
package otherwise.

> I also do
>  not quite see whatever you want to do with the GitLab CI that
>  could not properly handle the bigger repo we have today
>  (especially given that Buildbot is today fine with it)

It is not a CI issue. It is a build and testing issue (which is automated 
though the CI but that is secondary).
What do you mean buildbot is fine with it? _I_ as a developer am not fine with 
buildbot being triggered to run make check on gnunet.git because a gtk+ widget 
changes.
I am also not file with buildbot being blocked for half an hour because a line 
in gnunet-service-conversation changes.
It is very simple separation of concerns.

> 
> 
> - reclaim (not in that architecture figure because its outdated)
>  --- some of it is already separate, right? Still, I don't quite
>  see why you want to split it off. The result would just be that
>  when I change stuff in GNS and break reclaim, I couldn't even see
>  I broke reclaim and you would get to pick up my mess.

You do not see how
1. This is happening anyway because _you_ (also me) never actually build 
reclaim or <insert other subsystem with deps like gstreamer>.
2. Yes, this is exactly what should happen! You cannot see that you broke 
reclaim and you should not care!
I mean, you should care about breaking API etc, but if you carefully made a 
design decision and implemented it (ideally in a feature branch) and this is
then pushed it is perfectly acceptable that a service that uses it breaks (in 
HEAD only mind you).
An update like that should trigger the build of the subsystem anyway. But the 
point is: You, as the person that changed the core do not break the core build 
and you do not have to care about fixing X subsystems before you can have a 
passing testsuite again which is completely bonkers and we both have done that 
a few times now.

You see, the reasons you give for keeping the complexity are a result from the 
complex dependencies itself as well as the dusted development methodology: 
Development in a single branch of a huge code base with a buildsystem that 
cannot possibly effectively test everything in a reasonable time frame.

> Ok, you
>  sometimes get to do that anyway, but I predict this would increase
>  the frequency ;-).
>  So here, I would rather see libgabe being dragged *into* our
>  build (src/gabe/) than reclaim being pushed out.  After all,
>  ABE might be useful for other things in the future (there is
>  a feature proposal for Taler related to ABE, for example).
> 
> 
> So from this list, the one thing that today for me would make sense to
> move to another repo is the Secushare family. In terms of sanity when
> building from source (and that *is* what developers do) I'd rather
> integrate gabe than split off reclaim.
> 
> Note again that this has nothing to do with the binary package
> structure, this is just for sanity when building from source. The
> proposed structure that might end up with 50+ interdependent Gits would
> drive *me* bananas, and I can't see it being helpful for downstream either.
> 
> But most importantly, I simply don't share your aversion to *optional*
> dependencies (libgabe, libgstreamer, libpq, libmysql, libjansson). None
> of these really justifies a change in our *Git* structure to me.
> 
> 
> For me, any Git structure imposes two main constraints: (1) access
> control (who has commit rights) --- and there we have so far never even
> considered giving some devs access to some repos but not others. (2) is
> that the resulting TGZs usually are 1:1 per source Git. Here, we DO have
> some people building from source (downstream, students, developers, me),
> and each additional TGZ/repo they need to download, configure and build
> takes time without being inherently helpful for anything.
> 
> Sure, if libgabe was NOT an optional dependency and I had to install it
> first, that would be a pain if I don't care about reclaim, but as long
> as it is optional, I really don't see any pain for users to have reclaim
> in the main tree. OTOH, it will take longer to install from source if
> reclaim is a separate repo -- and it'll be easier to install reclaim
> from source if libgabe is integrated.
> 
> 
> But central to the entire argument seems to be that for CI, I don't at
> all see the benefit of more repos either. It just means that the CI
> configuration becomes also more complex, as I now have to teach the CI
> the repo and dependency structure.  Sure, we get in theory a
> combinatorial explosion of feature on/off states, but in practice
> building once with Postgres and once without Postgres doesn't mean we
> have to also try this with all _other_ possible on/off switches as the
> features are usually sufficiently orthogonal.
> 
> 
> Anyway, I should get back to actual coding and cleaning, maybe we should
> resume this discussion at the GNUnet Hacker Meeting? At least I suspect
> that might be more productive and I don't see any urgency here.

Yes, this is fundamental stuff and we cannot solve this in one sweep anyway.
But especially you who introduces a lot of (justified) API breaking changes 
repeatedly should be bothered that you have
to fix all subsystems agains before tests show up green again. Some subsystem 
integrations can, and should, be delayed.

> 
> 
> Happy hacking!
> 
> Christian
> 
> 
> On 2/9/19 11:16 AM, Schanzenbach, Martin wrote:
>> Well I proposed that we look at 
>> https://stage.gnunet.org/en/architecture.html (which is not up to date 
>> currently).
>> IMO you can quickly identify which components are basically just 
>> applications that do not provide fundamental functionality (ie. what I 
>> referred to as "gnunet core"-- Note: this has nothing to do with the GNUnet 
>> core service).
>> 
>> Example:
>> 
>> - Secushare + social
>> - Conversation + speaker + microphone
>> - psyc + psycstore
>> - voting + secretsharing
>> - fs (+datastore)
>> - pt (not 100% sure what it does maybe it can stay in code depending on how 
>> niche the API is)
>> - reclaim (not in that architecture figure because its outdated)
>> - rest
>> - Gtk UI (already in separate repo)
>> 
>> I would argue that each of the above _could_ go into a separate repo for 2 
>> major reasons:
>> 
>> 1. In the overwhelming case, they do not provide core functions to other 
>> services/apps (they do not have "incoming" arrows in the figure)
>> 2. Often, they pull in unreasonable dependencies not required by any other 
>> component (rest: jansson, reclaim: libgabe, conversation: gstreamer etc)
>> 
>> As soon as a component does not fit into any of the above categories, it 
>> could be merged into the "core" repo again.
>> 
>> BR
>> 
>>> On 8. Feb 2019, at 15:53, t3sserakt <address@hidden> wrote:
>>> 
>>> Hey *,
>>> 
>>> I also think it is better to have several repos. I can not tell how to 
>>> split up the gnunet.git repo, but we should not merge gnunet-gtk.git into 
>>> gnunet.git.
>>> 
>>> cheers
>>> 
>>> t3sserakt
>>> 
>>> On 08.02.19 15:00, Schanzenbach, Martin wrote:
>>>> Yes, I do not think this is a good idea at all and is contrary to the 
>>>> initial motivation of this thread.
>>>> 
>>>> We already agree the from a user perspective, the packages (.deb/.rpm et 
>>>> al) should ideally be split into
>>>> the respective services/applications and, of course, also Gtk+. For sane 
>>>> dependency resolution at least.
>>>> 
>>>> But it is also reasonable to separate things at source level as I already 
>>>> gave various reasons, to which I have not heard a counterargument yet 
>>>> except:
>>>> Usability (???).
>>>> You cannot argue with usability because USERS DO NOT INSTALL FROM THE GIT 
>>>> REPO THEY INSTALL PACKAGES.
>>>> And even the packages should be separate as you already agreed!
>>>> 
>>>> A monolith _will_ bite us when it comes to testing and CI.
>>>> Working on a single, huge codebase with a variety of build switches is a 
>>>> pain for testing, development and deployment.
>>>> Not to mention it is difficult to ascertain and ensure for an application 
>>>> what components are built.
>>>> Example: Do you really want to test everthing of the core gnunet functions 
>>>> if a Gtk widget changes?
>>>> Because that will inevitably happen.
>>>> It will be really difficult to setup a CI/automated testing that correctly 
>>>> separates this.
>>>> It will be possible, maybe, but then we have a test process that is 
>>>> equally difficult as our build process.
>>>> 
>>>> 
>>>> 
>>>>> On 8. Feb 2019, at 14:39, Christian Grothoff <address@hidden>
>>>>> wrote:
>>>>> 
>>>>> On 2/7/19 3:21 PM, Hartmut Goebel wrote:
>>>>> 
>>>>>> Am 02.02.19 um 16:09 schrieb Christian Grothoff:
>>>>>> 
>>>>>>> And I wonder if it wouldn't make sense to have the gnunet.git
>>>>>>> configure.ac test for Gtk+ and *if* libgtk is detected, _then_ build Gtk
>>>>>>> GUIs that are _included_ in gnunet.git, instead of requiring the user to
>>>>>>> download and configure yet another TGZ.
>>>>>>> 
>>>>>> *If* the gui is merged into the main repo, I suggest adding
>>>>>> configure-options like `--without-gui`(which AFAIK is a autotools
>>>>>> standard thing) to avoid building the gui even if libgtk is detected.
>>>>>> This might happen if e.g. one is developing on her/his desktop.
>>>>>> 
>>>>> Sure, that makes sense. Any opinions from the silent masses on merging
>>>>> gnunet-gtk.git into gnunet.git and merging the source TGZs?
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> GNUnet-developers mailing list
>>>>> 
>>>>> address@hidden
>>>>> https://lists.gnu.org/mailman/listinfo/gnunet-developers
>>>> 
>>>> 
>>>> _______________________________________________
>>>> GNUnet-developers mailing list
>>>> 
>>>> address@hidden
>>>> https://lists.gnu.org/mailman/listinfo/gnunet-developers
>> 
>> 
>> _______________________________________________
>> GNUnet-developers mailing list
>> address@hidden
>> https://lists.gnu.org/mailman/listinfo/gnunet-developers
>> 
> 
> 
>

signature.asc
Description: Message signed with OpenPGP