Re: [GNUnet-developers] Proposal: Make GNUnet Great Again?

[Florian Dold]

On 2/10/19 1:55 PM, Schanzenbach, Martin wrote:

>> An example for such
>> tooling would be Googles's Repo tool
>> (https://source.android.com/setup/develop /
>> https://source.android.com/setup/develop/repo).
> 
> 
> Actually, google is an example for a proponents of monorepos. So your point 
> is moot here.
> They need all this tooling _because_ they use a single repo.

Not quite.  While most of Google's code is in a monorepo (that doesn't
use git), Android is split over multiple git repos.

Also they need all this tooling because they have many projects in one
monorepo.  What I'm saying here is that we shouldn't split up one
*project* too much.  Then we don't need that extra tooling!

> Did you even read my last comment? Do you really consider all of the 
> applications as one "GNUnet" that every
> user (and developer!) actually cares about?
> I can tell you the number of times I used / developed something for fs / 
> social: 0.
> And, of course smaller repos make CI faster. It will result in smaller builds 
> (and, more importantly, builds which actually build things that have changed).
> And please no arguments for stateful builds/runners. I hope we can at least 
> agree that tests and builds should be done in clean environments every time.
> Else you will not catch a lot of stuff that can go wrong (I experienced this 
> myself when I setup my docker builds for GNUnet which, unlike BB, actually 
> build from scratch)

Is tracking dependencies between repos really that much easier than
tracking dependencies between subdirectories?

You are (at least I think) making the assumption that high-level GNUnet
repos would depend on stable versions of GNUnet?

Otherwise, the cost for CI would be the same.  If I make a change in
GNUnet-base, of course I'll also have to run tests for everything that
depends on it.  Are you suggesting to just not do integration testing
anymore?

> How would that happen? Can you give a _concrete_ example in fs/social/reclaim 
> where this is true?
> It is exactly the point that it is completely unclear what effects a 
> configuration switch on what components.
> If we separate this, we might get _some_ overhead in the configure.ac's but 
> from my experience I expect this to be very little.
> And our configure.ac is not something I would consider "developer friendly" 
> through its sheer size and complexity.

Let's say I want to enable verbose logging.  Then I need to re-configure
and re-compile all of the repos I depend on, in the right order.  That
sucks!

(Re-linking doesn't suffice, as logging is done via CPP macros.)

> And think about it this way: If a new developer decides to write a new 
> service / application on top of GNUnet, what will he be faced with?
> Image this app needs only GNS and maybe CADET.
> The dev will need to integrate and understand the full build and test in 
> order to properly setup this project.
> If we had a few separate examples of how this can be done in a separate repo, 
> this would go a long way.

I don't see any issues with "incubating" new GNUnet
applications/libraries in their own repo.  Maybe some of them can then
be later integrated with the main gnunet.git.

(I just saw that in your latest email, you're suggesting exactly this
for reclaimid.  That's is fine, but of course now makes it hard to see
for other GNUnet devs locally whether they are breaking your code.
Well, at least the CI should eventually complain.)

- Florian

>> On 2/10/19 1:02 AM, Amirouche Boubekki wrote:
>>> I think splitting the codebase will be a pain for gnunet.
>>>
>>> The only *good* reasons for manyrepos are social or ego politics "this
>>> is my lawn" or legal. The only one that applies to gnunet is legal
>>> because one needs to fill a gnu form to be able to contribute.
>>>
>>> I am biased toward monorepo by experience dealing with big project
>>> (100k+ SLOC) and the only time it made sens to split the project into
>>> many repositories because it was different teams / workflow (social) and
>>> different legal terms for the various services/daemons, at previous
>>> $WORK, they had to fork gentoo to make it work.
>>>
>>> Otherwise, each time I saw another repository it was a source of pain:
>>>
>>> - Need to manage several versions
>>> - git submodule workflow is not good enough, it doesn't track branch, I
>>> personally I never remember how to know the branch of a commit, plus it
>>> requires some more git-fu to bump the submodule.
>>> - refactoring anyone?
>>> - generally speaking manyrepos at small scale is more work
>>>
>>> And again, it requires somehow to track down every versions (what works
>>> with what) and you end up with another repository (or distribution) with
>>> another build system that puts everything together. Continuous
>>> Integration can do that? Where is the code of the CI? Another repo? More
>>> versions, more git clone more grep across repositories / directories not
>>> even in sync.
>>>
>>> Popularity arguments:
>>>
>>> a) Ok, everybody know GAFAM love monorepos and that is a also a source
>>> of pain (dedicated team and software). That said, gnunet is not the size
>>> of any GAFAM, hence it will not suffer from monorepo pain points.
>>>
>>> b) Github and Javascript made the manyrepos popular for various ego
>>> reasons and because JavaScript is not good. I won't take inspiration
>>> from that part of the JavaScript noosphere. gnunet-leftpad anyone?
>>>
>>> c) Now, there is GNOME. GNOME is famous for its bazaar model of
>>> development and also famous for the adoption of meson (maybe even its
>>> inception) or its previous incarnation jhbuild. Anyway, even if GNOME
>>> and GNU (which is also a bazaar) success is appealing, gnunet is not GNU
>>> or GNOME. From my point of view the bazaar development model scales
>>> better / more easily in a socially distributed setting. Also why Linux
>>> is still a single repository?
>>>
>>> Le sam. 9 févr. 2019 à 18:16, Schanzenbach, Martin
>>> <address@hidden <mailto:address@hidden>> a écrit :
>>>
>>>
>>>
>>>> On 9. Feb 2019, at 17:13, Christian Grothoff <address@hidden
>>>    <mailto:address@hidden>> wrote:
>>>>
>>>> On 2/9/19 5:04 PM, Schanzenbach, Martin wrote:
>>>>> I have some inline comments as well below, but let us bring this
>>>    discussion down to a more practical consensus maybe.
>>>>> I think we are arguing too much in the extremes and that is not
>>>    helpful. I am not saying we should compartmentalise
>>>>> GNUnet into the tiniest possible components.
>>>>> It's just that I think it is becoming a bit bloated.
>>>>>
>>>>> That being said, _most_ of what is in GNUnet today is perfectly
>>>    fine in a single repo and package.
>>>>> For now, at least let us not add another one (gtk) as well?
>>>>>
>>>>> Then, we remain with
>>>>>
>>>>> - reclaim (+the things reclaim needs wrt libraries)
>>>>> - conversation (+X)
>>>>> - secureshare (+X)
>>>>> - fs (+X)
>>>>>
>>>>> as components/services on my personal "list".
>>>>> I suggest that _if_ I find the time, I could extract reclaim into
>>>    a separate repo as soon as we have a CI and I can
>>>>> test how it works and we can learn from the experience.
>>>>> Then, we can discuss if we want to do the same with other
>>>    components, one at a time, if there is consensus and a person that
>>>>> would be willing to take ownership (I am pretty sure we talked
>>>    about this concept last summer as well).
>>>>
>>>> Maybe you could start with extracting the SecuShare components? That
>>>> should do for a first "experience", and be a bit more effective at
>>>> reducing bloat as well ;-).
>>>
>>>    Well, I could, but our secushare people are quite active so maybe
>>>    there are volunteers (if they agree with the proposal at all).
>>>    Regarding "bloat". If we want to effectively eliminate bloat than
>>>    let's look at numbers:
>>>
>>>    File Sharing:
>>>    src/fs: 36918 (!) LOC in .c files
>>>    src/datastore/cache: ~15k LOC in .c files
>>>
>>>    Conversation:
>>>    src/conversation: 10538 LOC in .c files
>>>
>>>    SecuShare:
>>>    src/psyc* : ~17000 LOC in .c files (altough I am not sure about this
>>>    because theoretically psyc is a general use protocol, no?)
>>>    src/social: 9447 LOC in .c files
>>>    src/multicast: 5633 LOC in .c files
>>>
>>>    Reclaim:
>>>    src/reclaim* : ~6500 LOC in .c files
>>>
>>>    Now, considering that fs is practically always built for everybody
>>>    and SecuShare and reclaim are experimental, it hurts the most for
>>>    devs that actually compile from source.
>>>    Everything combined are 110000+ LOC which is 22% of the codebase
>>>    (~500k, oO). Considering that there is a significant redundancy in
>>>    transport/ (75k) at the moment, this number is probably closer to 25%.
>>>    Granted, this is a lot less than I expected ;), but maybe
>>>    illustrates the dimensions.
>>>
>>>
>>>>
>>>> That said, splitting of reclaim seems also much less problematic than
>>>> fs/conversation, and if you then integrate reclaim with the libgabe
>>>> tree, the overall number of downloads/installation for reclaim
>>>    wouldn't
>>>> go up, so that would certainly kill my argument of making the
>>>> installation more complex (might indeed simplify it, as one
>>>    doesn't have
>>>> to remember to install libgabe before GNUnet to get reclaim).
>>>
>>>    Could do, but libgabe has some nasty additional deps (libpbc and
>>>    gmp) which we _might_ eventually get rid of completely by
>>>    implementing GNS-based encryption.
>>>
>>>
>>>    _______________________________________________
>>>    GNUnet-developers mailing list
>>>    address@hidden <mailto:address@hidden>
>>>    https://lists.gnu.org/mailman/listinfo/gnunet-developers
>>>
>>>
>>> _______________________________________________
>>> GNUnet-developers mailing list
>>> address@hidden
>>> https://lists.gnu.org/mailman/listinfo/gnunet-developers
>>>
>>
>> _______________________________________________
>> GNUnet-developers mailing list
>> address@hidden
>> https://lists.gnu.org/mailman/listinfo/gnunet-developers
>

signature.asc
Description: OpenPGP digital signature