Re: [GNUnet-developers] Proposal: Make GNUnet Great Again?

[Christian Grothoff]

On 2/11/19 5:18 PM, Schanzenbach, Martin wrote:
> 
>> On 11. Feb 2019, at 16:51, Christian Grothoff <address@hidden> wrote:
>>
>> Martin, I'm not sure (1) that all of these really compare to GNUnet, and
>> (2) if I look at say Tor, the "tor.git" is 350 kLOC as _one_ component.
>> gnunet.git is 500kLOC, so just a _bit_ larger. Like Tor, we also have
>> many semi-related repos, you can think of ooni == Taler, ascension
>> ==bridgedb, etc. The _core_ Tor system are not the 50 repos you find
>> they have. All a _normal_ user needs is tor.git, which is _completely_
>> self-contained and already usable.
> This is kind of my point. Tor does pretty much one thing. GNUnet is a 
> collection of services.
> Now, there might be a baseline of services/libs without which GNUnet does not 
> really work as a framework/platform.
> But I am arguing that I have trouble accepting fs and social and reclaim as 
> an acceptable baseline.

Btw, related: are you aware the the SecuShare people have discussed
integrating FS-functions into SecuShare? At least that was in Lynx's
plan in the past. Just FYI.

>> Looking at another example, I checked out kdelibs, which is arguably the
>> equivalent of the 'framework' for GNUnet.  After all, the question is at
>> what size one really needs to break things up more, right?  kdelibs has
>> 1399 kLOC (yep, that's just the basic library). So in what respect do
>> these projects where the most fundamental base logic is 3x as big as
>> what we have really _compare_?
> Hmm I am actually not really sure of looking at the raw LOC is telling us 
> anything.
> To me, the GNUnet "base framework" could have any number of LOC. I am more 
> concerned with the
> things that are within this source repo.
> 
> Another way (again) to look at it: What if the fs service or the social or 
> reclaim need "minor" fixes within
> a "major" release cycle of GNUnet? We will always have to bump GNUnet, 
> instead of just one component. (see also grohoffs comments on this, which I 
> agree with)

Yes, I would indeed just bump the GNUnet framework/base minor version
number. Release early, release often. If there is truly no relevant
change in other subsystems, we can always indicate that in the release
notes and people who don't care about the changes can then always skip
that update.

>> The main argument -- build/CI/test time, internal complexity, all falls
>> apart given that they are (overall) at least an order of magnitude
>> larger, and their basic components dwarf our core libraries at this time.
> That is true. But it is not my main argument that build times are too long.
> My argument is that build time is long, and for my service (reclaim) it is 
> not necessary at all to build everything when I fix a bug (locally that does 
> not affect me, but it does affect the CI).
> 

Well, my feeling for the CI was: write better (faster) tests, change
them if possible to run in parallel, and then we can throw our 96 core
system at it and I cannot imagine that until the code grows _much_
bigger we'd still have a serious worry.

But here, again I like data. So let's look at the data.  Assuming the CI
runs things properly isolated (i.e. no port number conflicts due to use
of containers, etc.), we could for example run the tests in every src/
subdirectory in parallel. We have less than 96 of those, so the slowest
directory would be the total runtime (assume we build everything for
every commit for now).  So how long do the tests take? Let me list those
> 10s:

configure time: 18s
Build time (full build): 1m16s

'make check' by subsystem:
ARM: 14s
ATS: 20s
ATS-TESTS: 21s
CADET: 3m46s (!!!)
CORE: 1m50s (!)
DHT: 44s
GNS: 25s
HOSTLIST: 10s
Integration-tests: (hang right now)
NAMESTORE: 1m4s
RPS: 6m9s (!!!!)
SCALARPRODUCT: 20s
TESTBED: 1m36s
TRANSPORT: 19m6s (!!!!!)

You were debating separating out:
CONVERSATION: 1m1s
FS: 3m (!!)
MULTICAST: hangs

Even if we assume we run the tests sequentially, clearly eliminating
secushare/fs is only a small improvement (- 10%, modulo hangs).  Given
that we'd loose ~4x18s (+5%) on additional configure invocations, this
entire argument about speeding up CI by removing
fs/secushare/conversation is not supported by the data.

In most cases, the high delays are because of too high timeouts or other
issues that nobody bothered to fix yet.  For me, it is quite conceivable
that effort spent in optimizing the most offensive high-delay tests
(transport, rps) would make the test suite run faster than equivalent
effort in splitting up the Git.

And with a parallel CI, you'll CADET and TRANSPORT matter first, and if
we fix those, I suspect FS might inherently run faster as well.

So this buries arguments based on CI runtime or kLOC (code size). What
it leaves is:

* "as a developer I don't want to see FS if I work on Reclaim" (easy:
  don't look in those dirs),

* "if FS breaks, it shouldn't break my Reclaim work flow" (then work in
  a branch from a "stable" base and merge whenever the rest is also
  stable)

* "GNUnet is a collection of services" (yes, but is that really
  an argument for every user-facing service to be a separate
  Git/download? Isn't GNS a user-facing "service" as well? This
  seems a pretty arbitrary yardstick to draw such lines by.

Am I missing an argument here?

signature.asc
Description: OpenPGP digital signature