27/66: icse-2022: Add reviews and response.

[Ludovic Courtès]

civodul pushed a commit to branch master
in repository maintenance.

commit cece5df6183fa821a4f7ca04d4fff3f46e847dbe
Author: Ludovic Courtès <ludo@gnu.org>
AuthorDate: Mon Jan 10 08:51:38 2022 +0100

    icse-2022: Add reviews and response.
    
    * doc/icse-2022/icse-response.txt,
    doc/icse-2022/icse-reviews.txt: New files.
---
 doc/icse-2022/icse-response.txt |  88 +++++++++
 doc/icse-2022/icse-reviews.txt  | 385 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 473 insertions(+)

diff --git a/doc/icse-2022/icse-response.txt b/doc/icse-2022/icse-response.txt
new file mode 100644
index 0000000..2920115
--- /dev/null
+++ b/doc/icse-2022/icse-response.txt
@@ -0,0 +1,88 @@
+-*- Markdown -*-
+
+We thank the reviewers for their detailed feedback.  Please find below
+responses to specific issues raised in each review, including factual
+inaccuracies we found in reviews #418B and #418C.
+
+# Review #418A
+
+   - **Novelty**: Other source-based package managers do not implement
+     any checkout authentication mechanism.  They rely on X.509
+     certificates via HTTPS access; this protects against eavesdropping
+     and host impersonation but does not provide any protection against
+     unauthorized modifications to the repository, for example if the
+     server is compromised, nor against downgrade attacks.
+   - Proposal: Explicitly state that similar tools (Nix, Spack, BSD
+     Ports, etc.) have no secure update mechanism in place other than
+     HTTPS.  Users of Git in general do not have any secure update
+     mechanism (Sec. 9).  (Both of these are stated in ref. [4].)
+   - Proposal: Augment the introduction to clarify what Guix is and what
+     the “secure update mechanism” is.
+   - Response: In case of a stolen developer laptop, the solution would
+     be to eventually push a revert of the offending commit; there is
+     currently no way to flag past commits.
+
+# Review #418B
+
+   - **Transparency**: source code is available; users can run `guix
+     pull` to measure the authentication throughput and verify the
+     performance claim of Sec. 8.3, contrary to what the review says.
+   - On the title: Secure update is a key aspect of supply chain
+     security.  Because Guix differs from more widespread package
+     managers, its secure update mechanism cannot be studied in
+     isolation.  Our goal is to show in Sec. 2 how Guix already
+     addresses other aspects of supply chain security through
+     reproducible/verifiable builds and an auditable, transparent
+     bootstrap.  Secure updates were the missing piece of the puzzle but
+     we deemed it important to show how all the pieces fit together to
+     achieve the overarching goal set in the title.
+   - **Novelty**: while Git commit authentication (signed commits) is not
+     novel, the authorization framework is entirely novel, contrary to
+     what the review suggests.  To our knowledge, there exist no tools
+     or protocols implementing a comparable authorization framework for
+     Git and other version control systems.
+   - As discussed near line 755, recent work on supply chain security
+     (in-toto, SLSA, sigstore) focuses on _attestation_ when Guix instead
+     insists on _verifiability_ and _auditability_ (Sec. 2).  That, too,
+     sets its apart from related work.
+   - **Soundness**: Sec. 4, 5, and 6 are our attempt at showing soundness of
+     the proposed techniques.  For example, whether a commit satisfies
+     the authorization invariant is trivially answered; we thus conclude
+     that this method allows us to detect 
+
+# Review #418C
+
+   - **Transparency**: Code is published (at https://guix.gnu.org),
+     contrary to what is written.
+   - Contrary to what the reviewer implies, Guix existed prior to the
+     work described here (Sec. 2, ref. [7]); this work focuses on
+     _extending_ it with a secure update mechanism (Sec. 3 and
+     subsequent).  Thus, authenticating Git checkouts is not the primary
+     motivation for writing Guix; instead, Guix existed and needed a
+     solution to this problem.  We do not see any ambiguity in the text
+     in this regard.
+   - **Novelty**: The motivation for the secure update mechanism is that
+     no existing solutions address the needs of Guix listed in Sec. 3.
+     In particular, the state-of-the-art solutions, TUF and in-toto, are
+     biased towards repositories of binary artifacts and unsuitable
+     (Sec. 9 and review #418A).
+   - The sentences “_Our main contribution is (…)_” (abstract), “_There
+     were no existing tools (…) supporting off-line checking
+     authentication_” (Sec. 4), “_this is the first time downgrade
+     prevention is implemented at this level_” (Sec. 6), and “_this is
+     the first client-side-only Git update authentication mechanism_”
+     (conclusion) were meant to be explicit about the novelty of this
+     work.
+   - The rules in lines 255–262 are one contribution.  The other
+     contributions are: trust establishment (Sec. 5), protection
+     against downgrade attacks by taking advantage of the commit graph
+     (Sec. 6), and mitigation against stale mirrors (Sec. 7).
+   - Proposal: As you suggest, we would add a few sentences in the
+     introduction to restate the goals and contributions upfront.
+   - Proposal: Augment abstract and introduction to explicitly state
+     that existing secure update solutions are inadequate (TUF) or
+     lacking (other source-based package managers provide no update
+     authentication to our knowledge).
+   - Line 441 should read “_because it starts from _A_, which is not a
+     descendant of the introductory commit_” (_A_ might lack
+     `.guix-authorizations`, but not necessarily).
diff --git a/doc/icse-2022/icse-reviews.txt b/doc/icse-2022/icse-reviews.txt
new file mode 100644
index 0000000..5c06eee
--- /dev/null
+++ b/doc/icse-2022/icse-reviews.txt
@@ -0,0 +1,385 @@
+From: "ICSE 2022 HotCRP" <noreply@icse2022.hotcrp.com>
+Subject: [ICSE 2022] Reviews Ready: Authors' Response by Nov 13 - Paper #418
+ "Building a Secure Software Supply Chain..."
+To: Ludovic Courtès <ludovic.courtes@inria.fr>
+Date: Mon,  8 Nov 2021 20:33:17 +0000 (UTC) (3 days, 17 hours ago)
+Reply-To: icse2022chairs@cispa.de
+
+Dear Ludovic Courtès,
+
+Thank you for your submission to the ICSE 2022 Technical Track
+
+       Building a Secure Software Supply Chain with GNU Guix
+
+The author response period is now open with the deadline for response by
+
+       Saturday November 13, 2021 (11:59pm, AoE).
+
+The reviews for your paper are attached to this mail. A pretty-printed
+version can be retrieved at
+
+       https://icse2022.hotcrp.com/paper/418
+
+
+What's new with ICSE 2022 Reviews and Authors' Responses
+--------------------------------------------------------
+
+* Reviews are organized along detailed *review criteria* (see #1, below)
+* Reviewers have been instructed to *list only questions that they
+  think could alter their stance* (see #8, below)
+* We have a *750 word soft limit* for authors' responses (see #10,
+  below).
+
+
+About ICSE 2022 Authors' Responses
+----------------------------------
+
+1. Reviewers have been instructed to assess your paper by _five
+criteria_, as detailed in the ICSE 2022 Reviewer Guidelines at
+
+       https://tinyurl.com/icse2022reviewing
+
+  notably Section 6 "Reviewing". Specifically, these criteria are:
+
+  * *Soundness*: The extent to which the paper's contributions and/or
+     innovations address its research questions and are supported by
+    rigorous application of appropriate research methods.
+
+  * *Significance*: The extent to which the paper's contributions can
+     impact the field of software engineering, and under which
+    assumptions (if any).
+
+  * *Novelty*: The extent to which the contributions are sufficiently
+     original with respect to the state-of-the-art.
+
+  * *Verifiability and Transparency*: The extent to which the paper
+     includes sufficient information to understand how an innovation
+    works; to understand how data was obtained, analyzed, and
+   interpreted; and how the paper supports independent verification or
+  replication of the paper's claimed contributions.
+
+  * *Presentation*: The extent to which the paper's quality of writing
+     meets the high standards of ICSE, including clear descriptions,
+    as well as adequate use of the English language, absence of major
+   ambiguity, clearly readable figures and tables, and adherence to
+  the formatting instructions.
+
+2. Each review provides a score from "reject" (score: 1) to "accept"
+(scores: 4 and 5). A "weak reject" (score: 2) and a "weak accept" 
+(score: 3) indicates that reviewers think they may still change their
+score in light of other reviews, the authors' response, or the 
+discussion.
+
+3. After the discussion following the author's response period, a
+paper typically needs at least one champion to be considered for
+acceptance - that is, at least one positive score (3 or higher).
+
+4. Please study the reviews carefully and try to understand the
+reviewers' concerns, as summarized in the fields on strengths and 
+weaknesses.
+
+5. The authors' response is an opportunity for clarification. If you
+believe the reviews contain factual errors, or that the reviewers 
+misunderstood your paper, please point this out, or include proposals
+on how to improve your paper's presentation to address
+misunderstandings as emerging from the reviews.
+
+6. The review process remains double-anonymous. Please do *not*
+provide additional links or clues that can disclose your identity, as
+doing so could result in rejection of your paper.
+
+7. Unless specifically asked for by reviewers, the authors' response
+cannot include any new research results (e.g., additional experimental 
+results) that are not in the paper as submitted.
+
+8. Reviewers provide “Questions for authors to respond to”, and it is
+these questions your response should address. Reviewers have been 
+instructed to list **only questions that they think could alter their
+stance**, and in decreasing order of importance. Hence, focus on
+factual errors first, and then the top questions.
+
+9. Authors' responses are optional, though providing a response will
+enable the reviewers to further discuss your paper.
+
+10. Reviewers need not read beyond the first 750 words of your
+response; thus, be sure to start your response with the most important
+answers and arguments. Words beyond the limit will be highlighted by
+the HotCRP submission system.
+
+11. Reviews may be updated after the end of the response period to
+reflect the discussion among reviewers and take your response into 
+account.
+
+
+How to Submit ICSE 2022 Authors' Responses
+------------------------------------------
+
+To submit your response, log into HotCRP:
+
+       https://icse2022.hotcrp.com/paper/418
+
+and click on the "Add Author-Comments response" at the
+bottom. Markdown styling and LaTeX math are supported; feel free to
+make use of enumerations and headings to structure your response.
+
+When your response is ready, check the "The response is ready for
+review" box and submit; otherwise, it remains as a draft and will not
+be visible to the reviewers.
+
+The program committee members will read your response carefully and
+take it into account during the discussions.
+
+We thank you again for your submission to ICSE 2022!
+
+Best regards,
+
+Daniela Damian + Andreas Zeller
+ICSE 2022 Program Co-Chairs
+
+
+--
+
+Review #418A
+===========================================================================
+
+Overall merit
+-------------
+3. Weak accept
+
+Paper summary
+-------------
+# Building A Secure Software Supply Chain with GNU Guix
+
+## Summary
+Supply chain attacks are an ongoing concern in software engineering. Numerous 
attacks have
+been successful so far and have resulted in leak of sensitive data, privacy, 
and financial
+damage. Hence, preventing such supply chain attacks are important. This paper 
presents one
+of the case studies in using a secure update frame with GNU Guix.
+
+GNU Guix is a part of a new family of Linux distributions/deployment tools 
(along with
+Nix) that aims at producing reproducible builds by removing nondeterministic 
parts of the
+build process (also called functional package management). With such systems, 
the idea is
+to capture all dependencies of a build artifact such that the artifact can be 
recreated bit
+by bit given the same inputs. This means that Guix is source centric, which 
makes the
+recommended update frameworks such as TUF (which focus on binaries) difficult 
to apply.
+
+The solution found here was to rely on a chain of trust based on the git 
version control
+system and signing the commits. The idea is to start with a trusted 
`.guix-authorizations`
+that list the keys of developers who are allowed to commit to a repository. 
These developers
+can then add or remove new collaborators at later signed commits, who are then 
given the same
+privilege, and each commit can then be authenticated to have come from trusted 
sources.
+
+This mechanism prevents both teleportation, and downgrade attacks by ensuring 
only
+later commits are pulled into a system. The system, Guix, has deployed this 
framework for over
+a year (since June 2020), and the downgrade protection has already proved 
useful.
+
+Strengths
+---------
+* Provides a detailed report of secure update system as implemented in GNU 
Guix, which is sufficient to understand the properties of the system.
+* The described system is sound, and has been deployed in the world.
+
+Weaknesses
+----------
+* The presentation can be improved. In particular, the introduction is very 
limited and gives the reader no idea of the system being described.
+* It would have been good to understand whether other source based systems 
such as BSDs and Nix have a secure update policy in place, and if so, in what 
respects they are lacking.
+
+Comments for authors
+--------------------
+## Soundness
+
+The described system for secure source updates is sound, with no obvious 
flaws. The system has
+been in production for over a year with no vulnerabilities found, which 
improves the confidence
+in its soundness. There are no research questions being evaluated per se, but 
as an experience
+report, I believe that the techniques described in this paper are sound.
+
+## Significance
+
+Supply chain attacks are an ongoing concern, and verifiability of updates is 
an important problem.
+Systems such as Nix and Guix are fundamentally source based distributions 
which are different
+from binary distributions such as apt. Hence, solutions that apply to binary 
based distributions
+do not necessarily apply to these systems. The systems such as Guix and Nix is 
important in their
+approach to provide reproducible builds. Hence, the security of such systems 
is of concern in software
+engineering. Hence, this contribution is significant in improving the state of 
the art in SE practice.
+
+## Novelty
+
+The described method has not been used in other projects, and hence, the 
process described is novel.
+
+## Verifiability
+
+Since this is an experience paper describing the update mechanism, there is 
little to verify.
+Furthermore, Guix is open source, and the lack of vulnerabilities in its 
updates can be verified.
+
+## Presentation
+
+The paper is well written. However, the structuring is somewhat confusing. The 
introduction is just
+two paragraphs (excluding listing of sections), and gives no indication of 
what the "secure update mechanism"
+is.
+
+Questions for authors to respond to
+-----------------------------------
+* Say there is a malicious commit being pushed from one of the authorized 
developers (the laptop was stolen, and the commit was pushed before the 
developer could revoke the key). This is then detected later. Is there a 
mechanism by which such a commit can be flagged? and the users who have already 
applied this update downgrade it securely and automatically? or does it require 
manual intervention?
+
+
+* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
+
+
+Review #418B
+===========================================================================
+
+Overall merit
+-------------
+2. Weak reject
+
+Paper summary
+-------------
+-      Topic is relevant to ICSE
+-      It is a sound implementation of the authorization and authentication of 
commits
+
+Strengths
+---------
+-      Topic is relevant to ICSE
+-      It is a sound implementation of the authorization and authentication of 
commits
+-      Clear practical contribution of the paper
+
+Weaknesses
+----------
+* Validation of the approach does not follow the scientific approach
+* The extension is not related to the title and focus of the paper. *
+ The paper focus too much on the design and implementation of Guix
+ authentication and authorization mechanisms and does not focus on
+ the topic of the title. * The research questions of the paper are not
+clear. * Research contribution is not clear
+
+Comments for authors
+--------------------
+•      Soundness: The paper is a practical implementation of an
+         authentication and authorization mechanism for Guix that can
+         provide effects to a secure software supply chain. But there
+         is a long way to get to build a secure software supply
+         chain. The developers’ authentication and authorization is
+         one of the weak links that needs to be addressed. The
+         implementation rational is well explained in the paper. But
+         the validation of the effectiveness of the approach is not
+         scientifically sound. The validation should show
+         scientifically the effects of the approach to the goals that
+         the authors have to the tool. As it is now, it is a nice
+         experience report of the implementation of the extension to
+         Guix and an anecdotal description of the possible
+         effects. The reproducibility of the study is not possible
+         with the way it is described. The scientific methodology for
+         the empirical study is not clear. •   Significance: Software
+        developers needs proper and transparent ways to securely
+        develop software. Then the contribution is relevant to the
+        field, specially to practice. But it is not clear how the
+        contribution is relevant to the state of the art.  •
+       Novelty: Adding authorization and authentication to the
+       commits is not per si a new concept, but the way the authors
+       have developed can potentially produce  the effects claimed by
+       the authors. But the paper is not adding to the evidence based
+       knowledge on that because of the lack of rigour on the
+       validation of the implementation. •     Verifiability and
+      Transparency: The study is not reproducible. The implementation
+      of the extension is well explained that it is possible to
+      reproduce if other researchers would like to do in another
+      tool. •  Presentation: The title does not match the paper. The
+     authors are overstating on the title to the work that was
+     done. The validation needs to be scientifically performed.
+
+
+* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
+
+
+Review #418C
+===========================================================================
+
+Overall merit
+-------------
+1. Reject
+
+Paper summary
+-------------
+GNU Guix is a set of software deployment tools, and it includes a package 
manager. Guix is a “source-based” deployment tool, which means that every piece 
of software is built from source. The authors of the paper present a Git 
checkout authentication mechanism that can help preventing attacks, such as 
downgrade attacks and teleport attacks.
+
+Strengths
+---------
+- The authors have deployed their approach for more than a year to experiment.
+- Supply chain security is an important and timely topic.
+
+Weaknesses
+----------
+- The contribution of the paper is not clear.
+_ The paper needs to be restructured.
+- The experiment result is incomplete.
+
+Comments for authors
+--------------------
+Soundness: The soundness is low - no clear research questions or rigorous 
application of appropriate research methods.
+Significance: The contributions are not clear.  The paper does not
+make clear the specific contributions of Guix over other solutions.
+Novelty: The contributions are not clear, though I suspect there are
+novel aspects to thew or.  Verifiability and Transparency: The data
+and code is not published.
+Presentation: The paper needs to be restructured. The motivation for
+needing a new tool needs to be clear at the start of the paper -- and
+be substantiated throughout the paper.  The threat models which are
+spread throughout the paper should be consolidated.  
+Detailed comments
+
+Abstract:
+- The abstract is incomplete. The goal of the paper is not clear (focuses on 
the security of updates with Guix for what?). No mentioning of research that 
was done -- research approach or findings.  Currently, it seems like an 
engineering, not a research paper.
+
+Introduction:
+- What is the motivation of your work in terms of shortcomings to other tools?
+- What is the goal statement of the paper?, What are research questions that 
you want to answer?, What is the summary of your approach?, What are the 
contributions of the paper.
+
+
+Background:
+- Readers often skip background sections.  I would not put the essential 
information about Guix in the background section (first two paragraphs).
+
+Rationale:
+- A research question is mentioned in the second paragraph. It is better to 
move it to the introduction.
+- In the last sentence you said "These are the kind of attacks we want to 
protect against." Are these just examples of attacks or all the attacks? You 
mentioned three types of attacks but just the two of them have names, what 
about the first one?
+- You might want to call this section "Threat Model".
+
+Authenticating Git Checkouts:
+- The section is not structured well. Is this a methodology section? It is not 
clear based on the title and the first paragraph of it.
+- In the first paragraph, you mentioned that the problem is not
+  specific to Guix. In that case, Guix is just a case study of your
+  work. It is not clear based on your title, abstract, and the
+  research question.  It seems now that authenticating git checkouts
+  may be the primary motivation for creating Guix since you say that
+  "no existing tools or protocols support[ing] off-line checkout
+  authentication".  If that is the primary shortcoming of existing
+  tools, you should put that in the Introduction.  - Are rules listed
+ in lines 257-261 your contributions? If yes, why in line 319 is it
+ mentioned that "This is what Guix pull implements"? If not, why in
+ line 255 it is mentioned "To implement that, we came up with the
+ following mechanism and rule"? It is not clear what is your
+ contribution and what is existing implementation?
+
+Establishing trust:
+- In line 441, why does A lack .guix-authorizations?
+
+Downgrade attacks:
+- What about other types of attacks you mentioned in lines 206-214? The one 
that has no name and teleport attack? Why is there no separate sections for 
them?
+- Is this section your contribution or just what exists in the literature? It 
is not clear.
+
+Mirrors and The Risk of Staleness:
+- Same as the previous section, it is not clear which part is your 
contribution?
+
+Implementation:
+- You have missed the methodology section before this section. We still don't 
know your contribution, and what process you perform.
+- The writing is lines 533-543 is not scientific. You can not make a choice 
because you "feel" something. You need justifications. This comment applies to 
some other parts of the paper as well.
+- Unfortunately, results in Section 8.5 are incomplete. For example, Why you 
say "Downgrade prevention has had a more visible impact"? What was the 
frequency of these attacks before and after your implementation? You need more 
analysis/collected data to report here.
+
+Related Works:
+- What are the similarities and differences of any of these related works to 
your study?
+
+Conclusion:
+- Again, what is the justification to say "our experience so far has been 
positive"? You need reported data to substantiate that.
+
+Questions for authors to respond to
+-----------------------------------
+1.  I understand software supply chain is a major issue.  But, what was 
missing from existing tools that motivated the creation of Guix?  Please make 
that clear in the paper.
+