Hi Guixers,
I raised an issue on libgit2 which effects the use of Guix with SSH git clones and additional references:
The issue in summary is that if I want to build off a non-standard git reference (say, a Pull Request), then even if I update my git config to include that, eg:
remote.origin.fetch=+refs/pull-requests/*/merge:refs/remotes/origin/pr/*
libgit2's initial clone will not pull down the extra refs, despite this being the behaviour of the git command line tool proper. After the initial clone, the git config is adhered to.
In Guix this means that the first time I build a PR it fails, and I have to do something like "guix build foo | guix build foo" which is at best a clumsy hack, but it works!
Whilst the proper solution will be be done in libgit2 I was thinking if I could quickly improve on my double-build workaround without having to change Guix itself here (by always fetching):
If nothing else I was hoping to learn a bit more about how packages were compiled as the details are a bit of blindspot for me! So my plan was to duplicate the git-checkout record and it's G-Exp compiler such that I could redirect to a modified version of update-cached-checkout when the source is read from the package.
(I should also mention that the far simpler option of providing a new "fetch" function like url-fetch or git-fetch doesn't work for me as the repo I'm accessing happens to require SSH authentication and thus I'm using the method as outlined here:
http://issues.guix.gnu.org/issue/31285#4 )
So I have something like:
(package
(name "my-test-repo")
(version production-version)
(source
(git-checkout-x-refs
(url "ssh://git@bitbucket:7999/ea/my-test-repo.git")
(commit commit-production)))
(build-system python-build-system)
and so on....
The code I've naively duplicated is at the bottom of this e-mail - it builds OK, so is syntactically correct, but fails when I reference it in a package like above. Having the record and git code as part of my local channel is wrong, I know, but I'm looking a short term workaround with the least disruptive footprint whilst I implement and then wait for release of the strategic solution, probably in libgit2.
I suspect the reason for the failure is that I'd have to import my new duplicated module somewhere in the Guix core code to make this work - i.e. having the code in the channel is never going to work?
I was wondering if anyway could confirm this and/or give me a pointer of where the compliation occurs (where the record in the source is handled and compiled into a git clone) - even if my approach is (quite possibly) unviable, I'd like to understand why!
Thanks,
Phil.
$ guix build -L /home/phil/git/guix/guix-packages/packages my-test-repo
guix build: warning: source _expression_ failed to match any pattern
error: git-checkout-x-refs: unbound variable
hint: Did you forget `(use-modules (my-tools git))'?
guix build: error: my-test-repo: unknown package
_________________________________________________
I put this in
/home/phil/git/guix/guix-packages/package/my-tools/git.scm- I've marked the actual change in bold - everything else is just boilerplate.
;; -*- mode: scheme; eval: (guix-devel-mode 1); geiser-scheme-implementation: guile -*-
(define-module (my-tools git)
#:use-module (git) ;; libgit
#:use-module (guix git) ;; CAREFUL could clash!
#:use-module (guix records) ;; define-record-type*
#:use-module (guix gexp)
#:use-module (ice-9 ftw) ;; scandir
#:use-module (ice-9 match)
#:use-module (srfi srfi-11) ;; let*-values
#:export (git-checkout-x-refs
git-checkout-x-refs?
git-checkout-x-refs-url
git-checkout-x-refs-branch
git-checkout-x-refs-commit
git-checkout-z-refs-recursive?))
;; local functions stolen from guix git
(define clone/swh-fallback (@@ (guix git) clone/swh-fallback))
(define with-libgit2 (@@ (guix git) with-libgit2))
;;;
;;; Checkouts.
;;;
;; Representation of the "latest" checkout of a branch or a specific commit.
;; Shadows git-checkout but uses difference function to get the repo.
(define-record-type* <git-checkout-x-refs>
git-checkout-x-refs make-git-checkout-x-refs
git-checkout-x-refs?
(url git-checkout-x-refs-url)
(branch git-checkout-x-refs-branch (default #f))
(commit git-checkout-x-refs-commit (default #f)) ;#f | tag | commit
(recursive? git-checkout-x-refs-recursive? (default #f)))
(define* (update-cached-checkout-x-ref url
#:key
(ref '())
recursive?
(check-out? #t)
starting-commit
(log-port (%make-void-port "w"))
(cache-directory
(url-cache-directory
url (%repository-cache-directory)
#:recursive? recursive?)))
"Update the cached checkout of URL to REF in CACHE-DIRECTORY. Return three
values: the cache directory name, and the SHA1 commit (a string) corresponding
to REF, and the relation of the new commit relative to STARTING-COMMIT (if
provided) as returned by 'commit-relation'.
REF is pair whose key is [branch | commit | tag | tag-or-commit ] and value
the associated data: [<branch name> | <sha1> | <tag name> | <string>].
If REF is the empty list, the remote HEAD is used.
When RECURSIVE? is true, check out submodules as well, if any.
When CHECK-OUT? is true, reset the cached working tree to REF; otherwise leave
it unchanged."
(define (cache-entries directory)
(filter-map (match-lambda
((or "." "..")
#f)
(file
(string-append directory "/" file)))
(or (scandir directory) '())))
(define canonical-ref
;; We used to require callers to specify "origin/" for each branch, which
;; made little sense since the cache should be transparent to them. So
;; here we append "origin/" if it's missing and otherwise keep it.
(match ref
(() '(symref . "refs/remotes/origin/HEAD"))
(('branch . branch)
`(branch . ,(if (string-prefix? "origin/" branch)
branch
(string-append "origin/" branch))))
(_ ref)))
(with-libgit2
(let* ((cache-exists? (openable-repository? cache-directory))
(repository (if cache-exists?
(repository-open cache-directory)
(clone/swh-fallback url ref cache-directory)))) ;; if the cache doesn't exist, clone
;; Always fetch remote, even if it has not been cloned just before.
(when ;;(and cache-exists?
(not (reference-available? repository ref)) ;;)
(remote-fetch (remote-lookup repository "origin")
#:fetch-options (make-default-fetch-options)))
(when recursive?
(update-submodules repository #:log-port log-port
#:fetch-options (make-default-fetch-options)))
;; Note: call 'commit-relation' from here because it's more efficient
;; than letting users re-open the checkout later on.
(let* ((oid (if check-out?
(switch-to-ref repository canonical-ref)
(object-id
(resolve-reference repository canonical-ref))))
(new (and starting-commit
(commit-lookup repository oid)))
(old (and starting-commit
(false-if-git-not-found
(commit-lookup repository
(string->oid starting-commit)))))
(relation (and starting-commit
(if old
(commit-relation old new)
'unrelated))))
;; Reclaim file descriptors and memory mappings associated with
;; REPOSITORY as soon as possible.
(repository-close! repository)
;; Update CACHE-DIRECTORY's mtime to so the cache logic sees it.
(match (gettimeofday)
((seconds . microseconds)
(let ((nanoseconds (* 1000 microseconds)))
(utime cache-directory
seconds seconds
nanoseconds nanoseconds))))
;; When CACHE-DIRECTORY is a sub-directory of the default cache
;; directory, remove expired checkouts that are next to it.
(let ((parent (dirname cache-directory)))
(when (string=? parent (%repository-cache-directory))
(maybe-remove-expired-cache-entries parent cache-entries
#:entry-expiration
cached-checkout-expiration
#:delete-entry delete-checkout
#:cleanup-period
%checkout-cache-cleanup-period)))
(values cache-directory (oid->string oid) relation)))))
(define* (latest-repository-commit-x-ref store url
#:key
recursive?
(log-port (%make-void-port "w"))
(cache-directory
(%repository-cache-directory))
(ref '()))
"Return two values: the content of the git repository at URL copied into a
store directory and the sha1 of the top level commit in this directory. The
reference to be checkout, once the repository is fetched, is specified by REF.
REF is pair whose key is [branch | commit | tag] and value the associated
data, respectively [<branch name> | <sha1> | <tag name>]. If REF is the empty
list, the remote HEAD is used.
When RECURSIVE? is true, check out submodules as well, if any.
Git repositories are kept in the cache directory specified by
%repository-cache-directory parameter.
Log progress and checkout info to LOG-PORT."
(define (dot-git? file stat)
(and (string=? (basename file) ".git")
(or (eq? 'directory (stat:type stat))
;; Submodule checkouts end up with a '.git' regular file that
;; contains metadata about where their actual '.git' directory
;; lives.
(and recursive?
(eq? 'regular (stat:type stat))))))
(format log-port "updating checkout of '~a'...~%" url)
(let*-values
(((checkout commit _)
(update-cached-checkout-x-ref url
#:recursive? recursive?
#:ref ref
#:cache-directory
(url-cache-directory url cache-directory
#:recursive?
recursive?)
#:log-port log-port))
((name)
(url+commit->name url commit)))
(format log-port "retrieved commit ~a~%" commit)
(values (add-to-store store name #t "sha256" checkout
#:select? (negate dot-git?))
commit)))
(define* (latest-repository-commit-x-refs* url #:key ref recursive? log-port)
;; Monadic variant of 'latest-repository-commit-x-refs'.
(lambda (store)
;; The caller--e.g., (guix scripts build)--may not handle 'git-error' so
;; translate it into '&message' conditions that we know will be properly
;; handled.
(catch 'git-error
(lambda ()
(values (latest-repository-commit-x-ref store url
#:ref ref
#:recursive? recursive?
#:log-port log-port)
store))
(lambda (key error . _)
(raise (condition
(&message
(message
(match ref
(('commit . commit)
(format #f (G_ "cannot fetch commit ~a from ~a: ~a")
commit url (git-error-message error)))
(('branch . branch)
(format #f (G_ "cannot fetch branch '~a' from ~a: ~a")
branch url (git-error-message error)))
(_
(format #f (G_ "Git failure while fetching ~a: ~a")
url (git-error-message error))))))))))))
(define-gexp-compiler (git-checkout-x-refs-compiler (checkout <git-checkout-x-refs>)
system target)
;; "Compile" CHECKOUT by updating the local checkout and adding it to the
;; store. Handle the libgit2 issue by fetching refs, even on a clone.
(match checkout
(($ <git-checkout-x-refs> url branch commit recursive?)
(latest-repository-commit-x-refs* url
#:ref (cond (commit
`(tag-or-commit . ,commit))
(branch
`(branch . ,branch))
(else '()))
#:recursive? recursive?
#:log-port (current-error-port)))))