[O] [PATCH] Fix and optimize publishing cache check

[Matt Lundin]

This patch does two things:

1. It prevents org-publish from perpetually republishing files
containing includes. Currently, if an included file changes, it is
published, as is the file that includes it. However, until changes are
made in the file that includes it (which may be never) its modification
time remain older than that of the included file, so it will be
republished every time org-publish is called, even if neither file has
changed. 

   - Note: This patch fixes this behavior by updating the modification
     time of the file that contains the includes. If this is deemed too
     destructive/meddlesome, we could consider alternate behaviors, such
     as setting the cached time with a simple float-time. Let me know
     what would be best.

2. It optimizes checking for included files. Currently,
org-publish-cache-file-needs-publishing visits every file in a project
to check for includes. On my underpowered box, this takes a long time
(over a minute) on a project with 1000+ files, thus defeating the
purpose of the cache. This patch causes org-publish to store information
about included files in the cache itself:

  a. If a file has changed it updates information about the files it
  includes and stores it in the cache.

  b. If a file has not changed, it checks the cache for included files
  and sees if any of those files have been updated (thus preventing the
  need to check every file in the project for includes)

Best,
Matt

>From 94a8061bb30e1992213fb8e71ee949d336d37435 Mon Sep 17 00:00:00 2001
From: Matt Lundin <address@hidden>
Date: Tue, 12 Aug 2014 07:51:44 -0500
Subject: [PATCH] Fix and optimize publish cache check

* lisp/ox-publish.el: (org-publish-cache-file-needs-publishing) Fix
  org-publish-cache-file-needs-publishing to change timestamp of files
  containing includes. Speed up check for includes by storing
  information about included files in cache itself.

This patch ensures that org-publish-cache-file-needs-publishing does
not keep publishing a file containing includes by updating the
modification time of that file. It also speeds up publishing by
caching information about included files, thus keeping
org-publish-cache-file-needs-publishing from having to visit every
file just to check includes (which can take a long time on a project
containing hundreds or thousands of files).
---
 lisp/ox-publish.el | 60 +++++++++++++++++++++++++++++++-----------------------
 1 file changed, 35 insertions(+), 25 deletions(-)

diff --git a/lisp/ox-publish.el b/lisp/ox-publish.el
index df40572..228411f 100644
--- a/lisp/ox-publish.el
+++ b/lisp/ox-publish.el
@@ -1168,32 +1168,42 @@ the file including them will be republished as well."
         (pstamp (org-publish-cache-get key))
         (org-inhibit-startup t)
         (visiting (find-buffer-visiting filename))
-        included-files-ctime buf)
-    (when (equal (file-name-extension filename) "org")
-      (setq buf (find-file (expand-file-name filename)))
+        (ctime (org-publish-cache-ctime-of-src filename))
+        (needsp (or (null pstamp) (< pstamp ctime)))
+        includes buf)
+    ;; if the file needs publishing, refresh the included-files cache property
+    (when (and needsp
+              (equal (file-name-extension filename) "org"))
+      (setq buf (find-file-noselect (expand-file-name filename)))
       (with-current-buffer buf
-       (goto-char (point-min))
-       (while (re-search-forward "^[ \t]*#\\+INCLUDE:" nil t)
-         (let* ((element (org-element-at-point))
-                (included-file
-                 (and (eq (org-element-type element) 'keyword)
-                      (let ((value (org-element-property :value element)))
-                        (and value
-                             (string-match "^\\(\".+?\"\\|\\S-+\\)" value)
-                             (org-remove-double-quotes
-                              (match-string 1 value)))))))
-           (when included-file
-             (add-to-list 'included-files-ctime
-                          (org-publish-cache-ctime-of-src
-                           (expand-file-name included-file))
-                          t)))))
-      (unless visiting (kill-buffer buf)))
-    (if (null pstamp) t
-      (let ((ctime (org-publish-cache-ctime-of-src filename)))
-       (or (< pstamp ctime)
-           (when included-files-ctime
-             (not (null (delq nil (mapcar (lambda (ct) (< ctime ct))
-                                          included-files-ctime))))))))))
+       (save-excursion
+         (goto-char (point-min))
+         (while (re-search-forward "^[ \t]*#\\+INCLUDE:" nil t)
+           (let* ((element (org-element-at-point))
+                  (included-file
+                   (and (eq (org-element-type element) 'keyword)
+                        (let ((value (org-element-property :value element)))
+                          (and value
+                               (string-match "^\\(\".+?\"\\|\\S-+\\)" value)
+                               (org-remove-double-quotes
+                                (match-string 1 value)))))))
+             (when included-file (add-to-list 'includes (expand-file-name 
included-file)))))))
+      (unless visiting (kill-buffer buf))
+      (when includes (org-publish-cache-set-file-property filename :includes 
includes)))
+    ;; return t if needsp or if included files have changed
+    (or needsp
+       (when (delq nil
+                   (mapcar (lambda (file)
+                             (let ((ct (org-publish-cache-ctime-of-src file)))
+                               (and (file-exists-p file)
+                                    (< ctime ct))))
+                           (org-publish-cache-get-file-property filename 
:includes)))
+         ;; update the timestamp of the published file, but only if buffer is 
not modified
+         (setq buf (find-file-noselect (expand-file-name filename)))
+         (with-current-buffer buf
+           (when (not (buffer-modified-p)) (set-buffer-modified-p t) 
(save-buffer)))
+         (unless visiting (kill-buffer buf))
+         t))))
 
 (defun org-publish-cache-set-file-property
   (filename property value &optional project-name)
-- 
2.0.4