Script to generate ChangeLogs automatically

[Siddhesh Poyarekar]

Hello,

There is a general consensus among the glibc community to stop 
maintaining a ChangeLog file and we were told that the requirement for 
doing that was to have a script that provided a representation of the 
git log that looks similar to a ChangeLog.

I have attempted to write such a tool, please see attached script.  The 
script does the following:

1. Given a start and end git ref, it generates a list of refs to build 
ChangeLog entries for

2. For each ref, parse the raw log of the change and identify files 
added, removed, renamed and modified

3. For C sources that are modified, the script then generates parse 
trees of the old and new version of the file and attempts to find 
differences between them at a top level token level.

So for the following diff:

~~~~~~~~~~~~~~~~~~~
diff --git a/ChangeLog b/ChangeLog
index c724d7dadc..20d61ee5aa 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,10 @@
+2018-08-01  H.J. Lu  <address@hidden>
+
+       * sysdeps/x86/cpu-features.c (get_common_indeces): Renamed to
+       ...
+       (get_common_indices): This.
+       (init_cpu_features): Updated.
+
 2018-08-01  Joseph Myers  <address@hidden>

        * sysdeps/generic/math-tests-snan-payload.h: New file.
diff --git a/sysdeps/x86/cpu-features.c b/sysdeps/x86/cpu-features.c
index ea0b64fdb9..51642f8b6a 100644
--- a/sysdeps/x86/cpu-features.c
+++ b/sysdeps/x86/cpu-features.c
@@ -57,7 +57,7 @@ get_extended_indices (struct cpu_features *cpu_features)
 }

 static void
-get_common_indeces (struct cpu_features *cpu_features,
+get_common_indices (struct cpu_features *cpu_features,
                    unsigned int *family, unsigned int *model,
                    unsigned int *extended_model, unsigned int *stepping)
 {
@@ -235,7 +235,7 @@ init_cpu_features (struct cpu_features *cpu_features)

       kind = arch_kind_intel;

-      get_common_indeces (cpu_features, &family, &model, &extended_model,
+      get_common_indices (cpu_features, &family, &model, &extended_model,
                          &stepping);

       get_extended_indices (cpu_features);
@@ -351,7 +351,7 @@ init_cpu_features (struct cpu_features *cpu_features)

       kind = arch_kind_amd;

-      get_common_indeces (cpu_features, &family, &model, &extended_model,
+      get_common_indices (cpu_features, &family, &model, &extended_model,
                          &stepping);

       get_extended_indices (cpu_features);
~~~~~~~~~~~

the tool generates the following ChangeLog entry:

2018-08-01  H.J. Lu  <address@hidden>

        COMMIT: 82c80ac2ebf9acc81ec460adfd951d4884836c7c
        * sysdeps/x86/cpu-features.c: Modified.
        (get_common_indeces): Removed.
        (init_cpu_features): Modified.
        (get_common_indices): New.

It is not exactly what one would write, but is immediately recognizable.

Capabilities
------------

The tool can currently detect the following kinds of changes in C sources:

- Top level objects like static variables, function declarations and 
definitions, structs and unions and assignments.

- Nesting within macro conditions.  It's not perfect, but surprisingly 
decent.

Limitations
-----------

These are things the tool cannot do yet, but could be improved iteratively.

- Detect changes to comments.  It's just dropping all comments right 
now, so changes that are comment-only end up seeming identical.

- Changes within function definitions, struct members.  The script only 
knows that it changed, not what changed inside.

- Weird macro nesting will break things.  I tried to adjust for some 
weirdness that I encountered in my testing, but I am confident that an 
'innovative' C programmer can break this.  As of now, I've verified 
changes between the glibc-2.26 to 2.28 range and hacked around the 
nesting oddities I noticed.

Once this script is deemed acceptable, I would like to propose dropping 
the ChangeLog requirement for the glibc project and add this script 
instead to aid those who would like to see a ChangeLog-like output from 
the git changes.  Adding the script to the glibc tree would have the 
added advantage of in-tree improvement of the script so that it gets 
more capabilities in future and allow it to become more accurate over time.

Siddhesh

gen-changed-entities.py
Description: Text Data