Re: [PATCH] org-test: Create a collaborative test set for Org buffer parser

[Max Nikulin]

On 11/12/2021 21:39, Ihor Radchenko wrote:
> The attached patch is an attempt to create something like shared repo
> for Org element parser tests.

"[PATCH]" prefix in the subject might be a reason why you message 
received less attention than it should.

> The test set is essentially a series of .org files alongside .el files
> containing normalised output of `org-element-parse-buffer'. (see the
> patch)

I think, the set should contain hundreds of tests to be helpful, thus 2 
files per test will likely be inconvenient since most of samples will be 
short. I suggest to group test input and results into large files. Such 
tests should be augmented by some metadata: keywords (labels, tags). 
E.g. besides heading

- sample: "* Simple Heading"
  keywords: heading

it should be possible to filter related cases with similar markup

- sample: "*Bold* emphasis"
  keywords: emphasis, heading
  description: Despite line is started from a star,
    there is no space after it, so it is not a heading.

- sample: " * Unordered list item"
  keywords: list, heading
  description: Due to leading space it is a list item,
    not a heading.

- sample: "*"
  keywords: text, heading
  description: Not a heading since there is no space after the star.

I omitted test IDs above.

Version of Org and test set should be included into metadata for the 
whole suite.

Since partial compliance is assumed, format of test results should be 
declared as well to be able to publish overview or comparison.

Are properties like :begin and :end mandatory for reference results of 
parsing? They make structures more verbose and harder to read. Often it 
is enough to compare content and similar properties.

> Any comments or suggestions?
> I am particularly looking for thoughts about licensing and possible
> distribution of the test set in separate repository.

Since these tests will unlikely become a part of some software, I do not 
think that GPL may be an obstacle for any project. Requirement of signed 
consent will likely prevent contributing of new cases from some developers.