[Gluster-devel] Automated regression tests now part of development work flow

[Anand Avati]

Hello all,

The recent ongoing changes to the development work flow are now complete. The highlight of the rework is the automated regression tests.

So far we have only had fixed smoke tests that were run against every patch pre-commit (dbench, POSIX compliance). Here after, every change submitted to Gerrit must be accompanied with test case scripts which (attempts to) prove the correctness of the code change - as part of the new regression test in the work flow.

These test cases will be much more concentrated and focused on the changes brought in by the patch itself. The test cases, once committed will be part of every future pre-commit test. This will make sure no commit in the future will accidentally break or change the current commit either directly or with side effects. If a future change must change the behavior of the current patch, then the future patch must include corresponding changes to the test cases as well. This policy will keep code and test cases in sync.

The wiring and framework for supporting this are already available in glusterfs.git and Jenkins. Currently the infrastructure supports test cases limited to single node. Most of the test cases can actually be implemented in single node by having multiple instances of bricks and client mounts. Support of multi node test cases will be added soon.

The automated smoke test's voting power in Gerrit is now downgraded to +0 Verified for PASS and -1 Verified for FAILURE. Regression test's voting powers are now +1 Verified and -1 Verified. This means just passing the smoke test can no more be a qualifier for verification. For those patches for which a regression test is required but currently not automatable into a test script, must be tested and voted manually for the Verification vote.

As the above described changes are already in effect, most of the currently submitted patches under review will have to be resubmitted with test cases. This is necessary to make the changes start being effective immediately. If you have currently

Note that this opens up a new avenue for contributors in the community. You can now contribute test cases. These contributions need not include source code changes but just extend our regression test set. These test cases can either be for coverage of old code and functionality, or for coverage of your use cases which could be applicable to others as well.

The workflow document is also updated at http://www.gluster.org/community/documentation/index.php/Development_Work_Flow. Specifically:

Sec 1.4 http://www.gluster.org/community/documentation/index.php/Development_Work_Flow#Commit_policy

Sec 1.9 http://www.gluster.org/community/documentation/index.php/Development_Work_Flow#Regression_tests_and_test_cases

More details on how to write test cases with examples and how the framework works can be found at -

1. "tests: pre-commit regression tests" https://github.com/gluster/glusterfs/commit/bb41c8ab88f1a3d8c54b635674d0a72133623496

2. tests/README in glusterfs.git

3. Example new change - http://review.gluster.org/4114

Please offer feedback on how the framework can be improved (inclusion of more tools?) to capture test cases better.

Happy Hacking!

Avati