Re: Guix Survey (follow up on "How can we decrease the cognitive overhead for contributors?")

[Katherine Cox-Buday]

This is awesome, thanks Wilko!

I've been talking with my wife who is the field of psychology. As part 
of her degree, and as part of her ongoing education, she's studied how 
to design studies/surveys so that they're not biased and don't produce 
contaminated data. She's not an expert at this, but she knows more about 
it than me :)

The things we could come up with which we thought were important to 
consider are:

- You must first define your goals for the survey. Is it meant to see
  who is using Guix? Who is contributing? How they find the
  contribution process? How they find using Guix? There are many
  dimensions, and we may need to create more than one survey.

- The medium of the survey is very important. E.g. some people won't
  reply to a survey served something that uses JavaScript. Some people
  may not be able to reply to a survey unless accessibility concerns
  are met. Some people won't overcome the barrier of having to log in
  to respond.

- Where solicitations to complete the survey are broadcast is very
  important. E.g. if we only send it to guix-dev, this skews the
  responses to questions like "where do you talk about Guix".

- When the solicitations are made is very important. Some religions do
  not allow use of electronics on certain days, or times of the year.
  Some people are away on holiday during parts of the year. Some
  people are trying to meet deadlines during fiscal quarters. It may
  be impossible to accommodate everyone, but giving a little
  consideration to the issue and a sufficient window of time may cover
  most cases.

- When soliciting responses to the survey, it's very important to set
  expectations about the survey in the solicitation. It is important
  to briefly describe what the survey is like and how long the survey
  will take. Without this, some people will have uncertainty about
  what they're committing to and not even try.

- The survey should endeavor to remain on the shorter end; many will
  not complete longer surveys.

- Does the survey need translation to eliminate language barriers?

- The survey should use a uniform measurement system throughout. Don't
  use scales with different magnitudes in different questions, and
  don't suddenly invert whether higher is better or worse.

- As you've already mentioned, free-form questions are very difficult
  to quantify, and I think we should use them with caution.
  Communities rooted in philosophical values, as Guix is, have
  impassioned people and resolving a large number of free-form
  responses to a quantitative statement may be difficult.

- Questions which are intended to solicit a agree/disagree should be
  phrased as "I" questions, e.g.:

        On a scale of 1-5, how much do you agree with the phrase "I
        like carrots"?

- Questions should not be leading, and be biased towards the positive.
  E.g., with the carrots example, don't do this:

       Carrots are disgusting. How much do you agree with this?

  and don't do this:

        On a scale of 1-5, how much do you agree with the phrase "I
        think carrots are disgusting!"

- Up front, it may be difficult to identify all the root-causes of
  something the project wants to know about. Instead of trying to
  infer these, ask the questions directly. E.g. instead of questions
  about liking crunchy vegetables, orange vegetables, and root
  vegetables, ask whether they like carrots.

  However, if you think you have some idea of the root-causes, you can
  ask those as well to see if the correlation you think exists does.

- You may want to ensure the survey has "marker questions" which
  clearly categorize your responder for you to make it easier to make
  the statements you'd like to make. E.g. if you're interested in
  analyzing what vegetarians vs omnivores think of carrots, ask that
  so you don't have to try and infer it later.

- We were unable to resolve the question of astroturfing wherein one
  malicious party responds many times to skew the data. This might be
  difficult to address without relying on a vendor who has solved this
  concern somehow, but requires logins, JavaScript, or something else
  people won't use.

And finally, I'd like to suggest:

I think since this is the result of a discussion about how to lower the 
cognitive overhead of contributing, the goal of this initial survey 
should be:

1. To quantify how easy it is to contribute to Guix.
2. To quantify how easy it is to maintain Guix.
3. To correlate (1) and (2) with people's opinion of using email for 
contributions.
4. To correlate (1) and (2) with people's opinion of using a forge for 
contributions.
5. To correlate (1) and (2) with people's opinion on only improving tooling.
6. To be able to do trend-analysis year-over-year on these issues.

I would suggest adding these questions to a survey exploring the 
contribution process:

    On a scale of 1-5, 1 being "Strongly Disagree" and 5 being
    "Strongly Agree", how much do you agree with the statements:

    "I think it is easy to contribute to Guix."

    "I think contributions to Guix will be reviewed in a timely
    manner."

    "I think email is the best way to manage introducing code to
    Guix."

    "I think a web-forge is the best way to manage introducing code to
    Guix."

    "I think working on tools made specifically for Guix is the best
    way to improve the contribution process."

I am very interested in the usage patterns of Guix, and I firmly believe 
some survey should explore this. I'm not sure if we should combine the 
two; does it make it too long of a survey?