Maintainership standards

[goldfirere]

I was cruising by and remembered some conversations I had had with @tomjaguarpaw about best practices in package maintainership. I wonder whether this group would be a good place to flesh out such standards and work to get them into part of the general Haskell community workflow. I'm opening this as an Issue, not a PR, because it's not all fully-formed enough in my head to write up a formal proposal. Yet the idea is well formed enough that I think writing something is better than nothing. Do feel free to just close this if this is too far from your immediate agenda and is unhelpful.

---

Goal: Have the community recognize some subset of packages as well maintained. These packages might be highlighted with a badge on Hackage, for example. The packages are then understood to be more likely to be reliable into the future, and more worthy of being the basis of, say, a commercial enterprise.

Non-goal: Getting every Haskell contributor to follow these standards or to mandate anything, anywhere. This is all opt-in.

Bronze-Standard Maintenance:

• The package lists a maintainer email address. Emails to this address are answered within 2 weeks. (Exceptions allowed around holidays.)
• The package lists a place to post bugs. Posts to this location receive some response within 2 weeks. (Exceptions allowed around holidays.)
• The package includes a license.
• The package includes a testsuite and CI infrastructure.
• The package builds with all (stable) versions of GHC released between 6 months ago and two years ago (among, perhaps, others). Exception: if the package uses features new in GHC since four years ago, it is appropriate to have a lower version bound on GHC version.

Silver-Standard Maintenance:

• All of the Bronze-Standard requirements.
• The package includes a changelog, detailing how it has evolved from release to release.
• The package is included in a Stackage LTS.
• For the past three releases of GHC (including minor releases), the package was up-to-date (the testsuite passed) within 1 month of GHC release.

Gold-Standard Maintenance:

• All of the Silver-Standard requirements.
• The package includes a backup maintainer (with email address), who has full technical credentials to take over the project.
• The package contains a takeover plan if the primary maintainer becomes unresponsive.
• The package's ticketing system contains a reference to a code of conduct or other structure to encourage professional, welcoming communication.
• The package's README or manual includes informative examples of how to use the package. (Not just type signatures, for example.) (Is this too subjective?)
• The package's testsuite includes performance tests.
• The package builds with all (stable) versions of GHC released between 1 month ago and four years ago (among, perhaps, others). Exception: if the package uses features new in GHC since four years ago, it is appropriate to have a lower version bound on GHC version.

Platinum-Standard Maintenance:

• All of the Gold-Standard requirements.
• The package includes a timestamped statement within the past year attesting the aim to meet the Platinum-Standard requirements.
• The testsuite is analyzed for code coverage and covers at least 95% of the code in the package.
• Issues reported against the package get triaged in at most 2 business days.

Certifying the standard to which a package adheres clearly takes some human intervention, but most of the tests above can be automated. A package would have to request getting the badge in order for a human to assess it. This kind of assessment might be something the HF could take on. I do think having these maintenance levels would be very useful to people deciding on what packages to use in their mission-critical project!

[bergmark]

I think this is relevant to the working group, it covers a lot of points that I had in mind for my yet-to-leave-my-brain proposal around maintenance. The majority of the listed requirements make sense to me.

I would not be surprised if we get a lot of bike-shedding around what the exact requirements should be. Do you need to meet every requirement? What if you are one checkbox away from leveling up to silver, but you meet several of the requirements for gold? Is it OK to disregard a requirement if you have a good reason? How do we evolve the requirements?

Some other requirements that I would consider:

• A major version must be supported and receive bug fixes and dependency bumps for X amount of time
• Have a migration guide and/or compatibilty packages (ala base-compat) to manage breaking changes. Also including changes that are not strictly PVP breaking (e.g. changing the format of serialization)

It seems like we could get something rolling with a reasonable amount of up-front effort.

I would like to have documentation on suggested ways to meet these requirements to make it easier for people to join. What's an easy way to set up CI? Is there a HF approved CoC that I can copy into my project?

Some notes on the proposed requirements:

The package lists a maintainer email address. Emails to this address are answered within 2 weeks. (Exceptions allowed around holidays.)

I prefer issue trackers, they are public and allow people who are not the maintainer to assist. Having either is fine in my book.

The package includes a license.

A FLOSS license, or can it be all rights reserved?

The package includes a testsuite

Does this test suite need to do anything? Is the intention to make it easy to add tests later on or should it cover some basic functionality?

The package is included in a Stackage LTS.

Here is one instance where I think we should accept a good excuse.

For the past three releases of GHC (including minor releases), the package was up-to-date (the testsuite passed) within 1 month of GHC release.

I think this would cause a lot of churn, as things stand now, GHC adoption takes a lot longer. And what if you depend on another silver package that does not update until a month has passed? I like the gist of this, but not sure how it should be formalized into a requirement.

The testsuite is analyzed for code coverage and covers at least 95% of the code in the package.

This is contentious for me, personally. We could get into the debate on usefulness of strict code coverage requirements. I haven't used it much in Haskell, is the tooling good enough where having e.g. doctests, regular tests, and system tests (running a separate application) can be combined into one coverage report (that's where I ran into trouble when trying to measure code coverage in Rust)?

I'm 👍 on the initative

What is the next step? Wait a while to see if we get more feedback here? Discuss it at the WG meeting? Formalize the proposal?

[goldfirere]

Do you need to meet every requirement? What if you are one checkbox away from leveling up to silver, but you meet several of the requirements for gold? Is it OK to disregard a requirement if you have a good reason? How do we evolve the requirements?

Good questions. Maybe we have a points system? Or no "standards", just lots of badges? I don't have good answers here, but that needn't stop us from proceeding. Maybe one way forward is to allow for a "maintainership statement" describing the particulars of an individual package. (Example, singletons meets most "Gold" requirements above, but we never aim to support more than one GHC release, for good reasons that we could document. So maybe we could apply for "Gold*", where the * means that we have an exception to one or more requirements, as documented in a statement.)

I would like to have documentation on suggested ways to meet these requirements to make it easier for people to join. What's an easy way to set up CI?

haskell-ci

Is there a HF approved CoC that I can copy into my project?

No, but there should be!

I prefer issue trackers, they are public and allow people who are not the maintainer to assist. Having either is fine in my book.

Agreed.

The package includes a license.

A FLOSS license, or can it be all rights reserved?

I'm personally fine with either. The goal in this initiative (for me) is to make it easier for folks to make decisions around adoption, and I don't see the status of the license (in this regard) to be a determining factor.

The package includes a testsuite

Does this test suite need to do anything? Is the intention to make it easy to add tests later on or should it cover some basic functionality?

Good points. This should be added onto.

The testsuite is analyzed for code coverage and covers at least 95% of the code in the package.

This is contentious for me, personally.

I just threw that in. I've never checked for code coverage, nor do I have any particular knowledge about whether code coverage checking is useful in practice. I was just thinking of ways of raising the bar. :)

I'm 👍 on the initative

Great. :)

[gbaz]

Note that hackage already requires a FLOSS license for upload.

Vis a vis "The package's testsuite includes performance tests" -- this is one area where technical work could be of use. Behavior tests are well supported between cabal and the wide ecosystem of testing packages and CI knows how to run them.

There's are libs that make performance testing straightforward. But there's no standard way to declare a performance test stanza and run it in a way that makes sense for performance tests with a library that integrates well with that spec and such that CI can "run performance tests" or the like (which is of course complicated by the fact that perf tests are necessarily unstable between different build and run environments, will vary widely on CI systems with shared load, etc). If we want to encourage people to include perf tests with their packages, we will need some sensible way of making this easy to do in a semi-standard way.

[hasufell]

We were just talking today about legal implications for open-source software and I proposed (jokingly) to add a liability clause to hackage.

Of course, that's not going to happen. But what would indeed be interesting is to have a "stability program" on hackage that signals end-users that this maintainer has agreed to e.g. respond to security bug reports quickly. That can be pretty important for decision making about which library to use in industry settings.

I'd even go so far to say that hackage should make it easy to fund those maintainers (e.g. via opencollective link or whatnot, or a HF maintained system). That could be incentive for maintainers to join the programme.

---

Wrt some details:

The package lists a place to post bugs

A public place. I had an argument with the maintainer of filepath-bytestring, because the only way to report bugs is their private email. I think this is insufficient.

The package includes a testsuite and CI infrastructure.

I disagree with CI infrastructure. It's totally fine for a maintainer to execute the testing manually. CI is about automation and reducing workload. If a maintainer chooses to do this work themselves, they should be free to do so.

The package is included in a Stackage LTS.

I disagree with that requirement. Getting your package into stackage is an entirely different matter and doesn't really signal commitment or quality. You could as well require people to get their software into debian/nixpkgs or similar. I don't think we should.

The package includes a backup maintainer (with email address), who has full technical credentials to take over the project.

I'm a little on the fence with this one. Even if you manage to find a backup maintainer, it's possible they don't actually have the time to take over the project when it becomes necessary. Instead we could require maintainers to allow faster temporary package take-overs through pre-existing hackage trustees workflow (e.g. if they don't respond within 2-4 weeks), unless they have put a backup maintainer in place.

The testsuite is analyzed for code coverage and covers at least 95% of the code in the package.

Uff 😅

[geekosaur]

I'm afraid to ask how that code coverage requirement applies to something like xmonad-contrib. (Most of which is not even testable without coming up with an xvfb-based test harness and injection and analysis of X events.)

I agree with requiring a public place to post bugs, but a private one should also be provided. You do not want to publicly advertise the 0day you just discovered.

[tomjaguarpaw]

I would caution people against taking Richard's suggestions of certification requirements too literally. I believe they were only meant as examples to start the discussions, not examples which he wants to move forwards on this proposal with.

That said, if we treat this as a brainstorming session it could be useful to collect agreements and disagreements with these examples To that end, I agree with most of @hasufell's disagreements, but I disagree with this disagreement:

The package includes a testsuite and CI infrastructure.

I disagree with CI infrastructure. It's totally fine for a maintainer to execute the testing manually. CI is about automation and reducing workload. If a maintainer chooses to do this work themselves, they should be free to do so.

To me the benefit of public CI is like the benefit of a public bug tracker. If the pass/fail status of the test suite is public it's much easier for everyone to know the status of the library.

[hasufell]

To me the benefit of public CI is like the benefit of a public bug tracker. If the pass/fail status of the test suite is public it's much easier for everyone to know the status of the library.

I'd argue that public CI can and should be handled on hackage. Currently the status is this: haskell/hackage-server#997

[geekosaur]

I'd argue that's asking a lot of hackage, to be honest. CI should be part of the project, and provisioned however that project wishes to do it.

[hasufell]

I think a compromise would be for a maintainer to publish test logs (including the commit/version it was run against) if they don't wish to install CI. I think that is just as good.

[goldfirere]

I would caution people against taking Richard's suggestions of certification requirements too literally. I believe they were only meant as examples to start the discussions, not examples with which he wants to move forwards on this proposal with.

Yes! The code coverage comment, in particular, was meant to be provocative, encouraging some thought outside of our usual. I have no experience or relevant training to suggest that this, in particular, is a good idea.

About CI in particular: I would want reproducible, public evidence that the testsuite passes. I think CI is the best way of producing and maintaining this evidence. But, as @hasufell suggests, some kind of manually updated public log would, I suppose, meet the requirement. From a community administration point-of-view, I think it would be easier to require CI and note that exceptions can be granted. Having wide standards with lots of ways of satisfying them can be confusing -- having a specific request is easier to understand and for the community to administer. (It may be harder for an individual contributor in a particular circumstance, which is why there should be a route for asking for exceptions.)

[hasufell]

How would we launch this? I feel waiting for proper hackage support may be an indefinite blocker. HF could maybe maintain such a list internally and publish it somewhere on HF site?

[goldfirere]

I think a v0 launch as just a page on our site is reasonable. There is certainly room for improvement, but such a page would already bring value. Someone would still have to come up with the set of standards and then design a process for packages to apply and then be vetted. Once that starts happening, there will be more incentive for Hackage to catch up.

[telser]

I would caution people against taking Richard's suggestions of certification requirements too literally. I believe they were only meant as examples to start the discussions, not examples with which he wants to move forwards on this proposal with.

Yes! The code coverage comment, in particular, was meant to be provocative, encouraging some thought outside of our usual. I have no experience or relevant training to suggest that this, in particular, is a good idea.

There are certainly challenges here and the xmonad example is a great highlight. I can imagine there being value in being able to say a particular package has a "good" test suite, but the definition of "good" seems highly variable.

About CI in particular: I would want reproducible, public evidence that the testsuite passes. I think CI is the best way of producing and maintaining this evidence. But, as @hasufell suggests, some kind of manually updated public log would, I suppose, meet the requirement. From a community administration point-of-view, I think it would be easier to require CI and note that exceptions can be granted. Having wide standards with lots of ways of satisfying them can be confusing -- having a specific request is easier to understand and for the community to administer. (It may be harder for an individual contributor in a particular circumstance, which is why there should be a route for asking for exceptions.)

I would go even further, personally, and say multiplatform CI as the default so we avoid some of the odd breakages that happen from time to time on different systems. Also, I'd be strongly in favor of the ability to get an exception.

@goldfirere We brought this up at the last stability meeting and concluded that bringing the discussion to the wider community would be valuable. Would you mind posting this to Discourse for further discussion?

[hasufell]

I would go even further, personally, and say multiplatform CI as the default so we avoid some of the odd breakages that happen from time to time on different systems.

For such demands, the HF should also have technical documentation on how to achieve that. Since I've done a lot of those myself wrt ghcup, HLS, cabal... I can safely say it isn't trivial (e.g. testing aarch64 M1).

Would you mind posting this to Discourse for further discussion?

I'm a bit worried it will end up in bikeshedding, as we partly have here (constructively). I'd propose instead to flesh out the details of the tiers beforehand.

[telser]

I would go even further, personally, and say multiplatform CI as the default so we avoid some of the odd breakages that happen from time to time on different systems.

For such demands, the HF should also have technical documentation on how to achieve that. Since I've done a lot of those myself wrt ghcup, HLS, cabal... I can safely say it isn't trivial (e.g. testing aarch64 M1).

I agree, and I would love to see the hackage CI be able to offer that service to the community. My thought here is if some of the initial pain can be abstracted away we could get more consistent testing and hopefully better end user experiences.

Would you mind posting this to Discourse for further discussion?

I'm a bit worried it will end up in bikeshedding, as we partly have here. I'd propose instead to flesh out the details of the tiers beforehand.

To play devils advocate a bit, might we not get a similar kind of push back with the tiers anyway?