This is a thought I got for the lv2 crate after some rustfmt shenenigans we got, and I realized that besides Travis I know next to nothing about GitHub CI apps, so I’m asking for some advice here (mainly about what to do for lv2, but we could also add stuff for the other projects).
In my opinion, a good CI pipeline should at least provide these:
Building and Testing on at least the three major platforms (Win, Mac, Linux), and enforcing that all tests pass.
Gather test coverage from the tests, and provide those detailed results (not necessarily enforced, but I think we should keep an eye on it shall it decrease suddenly).
Check with and enforce Clippy
Enforce formatting using Rustfmt (it may not be on its defaults, but it should be consistent across the repo, and we should not have to worry about it)
Check for outdated and/or vulnerability-affected dependencies.
… and maybe more, if you have ideas
What do you think are the best tools to achieve this?
I’ve experimented with these topics too and can already say some words about some of your points:
Regarding the first two, maybe three points, I think that Travis CI might be a good tool. I’ve already established automated builds and tests with (Windows, Mac, Linux) x (stable, beta, nightly) for lv2rs. However, I’ve only done building and testing, no clippy or rustfmt checks, you would need to adapt the scripts for that.
However: As far as I know, Travis only checks the code, it doesn’t really modify the code and creates commits. Therefore, enforcing rustfmt may only work by checking if rustfmt changes the code or not.
I also don’t think that dependency checks should be done automatically: Dependencies change rarely and therefore could be handled by humans. Also, how would you define “outdated”? If a finished crate has a quite small scope and therefore doesn’t need updates, it can’t “outdate”, can it?
Travis does indeed look like the obvious choice for building and testing, though I’ve been wondering about some deeper integration for things like code coverage.
I’ve heard about Codecov or Coveralls, I don’t know what those specific services are worth but I think those kind of services may be useful if we want to collaboratively track code coverage, and more importantly check regressions from Pull Requests.
Oh CI shouldn’t ever change the code, by “enforcing” I meant failing the build if rustfmt style is not enforced correctly. And it already has support to do exactly this.
In general, I think that the less repetitive work humans have to do, the better.
While automatically keeping an eye on outdated dependencies (i.e. a new version is released) can be interesting, I indeed think that running cargo outdated from time to time (mainly before publishing to crates.io) is good enough.
The big issue however is security vulnerabilities. Those just crop up whenever reported (CVE or RustSec, or others), and need to be dealt with ASAP. We could just add cargo-audit in the CI file, but there also seem to be tools like Dependabot that can check for vulnerabilities without having to trigger a build, and which creates a Pull Request automatically.
Good CI pipelines are immensely useful, from automated tests to linting and formatting checks and more.
The tools mentioned by others (Dependabot, Codecov / Coveralls, etc) can be very useful, to those too. A lot of the commercial services offer them for free for open-source projects!
About the CI services themselves, for anecdotal experience, in the past I’ve used Travis, CircleCI, GitLab CI, and all of them did their job pretty fine. However, I don’t have experience from setting up CI projects for something that would target all three major OS platforms at once
(One tiny detail to keep in mind is that Apple is pretty strict in their licensing regarding where you can run macOS - only on genuine Apple hardware - so the services providing macOS builds have to have literal macbook laptops, iMac desktops, or mac pros mounted to racks. This can be a bit less cost efficient than providing builds for other platforms, and may be visible in CI pricing…)
I wonder if it would be okay to be inspired / borrow / steal from CI&CD configurations made by other successful projects, see e.g. https://github.com/sharkdp/bat/
I also seem to recall reading some chat about the infra team of Rust project itself also thinking about improving their CI pipelines in the near future, wonder if there would be any lessons to be learned and heard from those people
I’ve set up a Travis CI configuration for rust-lv2 that uses all three operating systems and all three versions of Rust in a build matrix. I don’t think that it’s perfect and there is still room for optimization, but it builds the code, runs the tests and checks the style. It basically does what you asked for!