CNCF is gearing up to make a significant set of investments into Continuous Integration (CI) in 2017. I wanted to lay out some of our goals as well as what we are not trying to achieve. Feedback and updates on these are welcome. In general, we're looking for discussions to occur on cncf-ci-public@... so please join that list if you are interested in the subject.

Non-goals

All CNCF projects already make extensive use of CI. We are not aiming to replace their existing CI, or require them to use our platform, or force them to follow specific standards or processes. CI is a fast-moving ecosystem and project maintainers are in the best position to adjust intra-project CI based on their own requirements.

We don’t want to build our own testing framework or harness. There are tons of perfectly adequate ones. We want to integrate existing tools to provide useful results to our projects and our community, not develop a new testing project from scratch.

Goals

We want any testing work we fund to integrate continuously. Ideally, that means running on every commit via web hook. If technology or economic reasons dictate, it could instead run daily or other periodic basis. No manual setup should be needed.

We need any work we fund to be open source. It’s acceptable for it too be built on top of closed source platforms like CircleCI, TravisCI, AWS, etc., but if CNCF funding is paying for code to be written, we need that code to be open source. That allows it to be reviewed, improved, iterated on, etc. by others, especially by developers from individual projects.

All test frameworks eventually have problems of false positives, false negatives, and flapping (i.e., intermittent failures). We want to use best practices to document and minimize these.

The GitHub Status API is a de facto standard for third-party CI services to integrate with GitHub and other services. Any work we fund should (eventually) implement the GitHub Status API, and be integrated into relevant projects as long as false positives are at a sufficiently low level.

CNCF is helping develop a cloud native software stack that enables cross-cloud deployments. Cross-project CI that ensures ongoing interoperability is especially valuable. We’re also interested in a dashboard that shows the cross-project interoperability status of both the latest released version and HEAD of each CNCF project. We would also like to ensure that CNCF projects are able to deploy successfully on to most (and ideally all) public clouds as well as bare metal private clouds and common infrastructure like OpenStack and VMWare.