Contracting for conformance test development
Dan Kohn <dan@...>
CNCF has been collaborating with SIG-Architecture and SIG-Testing over the last year to deploy the Kubernetes Software Conformance Program <https://www.cncf.io/certification/software-conformance/>, and there are now 49 certified vendors.
According to NEC's Omichi-san, Kubernetes has 481 APIs and tests currently cover 53, which is 11%. CNCF would like to invest in test development to get that number much higher, which should help ensure that the Certified Kubernetes brand is meaningful.
Philosophically, CNCF generally avoids hiring engineers because we don't want to compete with our members. We are planning to make an exception in this case to try to pay off some of the technical debt of having features in the stable API without corresponding conformance tests.
CNCF has contracted with Globant to do test development. We are envisioning paying them several hundred thousand dollars in 2018 for engineers to write tests and some project management. CNCF is open to adjust that and/or look at alternative firms, based on their performance. All tests will be submitted as PRs and approved by SIG testing as normal.
Before approving the appropriation, CNCF's governing board has requested that if it is going to fund the missing conformance tests, K8s should adopt a policy not to allow new features to be added unless they include conformance tests for their public API. More plainly, the governing board is open to having CNCF backfill missing confomance tests, but does not want to sign up to an unending amount of conformance test development going forward.
Brian Grant of SIG Architecture has proposed that "GA/stable APIs must have conformance tests" and Clayton Coleman LGTM'ed <https://github.com/kubernetes/community/pull/1806>. If others have views on this new policy, they should comment on that issue.
We're expecting to go forward at this time, unless there are concerns or objections. If you want to help, a careful review of the first several PRs that Globant submits (to ensure they get on the right track from the start) would be very useful.