Re: Assistance testing Harbor for OCI conformance

Yan Wang

HI josh,


Did you use the latest conformance testing code? The csrf issue has been fixed by my PR


Here is my results:


06:16:27 Ran 27 of 54 Specs in 1.671 seconds

442506:16:27 FAIL! -- 24 Passed | 3 Failed | 0 Pending | 27 Skipped





From: <harbor-dev@...> on behalf of "Josh Dolitsky via" <>
Reply-To: "harbor-dev@..." <harbor-dev@...>
Date: Friday, 1 May 2020 at 6:49 AM
To: "harbor-dev@..." <harbor-dev@...>
Cc: Daniel Jiang <jiangd@...>, Alex Xu <xalex@...>
Subject: Re: [harbor-dev] Assistance testing Harbor for OCI conformance


I started setting up some workflows testing against the live site. Seeing a CSRF protection error that we haven't seen with the other registries:

{ "errors": [ { "code": "FORBIDDEN", "message": "CSRF token invalid" } ] }

Here is a URL with the results table now including Harbor:
Here is a sample report with the failure (expand first red section):

Do we just need to pass along the cookie returned from the token endpoint? (GET /service/token?scope=...). The code used for the raw HTTP requests which we will need to fix can be found here:




On Wed, Apr 29, 2020 at 11:59 AM Josh Dolitsky via <> wrote:

I'm happy to see it being tested on the Harbor side. This dashboard of live results is nice to have, but the real end goal is for registries to submit static test results (report.html etc.) as well as a README for how to reproduce as a PR against this repo: (see instructions)

This is based on the self-certification model used by Kubernetes:

You'll see in the k8s-conformance repo several directories for different k8s versions, within static test results generated by running Heptio Sonobuoy.

In the case of the oci-conformance repo, imagine directories for each release of the distribution-spec (currently pre-1.0). The conformance test suite is the equivalent of Sonobuoy in this scenario.

Feel free to reach out directly for clarification.




On Wed, Apr 29, 2020 at 10:32 AM Yan Wang <wangyan@...> wrote:

For the live testing, I have opened a PR at We’d like to enable the live testing as Harbor repo so that we can know which PR introduces new failures and fix it on time.


@Daniel Jiang @Alex Xu  your idea?





From: <harbor-dev@...> on behalf of "Josh Dolitsky via" <>
Reply-To: "harbor-dev@..." <harbor-dev@...>
Date: Wednesday, 29 April 2020 at 11:02 PM
To: "harbor-dev@..." <harbor-dev@...>
Subject: Re: [harbor-dev] Assistance testing Harbor for OCI conformance


Yan, thanks for helping with this. Do you know if it is possible to start a fresh new Harbor instance in GitHub actions? GitHub actions allows for a "services" section which you can launch containers for the test run. This is what we really need help from Harbor team on. Here is a example action using a redis:

Since Harbor is an opensource registry, it would be preferred to do it this way vs. testing a live version (i.e. This is definitely a great start regardless.

Thank you,





On Wed, Apr 29, 2020 at 4:41 AM Yan Wang <wangyan@...> wrote:

Hi Josh,


Thanks for your detailed info.


I have filed an PR to add Harbor conformance test, could you please help to review?

BTW, can you help to add two more secrets(Harbor_USERNAME and Harbor_PASSWORD) to store the testing account auth?





From: <harbor-dev@...> on behalf of "Daniel Jiang via" <>
Reply-To: "harbor-dev@..." <harbor-dev@...>
Date: Tuesday, 7 April 2020 at 2:05 AM
To: "harbor-dev@..." <harbor-dev@...>
Subject: Re: [harbor-dev] Assistance testing Harbor for OCI conformance




Thanks for the heads up.  The conformance test has been on our radar and we did some manual dry run locally.


Adding Harbor to the report is on the todo list, and we’ll have that incorporated within this or next week.


Best Regards


Daniel Jiang  | 姜坦

Engineer, VMware R&D, Beijing

+86 10-59934536



On 2020/4/6, 10:21 AM, "harbor-dev@... on behalf of Josh Dolitsky via" <harbor-dev@... on behalf of> wrote:


Hello Harbor team,

We're seeking help setting up test automation using GitHub actions to test Harbor for OCI conformance. See full issue here.

For background: the OCI Distribution Specification is an open spec based on Docker's Registry API, and is the current workspace for defining standards concerning container registries such as Harbor. There is a recent effort to enable registry providers to validate if a registry properly conforms to the specification. 


We need your expertise in setting up Harbor for tests purposes within the context of GitHub actions, as well as configuration setup for running the OCI conformance test suite.

You can view a live view of preliminary test results at Click one of the "test report" links to open up a detailed conformance report.

If you're interested in the larger discussion, I encourage you to join the the OCI mailing list.

Feel free to email me or ping me on Slack if you have any questions etc.


- Josh Dolitsky (@jdolitsky)

Join to automatically receive all group messages.