Hi Josh,

 

Thanks for your detailed info.

 

I have filed an PR to add Harbor conformance test, could you please help to review? https://github.com/bloodorangeio/oci-distribution-conformance-results/pull/10

BTW, can you help to add two more secrets(Harbor_USERNAME and Harbor_PASSWORD) to store the testing account auth?

 

Thanks,

Yan

 

From: <harbor-dev@...> on behalf of "Daniel Jiang via lists.cncf.io" <jiangd=vmware.com@...>
Reply-To: "harbor-dev@..." <harbor-dev@...>
Date: Tuesday, 7 April 2020 at 2:05 AM
To: "harbor-dev@..." <harbor-dev@...>
Subject: Re: [harbor-dev] Assistance testing Harbor for OCI conformance

 

Josh,

 

Thanks for the heads up.  The conformance test has been on our radar and we did some manual dry run locally.

 

Adding Harbor to the report is on the todo list, and we’ll have that incorporated within this or next week.

 

Best Regards

— 

Daniel Jiang  | 姜坦

Engineer, VMware R&D, Beijing

+86 10-59934536

 

 

On 2020/4/6, 10:21 AM, "harbor-dev@... on behalf of Josh Dolitsky via lists.cncf.io" <harbor-dev@... on behalf of jdolitsky=gmail.com@...> wrote:

 

Hello Harbor team,

We're seeking help setting up test automation using GitHub actions to test Harbor for OCI conformance. See full issue here.

For background: the OCI Distribution Specification is an open spec based on Docker's Registry API, and is the current workspace for defining standards concerning container registries such as Harbor. There is a recent effort to enable registry providers to validate if a registry properly conforms to the specification. 

 

We need your expertise in setting up Harbor for tests purposes within the context of GitHub actions, as well as configuration setup for running the OCI conformance test suite.

You can view a live view of preliminary test results at https://oci.bloodorange.io. Click one of the "test report" links to open up a detailed conformance report.

If you're interested in the larger discussion, I encourage you to join the the OCI mailing list.

Feel free to email me or ping me on Slack if you have any questions etc.

 

- Josh Dolitsky (@jdolitsky)

For the live testing, I have opened a PR at https://github.com/goharbor/harbor/pull/11801. We’d like to enable the live testing as Harbor repo so that we can know which PR introduces new failures and fix it on time.

 

@Daniel Jiang @Alex Xu  your idea?

 

Thanks,

Yan

 

From: <harbor-dev@...> on behalf of "Josh Dolitsky via lists.cncf.io" <jdolitsky=gmail.com@...>
Reply-To: "harbor-dev@..." <harbor-dev@...>
Date: Wednesday, 29 April 2020 at 11:02 PM
To: "harbor-dev@..." <harbor-dev@...>
Subject: Re: [harbor-dev] Assistance testing Harbor for OCI conformance

 

Yan, thanks for helping with this. Do you know if it is possible to start a fresh new Harbor instance in GitHub actions? GitHub actions allows for a "services" section which you can launch containers for the test run. This is what we really need help from Harbor team on. Here is a example action using a redis: https://github.com/actions/example-services/blob/master/.github/workflows/redis-service.yml#L18

Since Harbor is an opensource registry, it would be preferred to do it this way vs. testing a live version (i.e. demo.goharbor.io). This is definitely a great start regardless.

Thank you,

 

Josh

 

 

On Wed, Apr 29, 2020 at 4:41 AM Yan Wang <wangyan@...> wrote:

Hi Josh,

 

Thanks for your detailed info.

 

I have filed an PR to add Harbor conformance test, could you please help to review? https://github.com/bloodorangeio/oci-distribution-conformance-results/pull/10

BTW, can you help to add two more secrets(Harbor_USERNAME and Harbor_PASSWORD) to store the testing account auth?

 

Thanks,

Yan

 

From: <harbor-dev@...> on behalf of "Daniel Jiang via lists.cncf.io" <jiangd=vmware.com@...>
Reply-To: "harbor-dev@..." <harbor-dev@...>
Date: Tuesday, 7 April 2020 at 2:05 AM
To: "harbor-dev@..." <harbor-dev@...>
Subject: Re: [harbor-dev] Assistance testing Harbor for OCI conformance

 

Josh,

 

Thanks for the heads up.  The conformance test has been on our radar and we did some manual dry run locally.

 

Adding Harbor to the report is on the todo list, and we’ll have that incorporated within this or next week.

 

Best Regards

— 

Daniel Jiang  | 姜坦

Engineer, VMware R&D, Beijing

+86 10-59934536

 

 

On 2020/4/6, 10:21 AM, "harbor-dev@... on behalf of Josh Dolitsky via lists.cncf.io" <harbor-dev@... on behalf of jdolitsky=gmail.com@...> wrote:

 

Hello Harbor team,

We're seeking help setting up test automation using GitHub actions to test Harbor for OCI conformance. See full issue here.

For background: the OCI Distribution Specification is an open spec based on Docker's Registry API, and is the current workspace for defining standards concerning container registries such as Harbor. There is a recent effort to enable registry providers to validate if a registry properly conforms to the specification. 

 

We need your expertise in setting up Harbor for tests purposes within the context of GitHub actions, as well as configuration setup for running the OCI conformance test suite.

You can view a live view of preliminary test results at https://oci.bloodorange.io. Click one of the "test report" links to open up a detailed conformance report.

If you're interested in the larger discussion, I encourage you to join the the OCI mailing list.

Feel free to email me or ping me on Slack if you have any questions etc.

 

- Josh Dolitsky (@jdolitsky)

I'm happy to see it being tested on the Harbor side. This dashboard of live results is nice to have, but the real end goal is for registries to submit static test results (report.html etc.) as well as a README for how to reproduce as a PR against this repo: https://github.com/opencontainers/oci-conformance/blob/master/instructions.md (see instructions)

This is based on the self-certification model used by Kubernetes: https://github.com/cncf/k8s-conformance

You'll see in the k8s-conformance repo several directories for different k8s versions, within static test results generated by running Heptio Sonobuoy.

In the case of the oci-conformance repo, imagine directories for each release of the distribution-spec (currently pre-1.0). The conformance test suite is the equivalent of Sonobuoy in this scenario.

Feel free to reach out directly for clarification.

Josh

On Wed, Apr 29, 2020 at 10:32 AM Yan Wang <wangyan@...> wrote:

For the live testing, I have opened a PR at https://github.com/goharbor/harbor/pull/11801. We’d like to enable the live testing as Harbor repo so that we can know which PR introduces new failures and fix it on time.

 

@Daniel Jiang @Alex Xu  your idea?

 

Thanks,

Yan

 

From: <harbor-dev@...> on behalf of "Josh Dolitsky via lists.cncf.io" <jdolitsky=gmail.com@...>
Reply-To: "harbor-dev@..." <harbor-dev@...>
Date: Wednesday, 29 April 2020 at 11:02 PM
To: "harbor-dev@..." <harbor-dev@...>
Subject: Re: [harbor-dev] Assistance testing Harbor for OCI conformance

 

Yan, thanks for helping with this. Do you know if it is possible to start a fresh new Harbor instance in GitHub actions? GitHub actions allows for a "services" section which you can launch containers for the test run. This is what we really need help from Harbor team on. Here is a example action using a redis: https://github.com/actions/example-services/blob/master/.github/workflows/redis-service.yml#L18

Since Harbor is an opensource registry, it would be preferred to do it this way vs. testing a live version (i.e. demo.goharbor.io). This is definitely a great start regardless.

Thank you,

 

Josh

 

 

On Wed, Apr 29, 2020 at 4:41 AM Yan Wang <wangyan@...> wrote:

Hi Josh,

 

Thanks for your detailed info.

 

I have filed an PR to add Harbor conformance test, could you please help to review? https://github.com/bloodorangeio/oci-distribution-conformance-results/pull/10

BTW, can you help to add two more secrets(Harbor_USERNAME and Harbor_PASSWORD) to store the testing account auth?

 

Thanks,

Yan

 

From: <harbor-dev@...> on behalf of "Daniel Jiang via lists.cncf.io" <jiangd=vmware.com@...>
Reply-To: "harbor-dev@..." <harbor-dev@...>
Date: Tuesday, 7 April 2020 at 2:05 AM
To: "harbor-dev@..." <harbor-dev@...>
Subject: Re: [harbor-dev] Assistance testing Harbor for OCI conformance

 

Josh,

 

Thanks for the heads up.  The conformance test has been on our radar and we did some manual dry run locally.

 

Adding Harbor to the report is on the todo list, and we’ll have that incorporated within this or next week.

 

Best Regards

— 

Daniel Jiang  | 姜坦

Engineer, VMware R&D, Beijing

+86 10-59934536

 

 

On 2020/4/6, 10:21 AM, "harbor-dev@... on behalf of Josh Dolitsky via lists.cncf.io" <harbor-dev@... on behalf of jdolitsky=gmail.com@...> wrote:

 

Hello Harbor team,

We're seeking help setting up test automation using GitHub actions to test Harbor for OCI conformance. See full issue here.

For background: the OCI Distribution Specification is an open spec based on Docker's Registry API, and is the current workspace for defining standards concerning container registries such as Harbor. There is a recent effort to enable registry providers to validate if a registry properly conforms to the specification. 

 

We need your expertise in setting up Harbor for tests purposes within the context of GitHub actions, as well as configuration setup for running the OCI conformance test suite.

You can view a live view of preliminary test results at https://oci.bloodorange.io. Click one of the "test report" links to open up a detailed conformance report.

If you're interested in the larger discussion, I encourage you to join the the OCI mailing list.

Feel free to email me or ping me on Slack if you have any questions etc.

 

- Josh Dolitsky (@jdolitsky)

HI josh,

 

Did you use the latest conformance testing code? The csrf issue has been fixed by my PR https://github.com/opencontainers/distribution-spec/commit/ae19cbe28fafbe1d1e03d1c7bbdcd524c0347f69

 

Here is my results:

 

06:16:27 Ran 27 of 54 Specs in 1.671 seconds

442506:16:27 FAIL! -- 24 Passed | 3 Failed | 0 Pending | 27 Skipped

 

Thanks,

Yan

 

From: <harbor-dev@...> on behalf of "Josh Dolitsky via lists.cncf.io" <jdolitsky=gmail.com@...>
Reply-To: "harbor-dev@..." <harbor-dev@...>
Date: Friday, 1 May 2020 at 6:49 AM
To: "harbor-dev@..." <harbor-dev@...>
Cc: Daniel Jiang <jiangd@...>, Alex Xu <xalex@...>
Subject: Re: [harbor-dev] Assistance testing Harbor for OCI conformance

 

I started setting up some workflows testing against the live demo.goharbor.io site. Seeing a CSRF protection error that we haven't seen with the other registries:

{ "errors": [ { "code": "FORBIDDEN", "message": "CSRF token invalid" } ] }

Here is a URL with the results table now including Harbor: https://bloodorangeio.github.io/oci-conformance/distribution-spec/
Here is a sample report with the failure (expand first red section): https://oci-conformance.s3.amazonaws.com/distribution-spec/harbor/pull/report.html

Do we just need to pass along the cookie returned from the token endpoint? (GET /service/token?scope=...). The code used for the raw HTTP requests which we will need to fix can be found here: https://github.com/bloodorangeio/reggie

Thanks,

Josh

 

On Wed, Apr 29, 2020 at 11:59 AM Josh Dolitsky via lists.cncf.io <jdolitsky=gmail.com@...> wrote:

I'm happy to see it being tested on the Harbor side. This dashboard of live results is nice to have, but the real end goal is for registries to submit static test results (report.html etc.) as well as a README for how to reproduce as a PR against this repo: https://github.com/opencontainers/oci-conformance/blob/master/instructions.md (see instructions)

This is based on the self-certification model used by Kubernetes: https://github.com/cncf/k8s-conformance

You'll see in the k8s-conformance repo several directories for different k8s versions, within static test results generated by running Heptio Sonobuoy.

In the case of the oci-conformance repo, imagine directories for each release of the distribution-spec (currently pre-1.0). The conformance test suite is the equivalent of Sonobuoy in this scenario.

Feel free to reach out directly for clarification.

 

Josh

 

On Wed, Apr 29, 2020 at 10:32 AM Yan Wang <wangyan@...> wrote:

For the live testing, I have opened a PR at https://github.com/goharbor/harbor/pull/11801. We’d like to enable the live testing as Harbor repo so that we can know which PR introduces new failures and fix it on time.

 

@Daniel Jiang @Alex Xu  your idea?

 

Thanks,

Yan

 

From: <harbor-dev@...> on behalf of "Josh Dolitsky via lists.cncf.io" <jdolitsky=gmail.com@...>
Reply-To: "harbor-dev@..." <harbor-dev@...>
Date: Wednesday, 29 April 2020 at 11:02 PM
To: "harbor-dev@..." <harbor-dev@...>
Subject: Re: [harbor-dev] Assistance testing Harbor for OCI conformance

 

Yan, thanks for helping with this. Do you know if it is possible to start a fresh new Harbor instance in GitHub actions? GitHub actions allows for a "services" section which you can launch containers for the test run. This is what we really need help from Harbor team on. Here is a example action using a redis: https://github.com/actions/example-services/blob/master/.github/workflows/redis-service.yml#L18

Since Harbor is an opensource registry, it would be preferred to do it this way vs. testing a live version (i.e. demo.goharbor.io). This is definitely a great start regardless.

Thank you,

 

Josh

 

 

On Wed, Apr 29, 2020 at 4:41 AM Yan Wang <wangyan@...> wrote:

Hi Josh,

 

Thanks for your detailed info.

 

I have filed an PR to add Harbor conformance test, could you please help to review? https://github.com/bloodorangeio/oci-distribution-conformance-results/pull/10

BTW, can you help to add two more secrets(Harbor_USERNAME and Harbor_PASSWORD) to store the testing account auth?

 

Thanks,

Yan

 

From: <harbor-dev@...> on behalf of "Daniel Jiang via lists.cncf.io" <jiangd=vmware.com@...>
Reply-To: "harbor-dev@..." <harbor-dev@...>
Date: Tuesday, 7 April 2020 at 2:05 AM
To: "harbor-dev@..." <harbor-dev@...>
Subject: Re: [harbor-dev] Assistance testing Harbor for OCI conformance

 

Josh,

 

Thanks for the heads up.  The conformance test has been on our radar and we did some manual dry run locally.

 

Adding Harbor to the report is on the todo list, and we’ll have that incorporated within this or next week.

 

Best Regards

— 

Daniel Jiang  | 姜坦

Engineer, VMware R&D, Beijing

+86 10-59934536

 

 

On 2020/4/6, 10:21 AM, "harbor-dev@... on behalf of Josh Dolitsky via lists.cncf.io" <harbor-dev@... on behalf of jdolitsky=gmail.com@...> wrote:

 

Hello Harbor team,

We're seeking help setting up test automation using GitHub actions to test Harbor for OCI conformance. See full issue here.

For background: the OCI Distribution Specification is an open spec based on Docker's Registry API, and is the current workspace for defining standards concerning container registries such as Harbor. There is a recent effort to enable registry providers to validate if a registry properly conforms to the specification. 

 

We need your expertise in setting up Harbor for tests purposes within the context of GitHub actions, as well as configuration setup for running the OCI conformance test suite.

You can view a live view of preliminary test results at https://oci.bloodorange.io. Click one of the "test report" links to open up a detailed conformance report.

If you're interested in the larger discussion, I encourage you to join the the OCI mailing list.

Feel free to email me or ping me on Slack if you have any questions etc.

 

- Josh Dolitsky (@jdolitsky)