feat: load model from oci image

This commit is contained in:
2023-05-05 17:07:29 +02:00
parent b57698380e
commit 9fb01f7be9
441 changed files with 61395 additions and 15356 deletions

40
vendor/oras.land/oras-go/v2/.gitignore vendored Normal file
View File

@@ -0,0 +1,40 @@
# Copyright The ORAS Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib
# Test binary, build with `go test -c`
*.test
# Output of the go coverage tool, specifically when used with LiteIDE
*.out
# VS Code
.vscode
debug
# Jetbrains
.idea
# Custom
coverage.txt
bin/
dist/
*.tar.gz
vendor/
_dist/

View File

@@ -0,0 +1,2 @@
# Derived from OWNERS.md
* @sajayantony @shizhMSFT @stevelasker @Wwwsylvia

View File

@@ -0,0 +1,3 @@
# Code of Conduct
OCI Registry As Storage (ORAS) follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md).

201
vendor/oras.land/oras-go/v2/LICENSE vendored Normal file
View File

@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2021 ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

View File

@@ -0,0 +1,45 @@
# Migration Guide
In version `v2`, ORAS Go library has been completely refreshed with:
- More unified interfaces
- Notably fewer dependencies
- Higher test coverage
- Better documentation
**Besides, ORAS Go `v2` is now a registry client.**
## Major Changes in `v2`
- Moves `content.FileStore` to [file.Store](https://pkg.go.dev/oras.land/oras-go/v2/content/file#Store)
- Moves `content.OCIStore` to [oci.Store](https://pkg.go.dev/oras.land/oras-go/v2/content/oci#Store)
- Moves `content.MemoryStore` to [memory.Store](https://pkg.go.dev/oras.land/oras-go/v2/content/memory#Store)
- Provides [SDK](https://pkg.go.dev/oras.land/oras-go/v2/registry/remote) to interact with OCI-compliant and Docker-compliant registries
- Supports [Copy](https://pkg.go.dev/oras.land/oras-go/v2#Copy) with more flexible options
- Supports [Extended Copy](https://pkg.go.dev/oras.land/oras-go/v2#ExtendedCopy) with options *(experimental)*
- No longer supports `docker.Login` and `docker.Logout` (removes the dependency on `docker`); instead, provides authentication through [auth.Client](https://pkg.go.dev/oras.land/oras-go/v2/registry/remote/auth#Client)
Documentation and examples are available at [pkg.go.dev](https://pkg.go.dev/oras.land/oras-go/v2).
## Migrating from `v1` to `v2`
1. Get the `v2` package
```sh
go get oras.land/oras-go/v2
```
2. Import and use the `v2` package
```go
import "oras.land/oras-go/v2"
```
3. Run
```sh
go mod tidy
```
Since breaking changes are introduced in `v2`, code refactoring is required for migrating from `v1` to `v2`.
The migration can be done in an iterative fashion, as `v1` and `v2` can be imported and used at the same time.

38
vendor/oras.land/oras-go/v2/Makefile vendored Normal file
View File

@@ -0,0 +1,38 @@
# Copyright The ORAS Authors.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
.PHONY: test
test: vendor check-encoding
go test -race -v -coverprofile=coverage.txt -covermode=atomic ./...
.PHONY: covhtml
covhtml:
open .cover/coverage.html
.PHONY: clean
clean:
git status --ignored --short | grep '^!! ' | sed 's/!! //' | xargs rm -rf
.PHONY: check-encoding
check-encoding:
! find . -not -path "./vendor/*" -name "*.go" -type f -exec file "{}" ";" | grep CRLF
! find scripts -name "*.sh" -type f -exec file "{}" ";" | grep CRLF
.PHONY: fix-encoding
fix-encoding:
find . -not -path "./vendor/*" -name "*.go" -type f -exec sed -i -e "s/\r//g" {} +
find scripts -name "*.sh" -type f -exec sed -i -e "s/\r//g" {} +
.PHONY: vendor
vendor:
go mod vendor

11
vendor/oras.land/oras-go/v2/OWNERS.md vendored Normal file
View File

@@ -0,0 +1,11 @@
# Owners
Owners:
- Sajay Antony (@sajayantony)
- Shiwei Zhang (@shizhMSFT)
- Steve Lasker (@stevelasker)
- Sylvia Lei (@Wwwsylvia)
Emeritus:
- Avi Deitcher (@deitch)
- Josh Dolitsky (@jdolitsky)

53
vendor/oras.land/oras-go/v2/README.md vendored Normal file
View File

@@ -0,0 +1,53 @@
# ORAS Go library
![ORAS](https://github.com/oras-project/oras-www/raw/main/docs/assets/images/oras.png)
## Project status
### Versioning
The ORAS Go library follows [Semantic Versioning](https://semver.org/), where breaking changes are reserved for MAJOR releases, and MINOR and PATCH releases must be 100% backwards compatible.
### v2: stable
[![Build Status](https://github.com/oras-project/oras-go/actions/workflows/build.yml/badge.svg?event=push&branch=main)](https://github.com/oras-project/oras-go/actions/workflows/build.yml?query=workflow%3Abuild+event%3Apush+branch%3Amain)
[![codecov](https://codecov.io/gh/oras-project/oras-go/branch/main/graph/badge.svg)](https://codecov.io/gh/oras-project/oras-go)
[![Go Report Card](https://goreportcard.com/badge/oras.land/oras-go/v2)](https://goreportcard.com/report/oras.land/oras-go/v2)
[![Go Reference](https://pkg.go.dev/badge/oras.land/oras-go/v2.svg)](https://pkg.go.dev/oras.land/oras-go/v2)
The version `2` is actively developed in the [`main`](https://github.com/oras-project/oras-go/tree/main) branch with all new features.
Examples for common use cases can be found below:
- [Copy examples](https://pkg.go.dev/oras.land/oras-go/v2#pkg-examples)
- [Registry interaction examples](https://pkg.go.dev/oras.land/oras-go/v2/registry#pkg-examples)
- [Repository interaction examples](https://pkg.go.dev/oras.land/oras-go/v2/registry/remote#pkg-examples)
- [Authentication examples](https://pkg.go.dev/oras.land/oras-go/v2/registry/remote/auth#pkg-examples)
If you are seeking latest changes, you should use the [`main`](https://github.com/oras-project/oras-go/tree/main) branch (or a specific commit hash) over a tagged version when including the ORAS Go library in your project's `go.mod`.
The Go Reference for the `main` branch is available [here](https://pkg.go.dev/oras.land/oras-go/v2@main).
To migrate from `v1` to `v2`, see [MIGRATION_GUIDE.md](./MIGRATION_GUIDE.md).
### v1: stable
[![Build Status](https://github.com/oras-project/oras-go/actions/workflows/build.yml/badge.svg?event=push&branch=v1)](https://github.com/oras-project/oras-go/actions/workflows/build.yml?query=workflow%3Abuild+event%3Apush+branch%3Av1)
[![Go Report Card](https://goreportcard.com/badge/oras.land/oras-go)](https://goreportcard.com/report/oras.land/oras-go)
[![Go Reference](https://pkg.go.dev/badge/oras.land/oras-go.svg)](https://pkg.go.dev/oras.land/oras-go)
As there are various stable projects depending on the ORAS Go library `v1`, the
[`v1`](https://github.com/oras-project/oras-go/tree/v1) branch
is maintained for API stability, dependency updates, and security patches.
All `v1.*` releases are based upon this branch.
Since `v1` is in a maintenance state, you are highly encouraged
to use releases with major version `2` for new features.
## Docs
- [oras.land/client_libraries/go](https://oras.land/client_libraries/0_go/): Documentation for the ORAS Go library
- [Reviewing guide](https://github.com/oras-project/community/blob/main/REVIEWING.md): All reviewers must read the reviewing guide and agree to follow the project review guidelines.
## Code of Conduct
This project has adopted the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md). See [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) for further details.

412
vendor/oras.land/oras-go/v2/content.go vendored Normal file
View File

@@ -0,0 +1,412 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package oras
import (
"bytes"
"context"
"errors"
"fmt"
"io"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2/content"
"oras.land/oras-go/v2/errdef"
"oras.land/oras-go/v2/internal/cas"
"oras.land/oras-go/v2/internal/docker"
"oras.land/oras-go/v2/internal/interfaces"
"oras.land/oras-go/v2/internal/platform"
"oras.land/oras-go/v2/internal/registryutil"
"oras.land/oras-go/v2/internal/syncutil"
"oras.land/oras-go/v2/registry"
"oras.land/oras-go/v2/registry/remote/auth"
)
const (
// defaultTagConcurrency is the default concurrency of tagging.
defaultTagConcurrency int = 5 // This value is consistent with dockerd
// defaultTagNMaxMetadataBytes is the default value of
// TagNOptions.MaxMetadataBytes.
defaultTagNMaxMetadataBytes int64 = 4 * 1024 * 1024 // 4 MiB
// defaultResolveMaxMetadataBytes is the default value of
// ResolveOptions.MaxMetadataBytes.
defaultResolveMaxMetadataBytes int64 = 4 * 1024 * 1024 // 4 MiB
// defaultMaxBytes is the default value of FetchBytesOptions.MaxBytes.
defaultMaxBytes int64 = 4 * 1024 * 1024 // 4 MiB
)
// DefaultTagNOptions provides the default TagNOptions.
var DefaultTagNOptions TagNOptions
// TagNOptions contains parameters for [oras.TagN].
type TagNOptions struct {
// Concurrency limits the maximum number of concurrent tag tasks.
// If less than or equal to 0, a default (currently 5) is used.
Concurrency int
// MaxMetadataBytes limits the maximum size of metadata that can be cached
// in the memory.
// If less than or equal to 0, a default (currently 4 MiB) is used.
MaxMetadataBytes int64
}
// TagN tags the descriptor identified by srcReference with dstReferences.
func TagN(ctx context.Context, target Target, srcReference string, dstReferences []string, opts TagNOptions) (ocispec.Descriptor, error) {
switch len(dstReferences) {
case 0:
return ocispec.Descriptor{}, fmt.Errorf("dstReferences cannot be empty: %w", errdef.ErrMissingReference)
case 1:
return Tag(ctx, target, srcReference, dstReferences[0])
}
if opts.Concurrency <= 0 {
opts.Concurrency = defaultTagConcurrency
}
if opts.MaxMetadataBytes <= 0 {
opts.MaxMetadataBytes = defaultTagNMaxMetadataBytes
}
_, isRefFetcher := target.(registry.ReferenceFetcher)
_, isRefPusher := target.(registry.ReferencePusher)
if isRefFetcher && isRefPusher {
if repo, ok := target.(interfaces.ReferenceParser); ok {
// add scope hints to minimize the number of auth requests
ref, err := repo.ParseReference(srcReference)
if err != nil {
return ocispec.Descriptor{}, err
}
ctx = registryutil.WithScopeHint(ctx, ref, auth.ActionPull, auth.ActionPush)
}
desc, contentBytes, err := FetchBytes(ctx, target, srcReference, FetchBytesOptions{
MaxBytes: opts.MaxMetadataBytes,
})
if err != nil {
if errors.Is(err, errdef.ErrSizeExceedsLimit) {
err = fmt.Errorf(
"content size %v exceeds MaxMetadataBytes %v: %w",
desc.Size,
opts.MaxMetadataBytes,
errdef.ErrSizeExceedsLimit)
}
return ocispec.Descriptor{}, err
}
if err := tagBytesN(ctx, target, desc, contentBytes, dstReferences, TagBytesNOptions{
Concurrency: opts.Concurrency,
}); err != nil {
return ocispec.Descriptor{}, err
}
return desc, nil
}
desc, err := target.Resolve(ctx, srcReference)
if err != nil {
return ocispec.Descriptor{}, err
}
eg, egCtx := syncutil.LimitGroup(ctx, opts.Concurrency)
for _, dstRef := range dstReferences {
eg.Go(func(dst string) func() error {
return func() error {
if err := target.Tag(egCtx, desc, dst); err != nil {
return fmt.Errorf("failed to tag %s as %s: %w", srcReference, dst, err)
}
return nil
}
}(dstRef))
}
if err := eg.Wait(); err != nil {
return ocispec.Descriptor{}, err
}
return desc, nil
}
// Tag tags the descriptor identified by src with dst.
func Tag(ctx context.Context, target Target, src, dst string) (ocispec.Descriptor, error) {
refFetcher, okFetch := target.(registry.ReferenceFetcher)
refPusher, okPush := target.(registry.ReferencePusher)
if okFetch && okPush {
if repo, ok := target.(interfaces.ReferenceParser); ok {
// add scope hints to minimize the number of auth requests
ref, err := repo.ParseReference(src)
if err != nil {
return ocispec.Descriptor{}, err
}
ctx = registryutil.WithScopeHint(ctx, ref, auth.ActionPull, auth.ActionPush)
}
desc, rc, err := refFetcher.FetchReference(ctx, src)
if err != nil {
return ocispec.Descriptor{}, err
}
defer rc.Close()
if err := refPusher.PushReference(ctx, desc, rc, dst); err != nil {
return ocispec.Descriptor{}, err
}
return desc, nil
}
desc, err := target.Resolve(ctx, src)
if err != nil {
return ocispec.Descriptor{}, err
}
if err := target.Tag(ctx, desc, dst); err != nil {
return ocispec.Descriptor{}, err
}
return desc, nil
}
// DefaultResolveOptions provides the default ResolveOptions.
var DefaultResolveOptions ResolveOptions
// ResolveOptions contains parameters for [oras.Resolve].
type ResolveOptions struct {
// TargetPlatform ensures the resolved content matches the target platform
// if the node is a manifest, or selects the first resolved content that
// matches the target platform if the node is a manifest list.
TargetPlatform *ocispec.Platform
// MaxMetadataBytes limits the maximum size of metadata that can be cached
// in the memory.
// If less than or equal to 0, a default (currently 4 MiB) is used.
MaxMetadataBytes int64
}
// Resolve resolves a descriptor with provided reference from the target.
func Resolve(ctx context.Context, target ReadOnlyTarget, reference string, opts ResolveOptions) (ocispec.Descriptor, error) {
if opts.TargetPlatform == nil {
return target.Resolve(ctx, reference)
}
return resolve(ctx, target, nil, reference, opts)
}
// resolve resolves a descriptor with provided reference from the target, with
// specified caching.
func resolve(ctx context.Context, target ReadOnlyTarget, proxy *cas.Proxy, reference string, opts ResolveOptions) (ocispec.Descriptor, error) {
if opts.MaxMetadataBytes <= 0 {
opts.MaxMetadataBytes = defaultResolveMaxMetadataBytes
}
if refFetcher, ok := target.(registry.ReferenceFetcher); ok {
// optimize performance for ReferenceFetcher targets
desc, rc, err := refFetcher.FetchReference(ctx, reference)
if err != nil {
return ocispec.Descriptor{}, err
}
defer rc.Close()
switch desc.MediaType {
case docker.MediaTypeManifestList, ocispec.MediaTypeImageIndex,
docker.MediaTypeManifest, ocispec.MediaTypeImageManifest:
// cache the fetched content
if desc.Size > opts.MaxMetadataBytes {
return ocispec.Descriptor{}, fmt.Errorf(
"content size %v exceeds MaxMetadataBytes %v: %w",
desc.Size,
opts.MaxMetadataBytes,
errdef.ErrSizeExceedsLimit)
}
if proxy == nil {
proxy = cas.NewProxyWithLimit(target, cas.NewMemory(), opts.MaxMetadataBytes)
}
if err := proxy.Cache.Push(ctx, desc, rc); err != nil {
return ocispec.Descriptor{}, err
}
// stop caching as SelectManifest may fetch a config blob
proxy.StopCaching = true
return platform.SelectManifest(ctx, proxy, desc, opts.TargetPlatform)
default:
return ocispec.Descriptor{}, fmt.Errorf("%s: %s: %w", desc.Digest, desc.MediaType, errdef.ErrUnsupported)
}
}
desc, err := target.Resolve(ctx, reference)
if err != nil {
return ocispec.Descriptor{}, err
}
return platform.SelectManifest(ctx, target, desc, opts.TargetPlatform)
}
// DefaultFetchOptions provides the default FetchOptions.
var DefaultFetchOptions FetchOptions
// FetchOptions contains parameters for [oras.Fetch].
type FetchOptions struct {
// ResolveOptions contains parameters for resolving reference.
ResolveOptions
}
// Fetch fetches the content identified by the reference.
func Fetch(ctx context.Context, target ReadOnlyTarget, reference string, opts FetchOptions) (ocispec.Descriptor, io.ReadCloser, error) {
if opts.TargetPlatform == nil {
if refFetcher, ok := target.(registry.ReferenceFetcher); ok {
return refFetcher.FetchReference(ctx, reference)
}
desc, err := target.Resolve(ctx, reference)
if err != nil {
return ocispec.Descriptor{}, nil, err
}
rc, err := target.Fetch(ctx, desc)
if err != nil {
return ocispec.Descriptor{}, nil, err
}
return desc, rc, nil
}
if opts.MaxMetadataBytes <= 0 {
opts.MaxMetadataBytes = defaultResolveMaxMetadataBytes
}
proxy := cas.NewProxyWithLimit(target, cas.NewMemory(), opts.MaxMetadataBytes)
desc, err := resolve(ctx, target, proxy, reference, opts.ResolveOptions)
if err != nil {
return ocispec.Descriptor{}, nil, err
}
// if the content exists in cache, fetch it from cache
// otherwise fetch without caching
proxy.StopCaching = true
rc, err := proxy.Fetch(ctx, desc)
if err != nil {
return ocispec.Descriptor{}, nil, err
}
return desc, rc, nil
}
// DefaultFetchBytesOptions provides the default FetchBytesOptions.
var DefaultFetchBytesOptions FetchBytesOptions
// FetchBytesOptions contains parameters for [oras.FetchBytes].
type FetchBytesOptions struct {
// FetchOptions contains parameters for fetching content.
FetchOptions
// MaxBytes limits the maximum size of the fetched content bytes.
// If less than or equal to 0, a default (currently 4 MiB) is used.
MaxBytes int64
}
// FetchBytes fetches the content bytes identified by the reference.
func FetchBytes(ctx context.Context, target ReadOnlyTarget, reference string, opts FetchBytesOptions) (ocispec.Descriptor, []byte, error) {
if opts.MaxBytes <= 0 {
opts.MaxBytes = defaultMaxBytes
}
desc, rc, err := Fetch(ctx, target, reference, opts.FetchOptions)
if err != nil {
return ocispec.Descriptor{}, nil, err
}
defer rc.Close()
if desc.Size > opts.MaxBytes {
return ocispec.Descriptor{}, nil, fmt.Errorf(
"content size %v exceeds MaxBytes %v: %w",
desc.Size,
opts.MaxBytes,
errdef.ErrSizeExceedsLimit)
}
bytes, err := content.ReadAll(rc, desc)
if err != nil {
return ocispec.Descriptor{}, nil, err
}
return desc, bytes, nil
}
// PushBytes describes the contentBytes using the given mediaType and pushes it.
// If mediaType is not specified, "application/octet-stream" is used.
func PushBytes(ctx context.Context, pusher content.Pusher, mediaType string, contentBytes []byte) (ocispec.Descriptor, error) {
desc := content.NewDescriptorFromBytes(mediaType, contentBytes)
r := bytes.NewReader(contentBytes)
if err := pusher.Push(ctx, desc, r); err != nil {
return ocispec.Descriptor{}, err
}
return desc, nil
}
// DefaultTagBytesNOptions provides the default TagBytesNOptions.
var DefaultTagBytesNOptions TagBytesNOptions
// TagBytesNOptions contains parameters for [oras.TagBytesN].
type TagBytesNOptions struct {
// Concurrency limits the maximum number of concurrent tag tasks.
// If less than or equal to 0, a default (currently 5) is used.
Concurrency int
}
// TagBytesN describes the contentBytes using the given mediaType, pushes it,
// and tag it with the given references.
// If mediaType is not specified, "application/octet-stream" is used.
func TagBytesN(ctx context.Context, target Target, mediaType string, contentBytes []byte, references []string, opts TagBytesNOptions) (ocispec.Descriptor, error) {
if len(references) == 0 {
return PushBytes(ctx, target, mediaType, contentBytes)
}
desc := content.NewDescriptorFromBytes(mediaType, contentBytes)
if opts.Concurrency <= 0 {
opts.Concurrency = defaultTagConcurrency
}
if err := tagBytesN(ctx, target, desc, contentBytes, references, opts); err != nil {
return ocispec.Descriptor{}, err
}
return desc, nil
}
// tagBytesN pushes the contentBytes using the given desc, and tag it with the
// given references.
func tagBytesN(ctx context.Context, target Target, desc ocispec.Descriptor, contentBytes []byte, references []string, opts TagBytesNOptions) error {
eg, egCtx := syncutil.LimitGroup(ctx, opts.Concurrency)
if refPusher, ok := target.(registry.ReferencePusher); ok {
for _, reference := range references {
eg.Go(func(ref string) func() error {
return func() error {
r := bytes.NewReader(contentBytes)
if err := refPusher.PushReference(egCtx, desc, r, ref); err != nil && !errors.Is(err, errdef.ErrAlreadyExists) {
return fmt.Errorf("failed to tag %s: %w", ref, err)
}
return nil
}
}(reference))
}
} else {
r := bytes.NewReader(contentBytes)
if err := target.Push(ctx, desc, r); err != nil && !errors.Is(err, errdef.ErrAlreadyExists) {
return fmt.Errorf("failed to push content: %w", err)
}
for _, reference := range references {
eg.Go(func(ref string) func() error {
return func() error {
if err := target.Tag(egCtx, desc, ref); err != nil {
return fmt.Errorf("failed to tag %s: %w", ref, err)
}
return nil
}
}(reference))
}
}
return eg.Wait()
}
// TagBytes describes the contentBytes using the given mediaType, pushes it,
// and tag it with the given reference.
// If mediaType is not specified, "application/octet-stream" is used.
func TagBytes(ctx context.Context, target Target, mediaType string, contentBytes []byte, reference string) (ocispec.Descriptor, error) {
return TagBytesN(ctx, target, mediaType, contentBytes, []string{reference}, DefaultTagBytesNOptions)
}

View File

@@ -0,0 +1,40 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package content
import (
"github.com/opencontainers/go-digest"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2/internal/descriptor"
)
// NewDescriptorFromBytes returns a descriptor, given the content and media type.
// If no media type is specified, "application/octet-stream" will be used.
func NewDescriptorFromBytes(mediaType string, content []byte) ocispec.Descriptor {
if mediaType == "" {
mediaType = descriptor.DefaultMediaType
}
return ocispec.Descriptor{
MediaType: mediaType,
Digest: digest.FromBytes(content),
Size: int64(len(content)),
}
}
// Equal returns true if two descriptors point to the same content.
func Equal(a, b ocispec.Descriptor) bool {
return a.Size == b.Size && a.Digest == b.Digest && a.MediaType == b.MediaType
}

View File

@@ -0,0 +1,28 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package file
import "errors"
var (
ErrMissingName = errors.New("missing name")
ErrDuplicateName = errors.New("duplicate name")
ErrPathTraversalDisallowed = errors.New("path traversal disallowed")
ErrOverwriteDisallowed = errors.New("overwrite disallowed")
ErrStoreClosed = errors.New("store already closed")
)
var errSkipUnnamed = errors.New("unnamed descriptor skipped")

View File

@@ -0,0 +1,671 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package file provides implementation of a content store based on file system.
package file
import (
"compress/gzip"
"context"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"github.com/opencontainers/go-digest"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2/content"
"oras.land/oras-go/v2/errdef"
"oras.land/oras-go/v2/internal/cas"
"oras.land/oras-go/v2/internal/graph"
"oras.land/oras-go/v2/internal/ioutil"
"oras.land/oras-go/v2/internal/resolver"
)
// bufPool is a pool of byte buffers that can be reused for copying content
// between files.
var bufPool = sync.Pool{
New: func() interface{} {
// the buffer size should be larger than or equal to 128 KiB
// for performance considerations.
// we choose 1 MiB here so there will be less disk I/O.
buffer := make([]byte, 1<<20) // buffer size = 1 MiB
return &buffer
},
}
const (
// AnnotationDigest is the annotation key for the digest of the uncompressed content.
AnnotationDigest = "io.deis.oras.content.digest"
// AnnotationUnpack is the annotation key for indication of unpacking.
AnnotationUnpack = "io.deis.oras.content.unpack"
// defaultBlobMediaType specifies the default blob media type.
defaultBlobMediaType = ocispec.MediaTypeImageLayer
// defaultBlobDirMediaType specifies the default blob directory media type.
defaultBlobDirMediaType = ocispec.MediaTypeImageLayerGzip
// defaultFallbackPushSizeLimit specifies the default size limit for pushing no-name contents.
defaultFallbackPushSizeLimit = 1 << 22 // 4 MiB
)
// Store represents a file system based store, which implements `oras.Target`.
//
// In the file store, the contents described by names are location-addressed
// by file paths. Meanwhile, the file paths are mapped to a virtual CAS
// where all metadata are stored in the memory.
//
// The contents that are not described by names are stored in a fallback storage,
// which is a limited memory CAS by default.
// As all the metadata are stored in the memory, the file store
// cannot be restored from the file system.
//
// After use, the file store needs to be closed by calling the [Store.Close] function.
// The file store cannot be used after being closed.
type Store struct {
// TarReproducible controls if the tarballs generated
// for the added directories are reproducible.
// When specified, some metadata such as change time
// will be removed from the files in the tarballs. Default value: false.
TarReproducible bool
// AllowPathTraversalOnWrite controls if path traversal is allowed
// when writing files. When specified, writing files
// outside the working directory will be allowed. Default value: false.
AllowPathTraversalOnWrite bool
// DisableOverwrite controls if push operations can overwrite existing files.
// When specified, saving files to existing paths will be disabled.
// Default value: false.
DisableOverwrite bool
// ForceCAS controls if files with same content but different names are
// deduped after push operations. When a DAG is copied between CAS
// targets, nodes are deduped by content. By default, file store restores
// deduped successor files after a node is copied. This may result in two
// files with identical content. If this is not the desired behavior,
// ForceCAS can be specified to enforce CAS style dedup.
// Default value: false.
ForceCAS bool
// IgnoreNoName controls if push operations should ignore descriptors
// without a name. When specified, corresponding content will be discarded.
// Otherwise, content will be saved to a fallback storage.
// A typical scenario is pulling an arbitrary artifact masqueraded as OCI
// image to file store. This option can be specified to discard unnamed
// manifest and config file, while leaving only named layer files.
// Default value: false.
IgnoreNoName bool
workingDir string // the working directory of the file store
closed int32 // if the store is closed - 0: false, 1: true.
digestToPath sync.Map // map[digest.Digest]string
nameToStatus sync.Map // map[string]*nameStatus
tmpFiles sync.Map // map[string]bool
fallbackStorage content.Storage
resolver content.TagResolver
graph *graph.Memory
}
// nameStatus contains a flag indicating if a name exists,
// and a RWMutex protecting it.
type nameStatus struct {
sync.RWMutex
exists bool
}
// New creates a file store, using a default limited memory CAS
// as the fallback storage for contents without names.
// When pushing content without names, the size of content being pushed
// cannot exceed the default size limit: 4 MiB.
func New(workingDir string) (*Store, error) {
return NewWithFallbackLimit(workingDir, defaultFallbackPushSizeLimit)
}
// NewWithFallbackLimit creates a file store, using a default
// limited memory CAS as the fallback storage for contents without names.
// When pushing content without names, the size of content being pushed
// cannot exceed the size limit specified by the `limit` parameter.
func NewWithFallbackLimit(workingDir string, limit int64) (*Store, error) {
m := cas.NewMemory()
ls := content.LimitStorage(m, limit)
return NewWithFallbackStorage(workingDir, ls)
}
// NewWithFallbackStorage creates a file store,
// using the provided fallback storage for contents without names.
func NewWithFallbackStorage(workingDir string, fallbackStorage content.Storage) (*Store, error) {
workingDirAbs, err := filepath.Abs(workingDir)
if err != nil {
return nil, fmt.Errorf("failed to resolve absolute path for %s: %w", workingDir, err)
}
return &Store{
workingDir: workingDirAbs,
fallbackStorage: fallbackStorage,
resolver: resolver.NewMemory(),
graph: graph.NewMemory(),
}, nil
}
// Close closes the file store and cleans up all the temporary files used by it.
// The store cannot be used after being closed.
// This function is not go-routine safe.
func (s *Store) Close() error {
if s.isClosedSet() {
return nil
}
s.setClosed()
var errs []string
s.tmpFiles.Range(func(name, _ interface{}) bool {
if err := os.Remove(name.(string)); err != nil {
errs = append(errs, err.Error())
}
return true
})
if len(errs) > 0 {
return errors.New(strings.Join(errs, "; "))
}
return nil
}
// Fetch fetches the content identified by the descriptor.
func (s *Store) Fetch(ctx context.Context, target ocispec.Descriptor) (io.ReadCloser, error) {
if s.isClosedSet() {
return nil, ErrStoreClosed
}
// if the target has name, check if the name exists.
name := target.Annotations[ocispec.AnnotationTitle]
if name != "" && !s.nameExists(name) {
return nil, fmt.Errorf("%s: %s: %w", name, target.MediaType, errdef.ErrNotFound)
}
// check if the content exists in the store
val, exists := s.digestToPath.Load(target.Digest)
if exists {
path := val.(string)
fp, err := os.Open(path)
if err != nil {
if os.IsNotExist(err) {
return nil, fmt.Errorf("%s: %s: %w", target.Digest, target.MediaType, errdef.ErrNotFound)
}
return nil, err
}
return fp, nil
}
// if the content does not exist in the store,
// then fall back to the fallback storage.
return s.fallbackStorage.Fetch(ctx, target)
}
// Push pushes the content, matching the expected descriptor.
// If name is not specified in the descriptor, the content will be pushed to
// the fallback storage by default, or will be discarded when
// Store.IgnoreNoName is true.
func (s *Store) Push(ctx context.Context, expected ocispec.Descriptor, content io.Reader) error {
if s.isClosedSet() {
return ErrStoreClosed
}
if err := s.push(ctx, expected, content); err != nil {
if errors.Is(err, errSkipUnnamed) {
return nil
}
return err
}
if !s.ForceCAS {
if err := s.restoreDuplicates(ctx, expected); err != nil {
return fmt.Errorf("failed to restore duplicated file: %w", err)
}
}
return s.graph.Index(ctx, s, expected)
}
// push pushes the content, matching the expected descriptor.
// If name is not specified in the descriptor, the content will be pushed to
// the fallback storage by default, or will be discarded when
// Store.IgnoreNoName is true.
func (s *Store) push(ctx context.Context, expected ocispec.Descriptor, content io.Reader) error {
name := expected.Annotations[ocispec.AnnotationTitle]
if name == "" {
if s.IgnoreNoName {
return errSkipUnnamed
}
return s.fallbackStorage.Push(ctx, expected, content)
}
// check the status of the name
status := s.status(name)
status.Lock()
defer status.Unlock()
if status.exists {
return fmt.Errorf("%s: %w", name, ErrDuplicateName)
}
target, err := s.resolveWritePath(name)
if err != nil {
return fmt.Errorf("failed to resolve path for writing: %w", err)
}
if needUnpack := expected.Annotations[AnnotationUnpack]; needUnpack == "true" {
err = s.pushDir(name, target, expected, content)
} else {
err = s.pushFile(target, expected, content)
}
if err != nil {
return err
}
// update the name status as existed
status.exists = true
return nil
}
// restoreDuplicates restores successor files with same content but different names.
// See Store.ForceCAS for more info.
func (s *Store) restoreDuplicates(ctx context.Context, desc ocispec.Descriptor) error {
successors, err := content.Successors(ctx, s, desc)
if err != nil {
return err
}
for _, successor := range successors {
name := successor.Annotations[ocispec.AnnotationTitle]
if name == "" || s.nameExists(name) {
continue
}
if err := func() error {
desc := ocispec.Descriptor{
MediaType: successor.MediaType,
Digest: successor.Digest,
Size: successor.Size,
}
rc, err := s.Fetch(ctx, desc)
if err != nil {
return fmt.Errorf("%q: %s: %w", name, desc.MediaType, err)
}
defer rc.Close()
if err := s.push(ctx, successor, rc); err != nil {
return fmt.Errorf("%q: %s: %w", name, desc.MediaType, err)
}
return nil
}(); err != nil && !errors.Is(err, errdef.ErrNotFound) {
return err
}
}
return nil
}
// Exists returns true if the described content exists.
func (s *Store) Exists(ctx context.Context, target ocispec.Descriptor) (bool, error) {
if s.isClosedSet() {
return false, ErrStoreClosed
}
// if the target has name, check if the name exists.
name := target.Annotations[ocispec.AnnotationTitle]
if name != "" && !s.nameExists(name) {
return false, nil
}
// check if the content exists in the store
_, exists := s.digestToPath.Load(target.Digest)
if exists {
return true, nil
}
// if the content does not exist in the store,
// then fall back to the fallback storage.
return s.fallbackStorage.Exists(ctx, target)
}
// Resolve resolves a reference to a descriptor.
func (s *Store) Resolve(ctx context.Context, ref string) (ocispec.Descriptor, error) {
if s.isClosedSet() {
return ocispec.Descriptor{}, ErrStoreClosed
}
if ref == "" {
return ocispec.Descriptor{}, errdef.ErrMissingReference
}
return s.resolver.Resolve(ctx, ref)
}
// Tag tags a descriptor with a reference string.
func (s *Store) Tag(ctx context.Context, desc ocispec.Descriptor, ref string) error {
if s.isClosedSet() {
return ErrStoreClosed
}
if ref == "" {
return errdef.ErrMissingReference
}
exists, err := s.Exists(ctx, desc)
if err != nil {
return err
}
if !exists {
return fmt.Errorf("%s: %s: %w", desc.Digest, desc.MediaType, errdef.ErrNotFound)
}
return s.resolver.Tag(ctx, desc, ref)
}
// Predecessors returns the nodes directly pointing to the current node.
// Predecessors returns nil without error if the node does not exists in the
// store.
func (s *Store) Predecessors(ctx context.Context, node ocispec.Descriptor) ([]ocispec.Descriptor, error) {
if s.isClosedSet() {
return nil, ErrStoreClosed
}
return s.graph.Predecessors(ctx, node)
}
// Add adds a file into the file store.
func (s *Store) Add(_ context.Context, name, mediaType, path string) (ocispec.Descriptor, error) {
if s.isClosedSet() {
return ocispec.Descriptor{}, ErrStoreClosed
}
if name == "" {
return ocispec.Descriptor{}, ErrMissingName
}
// check the status of the name
status := s.status(name)
status.Lock()
defer status.Unlock()
if status.exists {
return ocispec.Descriptor{}, fmt.Errorf("%s: %w", name, ErrDuplicateName)
}
if path == "" {
path = name
}
path = s.absPath(path)
fi, err := os.Stat(path)
if err != nil {
return ocispec.Descriptor{}, fmt.Errorf("failed to stat %s: %w", path, err)
}
// generate descriptor
var desc ocispec.Descriptor
if fi.IsDir() {
desc, err = s.descriptorFromDir(name, mediaType, path)
} else {
desc, err = s.descriptorFromFile(fi, mediaType, path)
}
if err != nil {
return ocispec.Descriptor{}, fmt.Errorf("failed to generate descriptor from %s: %w", path, err)
}
if desc.Annotations == nil {
desc.Annotations = make(map[string]string)
}
desc.Annotations[ocispec.AnnotationTitle] = name
// update the name status as existed
status.exists = true
return desc, nil
}
// saveFile saves content matching the descriptor to the given file.
func (s *Store) saveFile(fp *os.File, expected ocispec.Descriptor, content io.Reader) (err error) {
defer func() {
closeErr := fp.Close()
if err == nil {
err = closeErr
}
}()
path := fp.Name()
buf := bufPool.Get().(*[]byte)
defer bufPool.Put(buf)
if err := ioutil.CopyBuffer(fp, content, *buf, expected); err != nil {
return fmt.Errorf("failed to copy content to %s: %w", path, err)
}
s.digestToPath.Store(expected.Digest, path)
return nil
}
// pushFile saves content matching the descriptor to the target path.
func (s *Store) pushFile(target string, expected ocispec.Descriptor, content io.Reader) error {
if err := ensureDir(filepath.Dir(target)); err != nil {
return fmt.Errorf("failed to ensure directories of the target path: %w", err)
}
fp, err := os.Create(target)
if err != nil {
return fmt.Errorf("failed to create file %s: %w", target, err)
}
return s.saveFile(fp, expected, content)
}
// pushDir saves content matching the descriptor to the target directory.
func (s *Store) pushDir(name, target string, expected ocispec.Descriptor, content io.Reader) (err error) {
if err := ensureDir(target); err != nil {
return fmt.Errorf("failed to ensure directories of the target path: %w", err)
}
gz, err := s.tempFile()
if err != nil {
return err
}
gzPath := gz.Name()
// the digest of the gz is verified while saving
if err := s.saveFile(gz, expected, content); err != nil {
return fmt.Errorf("failed to save gzip to %s: %w", gzPath, err)
}
checksum := expected.Annotations[AnnotationDigest]
buf := bufPool.Get().(*[]byte)
defer bufPool.Put(buf)
if err := extractTarGzip(target, name, gzPath, checksum, *buf); err != nil {
return fmt.Errorf("failed to extract tar to %s: %w", target, err)
}
return nil
}
// descriptorFromDir generates descriptor from the given directory.
func (s *Store) descriptorFromDir(name, mediaType, dir string) (desc ocispec.Descriptor, err error) {
// make a temp file to store the gzip
gz, err := s.tempFile()
if err != nil {
return ocispec.Descriptor{}, err
}
defer func() {
closeErr := gz.Close()
if err == nil {
err = closeErr
}
}()
// compress the directory
gzDigester := digest.Canonical.Digester()
gzw := gzip.NewWriter(io.MultiWriter(gz, gzDigester.Hash()))
defer func() {
closeErr := gzw.Close()
if err == nil {
err = closeErr
}
}()
tarDigester := digest.Canonical.Digester()
tw := io.MultiWriter(gzw, tarDigester.Hash())
buf := bufPool.Get().(*[]byte)
defer bufPool.Put(buf)
if err := tarDirectory(dir, name, tw, s.TarReproducible, *buf); err != nil {
return ocispec.Descriptor{}, fmt.Errorf("failed to tar %s: %w", dir, err)
}
// flush all
if err := gzw.Close(); err != nil {
return ocispec.Descriptor{}, err
}
if err := gz.Sync(); err != nil {
return ocispec.Descriptor{}, err
}
fi, err := gz.Stat()
if err != nil {
return ocispec.Descriptor{}, err
}
// map gzip digest to gzip path
gzDigest := gzDigester.Digest()
s.digestToPath.Store(gzDigest, gz.Name())
// generate descriptor
if mediaType == "" {
mediaType = defaultBlobDirMediaType
}
return ocispec.Descriptor{
MediaType: mediaType,
Digest: gzDigest, // digest for the compressed content
Size: fi.Size(),
Annotations: map[string]string{
AnnotationDigest: tarDigester.Digest().String(), // digest fot the uncompressed content
AnnotationUnpack: "true", // the content needs to be unpacked
},
}, nil
}
// descriptorFromFile generates descriptor from the given file.
func (s *Store) descriptorFromFile(fi os.FileInfo, mediaType, path string) (desc ocispec.Descriptor, err error) {
fp, err := os.Open(path)
if err != nil {
return ocispec.Descriptor{}, err
}
defer func() {
closeErr := fp.Close()
if err == nil {
err = closeErr
}
}()
dgst, err := digest.FromReader(fp)
if err != nil {
return ocispec.Descriptor{}, err
}
// map digest to file path
s.digestToPath.Store(dgst, path)
// generate descriptor
if mediaType == "" {
mediaType = defaultBlobMediaType
}
return ocispec.Descriptor{
MediaType: mediaType,
Digest: dgst,
Size: fi.Size(),
}, nil
}
// resolveWritePath resolves the path to write for the given name.
func (s *Store) resolveWritePath(name string) (string, error) {
path := s.absPath(name)
if !s.AllowPathTraversalOnWrite {
base, err := filepath.Abs(s.workingDir)
if err != nil {
return "", err
}
target, err := filepath.Abs(path)
if err != nil {
return "", err
}
rel, err := filepath.Rel(base, target)
if err != nil {
return "", ErrPathTraversalDisallowed
}
rel = filepath.ToSlash(rel)
if strings.HasPrefix(rel, "../") || rel == ".." {
return "", ErrPathTraversalDisallowed
}
}
if s.DisableOverwrite {
if _, err := os.Stat(path); err == nil {
return "", ErrOverwriteDisallowed
} else if !os.IsNotExist(err) {
return "", err
}
}
return path, nil
}
// status returns the nameStatus for the given name.
func (s *Store) status(name string) *nameStatus {
v, _ := s.nameToStatus.LoadOrStore(name, &nameStatus{sync.RWMutex{}, false})
status := v.(*nameStatus)
return status
}
// nameExists returns if the given name exists in the file store.
func (s *Store) nameExists(name string) bool {
status := s.status(name)
status.RLock()
defer status.RUnlock()
return status.exists
}
// tempFile creates a temp file with the file name format "oras_file_randomString",
// and returns the pointer to the temp file.
func (s *Store) tempFile() (*os.File, error) {
tmp, err := os.CreateTemp("", "oras_file_*")
if err != nil {
return nil, err
}
s.tmpFiles.Store(tmp.Name(), true)
return tmp, nil
}
// absPath returns the absolute path of the path.
func (s *Store) absPath(path string) string {
if filepath.IsAbs(path) {
return path
}
return filepath.Join(s.workingDir, path)
}
// isClosedSet returns true if the `closed` flag is set, otherwise returns false.
func (s *Store) isClosedSet() bool {
return atomic.LoadInt32(&s.closed) == 1
}
// setClosed sets the `closed` flag.
func (s *Store) setClosed() {
atomic.StoreInt32(&s.closed, 1)
}
// ensureDir ensures the directories of the path exists.
func ensureDir(path string) error {
return os.MkdirAll(path, 0777)
}

View File

@@ -0,0 +1,261 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package file
import (
"archive/tar"
"compress/gzip"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"time"
"github.com/opencontainers/go-digest"
)
// tarDirectory walks the directory specified by path, and tar those files with a new
// path prefix.
func tarDirectory(root, prefix string, w io.Writer, removeTimes bool, buf []byte) (err error) {
tw := tar.NewWriter(w)
defer func() {
closeErr := tw.Close()
if err == nil {
err = closeErr
}
}()
return filepath.Walk(root, func(path string, info os.FileInfo, err error) (returnErr error) {
if err != nil {
return err
}
// Rename path
name, err := filepath.Rel(root, path)
if err != nil {
return err
}
name = filepath.Join(prefix, name)
name = filepath.ToSlash(name)
// Generate header
var link string
mode := info.Mode()
if mode&os.ModeSymlink != 0 {
if link, err = os.Readlink(path); err != nil {
return err
}
}
header, err := tar.FileInfoHeader(info, link)
if err != nil {
return fmt.Errorf("%s: %w", path, err)
}
header.Name = name
header.Uid = 0
header.Gid = 0
header.Uname = ""
header.Gname = ""
if removeTimes {
header.ModTime = time.Time{}
header.AccessTime = time.Time{}
header.ChangeTime = time.Time{}
}
// Write file
if err := tw.WriteHeader(header); err != nil {
return fmt.Errorf("tar: %w", err)
}
if mode.IsRegular() {
fp, err := os.Open(path)
if err != nil {
return err
}
defer func() {
closeErr := fp.Close()
if returnErr == nil {
returnErr = closeErr
}
}()
if _, err := io.CopyBuffer(tw, fp, buf); err != nil {
return fmt.Errorf("failed to copy to %s: %w", path, err)
}
}
return nil
})
}
// extractTarGzip decompresses the gzip
// and extracts tar file to a directory specified by the `dir` parameter.
func extractTarGzip(dir, prefix, filename, checksum string, buf []byte) (err error) {
fp, err := os.Open(filename)
if err != nil {
return err
}
defer func() {
closeErr := fp.Close()
if err == nil {
err = closeErr
}
}()
gzr, err := gzip.NewReader(fp)
if err != nil {
return err
}
defer func() {
closeErr := gzr.Close()
if err == nil {
err = closeErr
}
}()
var r io.Reader = gzr
var verifier digest.Verifier
if checksum != "" {
if digest, err := digest.Parse(checksum); err == nil {
verifier = digest.Verifier()
r = io.TeeReader(r, verifier)
}
}
if err := extractTarDirectory(dir, prefix, r, buf); err != nil {
return err
}
if verifier != nil && !verifier.Verified() {
return errors.New("content digest mismatch")
}
return nil
}
// extractTarDirectory extracts tar file to a directory specified by the `dir`
// parameter. The file name prefix is ensured to be the string specified by the
// `prefix` parameter and is trimmed.
func extractTarDirectory(dir, prefix string, r io.Reader, buf []byte) error {
tr := tar.NewReader(r)
for {
header, err := tr.Next()
if err != nil {
if err == io.EOF {
return nil
}
return err
}
// Name check
name := header.Name
path, err := ensureBasePath(dir, prefix, name)
if err != nil {
return err
}
path = filepath.Join(dir, path)
// Create content
switch header.Typeflag {
case tar.TypeReg:
err = writeFile(path, tr, header.FileInfo().Mode(), buf)
case tar.TypeDir:
err = os.MkdirAll(path, header.FileInfo().Mode())
case tar.TypeLink:
var target string
if target, err = ensureLinkPath(dir, prefix, path, header.Linkname); err == nil {
err = os.Link(target, path)
}
case tar.TypeSymlink:
var target string
if target, err = ensureLinkPath(dir, prefix, path, header.Linkname); err == nil {
err = os.Symlink(target, path)
}
default:
continue // Non-regular files are skipped
}
if err != nil {
return err
}
// Change access time and modification time if possible (error ignored)
os.Chtimes(path, header.AccessTime, header.ModTime)
}
}
// ensureBasePath ensures the target path is in the base path,
// returning its relative path to the base path.
// target can be either an absolute path or a relative path.
func ensureBasePath(baseAbs, baseRel, target string) (string, error) {
base := baseRel
if filepath.IsAbs(target) {
// ensure base and target are consistent
base = baseAbs
}
path, err := filepath.Rel(base, target)
if err != nil {
return "", err
}
cleanPath := filepath.ToSlash(filepath.Clean(path))
if cleanPath == ".." || strings.HasPrefix(cleanPath, "../") {
return "", fmt.Errorf("%q is outside of %q", target, baseRel)
}
// No symbolic link allowed in the relative path
dir := filepath.Dir(path)
for dir != "." {
if info, err := os.Lstat(filepath.Join(baseAbs, dir)); err != nil {
if !os.IsNotExist(err) {
return "", err
}
} else if info.Mode()&os.ModeSymlink != 0 {
return "", fmt.Errorf("no symbolic link allowed between %q and %q", baseRel, target)
}
dir = filepath.Dir(dir)
}
return path, nil
}
// ensureLinkPath ensures the target path pointed by the link is in the base
// path. It returns target path if validated.
func ensureLinkPath(baseAbs, baseRel, link, target string) (string, error) {
// resolve link
path := target
if !filepath.IsAbs(target) {
path = filepath.Join(filepath.Dir(link), target)
}
// ensure path is under baseAbs or baseRel
if _, err := ensureBasePath(baseAbs, baseRel, path); err != nil {
return "", err
}
return target, nil
}
// writeFile writes content to the file specified by the `path` parameter.
func writeFile(path string, r io.Reader, perm os.FileMode, buf []byte) (err error) {
file, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, perm)
if err != nil {
return err
}
defer func() {
closeErr := file.Close()
if err == nil {
err = closeErr
}
}()
_, err = io.CopyBuffer(file, r, buf)
return err
}

View File

@@ -0,0 +1,106 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package content
import (
"context"
"encoding/json"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2/internal/docker"
)
// PredecessorFinder finds out the nodes directly pointing to a given node of a
// directed acyclic graph.
// In other words, returns the "parents" of the current descriptor.
// PredecessorFinder is an extension of Storage.
type PredecessorFinder interface {
// Predecessors returns the nodes directly pointing to the current node.
Predecessors(ctx context.Context, node ocispec.Descriptor) ([]ocispec.Descriptor, error)
}
// GraphStorage represents a CAS that supports direct predecessor node finding.
type GraphStorage interface {
Storage
PredecessorFinder
}
// ReadOnlyGraphStorage represents a read-only GraphStorage.
type ReadOnlyGraphStorage interface {
ReadOnlyStorage
PredecessorFinder
}
// Successors returns the nodes directly pointed by the current node.
// In other words, returns the "children" of the current descriptor.
func Successors(ctx context.Context, fetcher Fetcher, node ocispec.Descriptor) ([]ocispec.Descriptor, error) {
switch node.MediaType {
case docker.MediaTypeManifest:
content, err := FetchAll(ctx, fetcher, node)
if err != nil {
return nil, err
}
// OCI manifest schema can be used to marshal docker manifest
var manifest ocispec.Manifest
if err := json.Unmarshal(content, &manifest); err != nil {
return nil, err
}
return append([]ocispec.Descriptor{manifest.Config}, manifest.Layers...), nil
case ocispec.MediaTypeImageManifest:
content, err := FetchAll(ctx, fetcher, node)
if err != nil {
return nil, err
}
var manifest ocispec.Manifest
if err := json.Unmarshal(content, &manifest); err != nil {
return nil, err
}
var nodes []ocispec.Descriptor
if manifest.Subject != nil {
nodes = append(nodes, *manifest.Subject)
}
nodes = append(nodes, manifest.Config)
return append(nodes, manifest.Layers...), nil
case docker.MediaTypeManifestList, ocispec.MediaTypeImageIndex:
content, err := FetchAll(ctx, fetcher, node)
if err != nil {
return nil, err
}
// docker manifest list and oci index are equivalent for successors.
var index ocispec.Index
if err := json.Unmarshal(content, &index); err != nil {
return nil, err
}
return index.Manifests, nil
case ocispec.MediaTypeArtifactManifest:
content, err := FetchAll(ctx, fetcher, node)
if err != nil {
return nil, err
}
var manifest ocispec.Artifact
if err := json.Unmarshal(content, &manifest); err != nil {
return nil, err
}
var nodes []ocispec.Descriptor
if manifest.Subject != nil {
nodes = append(nodes, *manifest.Subject)
}
return append(nodes, manifest.Blobs...), nil
}
return nil, nil
}

View File

@@ -0,0 +1,50 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package content
import (
"context"
"fmt"
"io"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2/errdef"
)
// LimitedStorage represents a CAS with a push size limit.
type LimitedStorage struct {
Storage // underlying storage
PushLimit int64 // max size for push
}
// Push pushes the content, matching the expected descriptor.
// The size of the content cannot exceed the push size limit.
func (ls *LimitedStorage) Push(ctx context.Context, expected ocispec.Descriptor, content io.Reader) error {
if expected.Size > ls.PushLimit {
return fmt.Errorf(
"content size %v exceeds push size limit %v: %w",
expected.Size,
ls.PushLimit,
errdef.ErrSizeExceedsLimit)
}
return ls.Storage.Push(ctx, expected, io.LimitReader(content, expected.Size))
}
// LimitStorage returns a storage with a push size limit.
func LimitStorage(s Storage, n int64) *LimitedStorage {
return &LimitedStorage{s, n}
}

View File

@@ -0,0 +1,141 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package content
import (
"errors"
"fmt"
"io"
"github.com/opencontainers/go-digest"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
)
var (
// ErrInvalidDescriptorSize is returned by ReadAll() when
// the descriptor has an invalid size.
ErrInvalidDescriptorSize = errors.New("invalid descriptor size")
// ErrMismatchedDigest is returned by ReadAll() when
// the descriptor has an invalid digest.
ErrMismatchedDigest = errors.New("mismatched digest")
// ErrTrailingData is returned by ReadAll() when
// there exists trailing data unread when the read terminates.
ErrTrailingData = errors.New("trailing data")
)
var (
// errEarlyVerify is returned by VerifyReader.Verify() when
// Verify() is called before completing reading the entire content blob.
errEarlyVerify = errors.New("early verify")
)
// VerifyReader reads the content described by its descriptor and verifies
// against its size and digest.
type VerifyReader struct {
base *io.LimitedReader
verifier digest.Verifier
verified bool
err error
}
// Read reads up to len(p) bytes into p. It returns the number of bytes
// read (0 <= n <= len(p)) and any error encountered.
func (vr *VerifyReader) Read(p []byte) (n int, err error) {
if vr.err != nil {
return 0, vr.err
}
n, err = vr.base.Read(p)
if err != nil {
if err == io.EOF && vr.base.N > 0 {
err = io.ErrUnexpectedEOF
}
vr.err = err
}
return
}
// Verify verifies the read content against the size and the digest.
func (vr *VerifyReader) Verify() error {
if vr.verified {
return nil
}
if vr.err == nil {
if vr.base.N > 0 {
return errEarlyVerify
}
} else if vr.err != io.EOF {
return vr.err
}
if err := ensureEOF(vr.base.R); err != nil {
vr.err = err
return vr.err
}
if !vr.verifier.Verified() {
vr.err = ErrMismatchedDigest
return vr.err
}
vr.verified = true
vr.err = io.EOF
return nil
}
// NewVerifyReader wraps r for reading content with verification against desc.
func NewVerifyReader(r io.Reader, desc ocispec.Descriptor) *VerifyReader {
verifier := desc.Digest.Verifier()
lr := &io.LimitedReader{
R: io.TeeReader(r, verifier),
N: desc.Size,
}
return &VerifyReader{
base: lr,
verifier: verifier,
}
}
// ReadAll safely reads the content described by the descriptor.
// The read content is verified against the size and the digest
// using a VerifyReader.
func ReadAll(r io.Reader, desc ocispec.Descriptor) ([]byte, error) {
if desc.Size < 0 {
return nil, ErrInvalidDescriptorSize
}
buf := make([]byte, desc.Size)
vr := NewVerifyReader(r, desc)
if _, err := io.ReadFull(vr, buf); err != nil {
return nil, fmt.Errorf("read failed: %w", err)
}
if err := vr.Verify(); err != nil {
return nil, err
}
return buf, nil
}
// ensureEOF ensures the read operation ends with an EOF and no
// trailing data is present.
func ensureEOF(r io.Reader) error {
var peek [1]byte
_, err := io.ReadFull(r, peek[:])
if err != io.EOF {
return ErrTrailingData
}
return nil
}

View File

@@ -0,0 +1,41 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package content provides implementations to access content stores.
package content
import (
"context"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
)
// Resolver resolves reference tags.
type Resolver interface {
// Resolve resolves a reference to a descriptor.
Resolve(ctx context.Context, reference string) (ocispec.Descriptor, error)
}
// Tagger tags reference tags.
type Tagger interface {
// Tag tags a descriptor with a reference string.
Tag(ctx context.Context, desc ocispec.Descriptor, reference string) error
}
// TagResolver provides reference tag indexing services.
type TagResolver interface {
Tagger
Resolver
}

View File

@@ -0,0 +1,80 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package content
import (
"context"
"io"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
)
// Fetcher fetches content.
type Fetcher interface {
// Fetch fetches the content identified by the descriptor.
Fetch(ctx context.Context, target ocispec.Descriptor) (io.ReadCloser, error)
}
// Pusher pushes content.
type Pusher interface {
// Push pushes the content, matching the expected descriptor.
// Reader is perferred to Writer so that the suitable buffer size can be
// chosen by the underlying implementation. Furthermore, the implementation
// can also do reflection on the Reader for more advanced I/O optimization.
Push(ctx context.Context, expected ocispec.Descriptor, content io.Reader) error
}
// Storage represents a content-addressable storage (CAS) where contents are
// accessed via Descriptors.
// The storage is designed to handle blobs of large sizes.
type Storage interface {
ReadOnlyStorage
Pusher
}
// ReadOnlyStorage represents a read-only Storage.
type ReadOnlyStorage interface {
Fetcher
// Exists returns true if the described content exists.
Exists(ctx context.Context, target ocispec.Descriptor) (bool, error)
}
// Deleter removes content.
// Deleter is an extension of Storage.
type Deleter interface {
// Delete removes the content identified by the descriptor.
Delete(ctx context.Context, target ocispec.Descriptor) error
}
// FetchAll safely fetches the content described by the descriptor.
// The fetched content is verified against the size and the digest.
func FetchAll(ctx context.Context, fetcher Fetcher, desc ocispec.Descriptor) ([]byte, error) {
rc, err := fetcher.Fetch(ctx, desc)
if err != nil {
return nil, err
}
defer rc.Close()
return ReadAll(rc, desc)
}
// FetcherFunc is the basic Fetch method defined in Fetcher.
type FetcherFunc func(ctx context.Context, target ocispec.Descriptor) (io.ReadCloser, error)
// Fetch performs Fetch operation by the FetcherFunc.
func (fn FetcherFunc) Fetch(ctx context.Context, target ocispec.Descriptor) (io.ReadCloser, error) {
return fn(ctx, target)
}

426
vendor/oras.land/oras-go/v2/copy.go vendored Normal file
View File

@@ -0,0 +1,426 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package oras
import (
"context"
"errors"
"fmt"
"io"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"golang.org/x/sync/semaphore"
"oras.land/oras-go/v2/content"
"oras.land/oras-go/v2/errdef"
"oras.land/oras-go/v2/internal/cas"
"oras.land/oras-go/v2/internal/descriptor"
"oras.land/oras-go/v2/internal/platform"
"oras.land/oras-go/v2/internal/registryutil"
"oras.land/oras-go/v2/internal/status"
"oras.land/oras-go/v2/internal/syncutil"
"oras.land/oras-go/v2/registry"
)
// defaultConcurrency is the default value of CopyGraphOptions.Concurrency.
const defaultConcurrency int = 3 // This value is consistent with dockerd and containerd.
// errSkipDesc signals copyNode() to stop processing a descriptor.
var errSkipDesc = errors.New("skip descriptor")
// DefaultCopyOptions provides the default CopyOptions.
var DefaultCopyOptions CopyOptions = CopyOptions{
CopyGraphOptions: DefaultCopyGraphOptions,
}
// CopyOptions contains parameters for [oras.Copy].
type CopyOptions struct {
CopyGraphOptions
// MapRoot maps the resolved root node to a desired root node for copy.
// When MapRoot is provided, the descriptor resolved from the source
// reference will be passed to MapRoot, and the mapped descriptor will be
// used as the root node for copy.
MapRoot func(ctx context.Context, src content.ReadOnlyStorage, root ocispec.Descriptor) (ocispec.Descriptor, error)
}
// WithTargetPlatform configures opts.MapRoot to select the manifest whose
// platform matches the given platform. When MapRoot is provided, the platform
// selection will be applied on the mapped root node.
// - If the given platform is nil, no platform selection will be applied.
// - If the root node is a manifest, it will remain the same if platform
// matches, otherwise ErrNotFound will be returned.
// - If the root node is a manifest list, it will be mapped to the first
// matching manifest if exists, otherwise ErrNotFound will be returned.
// - Otherwise ErrUnsupported will be returned.
func (opts *CopyOptions) WithTargetPlatform(p *ocispec.Platform) {
if p == nil {
return
}
mapRoot := opts.MapRoot
opts.MapRoot = func(ctx context.Context, src content.ReadOnlyStorage, root ocispec.Descriptor) (desc ocispec.Descriptor, err error) {
if mapRoot != nil {
if root, err = mapRoot(ctx, src, root); err != nil {
return ocispec.Descriptor{}, err
}
}
return platform.SelectManifest(ctx, src, root, p)
}
}
// defaultCopyMaxMetadataBytes is the default value of
// CopyGraphOptions.MaxMetadataBytes.
const defaultCopyMaxMetadataBytes int64 = 4 * 1024 * 1024 // 4 MiB
// DefaultCopyGraphOptions provides the default CopyGraphOptions.
var DefaultCopyGraphOptions CopyGraphOptions
// CopyGraphOptions contains parameters for [oras.CopyGraph].
type CopyGraphOptions struct {
// Concurrency limits the maximum number of concurrent copy tasks.
// If less than or equal to 0, a default (currently 3) is used.
Concurrency int
// MaxMetadataBytes limits the maximum size of the metadata that can be
// cached in the memory.
// If less than or equal to 0, a default (currently 4 MiB) is used.
MaxMetadataBytes int64
// PreCopy handles the current descriptor before copying it.
PreCopy func(ctx context.Context, desc ocispec.Descriptor) error
// PostCopy handles the current descriptor after copying it.
PostCopy func(ctx context.Context, desc ocispec.Descriptor) error
// OnCopySkipped will be called when the sub-DAG rooted by the current node
// is skipped.
OnCopySkipped func(ctx context.Context, desc ocispec.Descriptor) error
// FindSuccessors finds the successors of the current node.
// fetcher provides cached access to the source storage, and is suitable
// for fetching non-leaf nodes like manifests. Since anything fetched from
// fetcher will be cached in the memory, it is recommended to use original
// source storage to fetch large blobs.
// If FindSuccessors is nil, content.Successors will be used.
FindSuccessors func(ctx context.Context, fetcher content.Fetcher, desc ocispec.Descriptor) ([]ocispec.Descriptor, error)
}
// Copy copies a rooted directed acyclic graph (DAG) with the tagged root node
// in the source Target to the destination Target.
// The destination reference will be the same as the source reference if the
// destination reference is left blank.
//
// Returns the descriptor of the root node on successful copy.
func Copy(ctx context.Context, src ReadOnlyTarget, srcRef string, dst Target, dstRef string, opts CopyOptions) (ocispec.Descriptor, error) {
if src == nil {
return ocispec.Descriptor{}, errors.New("nil source target")
}
if dst == nil {
return ocispec.Descriptor{}, errors.New("nil destination target")
}
if dstRef == "" {
dstRef = srcRef
}
// use caching proxy on non-leaf nodes
if opts.MaxMetadataBytes <= 0 {
opts.MaxMetadataBytes = defaultCopyMaxMetadataBytes
}
proxy := cas.NewProxyWithLimit(src, cas.NewMemory(), opts.MaxMetadataBytes)
root, err := resolveRoot(ctx, src, srcRef, proxy)
if err != nil {
return ocispec.Descriptor{}, fmt.Errorf("failed to resolve %s: %w", srcRef, err)
}
if opts.MapRoot != nil {
proxy.StopCaching = true
root, err = opts.MapRoot(ctx, proxy, root)
if err != nil {
return ocispec.Descriptor{}, err
}
proxy.StopCaching = false
}
if err := prepareCopy(ctx, dst, dstRef, proxy, root, &opts); err != nil {
return ocispec.Descriptor{}, err
}
if err := copyGraph(ctx, src, dst, root, proxy, nil, nil, opts.CopyGraphOptions); err != nil {
return ocispec.Descriptor{}, err
}
return root, nil
}
// CopyGraph copies a rooted directed acyclic graph (DAG) from the source CAS to
// the destination CAS.
func CopyGraph(ctx context.Context, src content.ReadOnlyStorage, dst content.Storage, root ocispec.Descriptor, opts CopyGraphOptions) error {
return copyGraph(ctx, src, dst, root, nil, nil, nil, opts)
}
// copyGraph copies a rooted directed acyclic graph (DAG) from the source CAS to
// the destination CAS with specified caching, concurrency limiter and tracker.
func copyGraph(ctx context.Context, src content.ReadOnlyStorage, dst content.Storage, root ocispec.Descriptor,
proxy *cas.Proxy, limiter *semaphore.Weighted, tracker *status.Tracker, opts CopyGraphOptions) error {
if proxy == nil {
// use caching proxy on non-leaf nodes
if opts.MaxMetadataBytes <= 0 {
opts.MaxMetadataBytes = defaultCopyMaxMetadataBytes
}
proxy = cas.NewProxyWithLimit(src, cas.NewMemory(), opts.MaxMetadataBytes)
}
if limiter == nil {
// if Concurrency is not set or invalid, use the default concurrency
if opts.Concurrency <= 0 {
opts.Concurrency = defaultConcurrency
}
limiter = semaphore.NewWeighted(int64(opts.Concurrency))
}
if tracker == nil {
// track content status
tracker = status.NewTracker()
}
// if FindSuccessors is not provided, use the default one
if opts.FindSuccessors == nil {
opts.FindSuccessors = content.Successors
}
// traverse the graph
var fn syncutil.GoFunc[ocispec.Descriptor]
fn = func(ctx context.Context, region *syncutil.LimitedRegion, desc ocispec.Descriptor) (err error) {
// skip the descriptor if other go routine is working on it
done, committed := tracker.TryCommit(desc)
if !committed {
return nil
}
defer func() {
if err == nil {
// mark the content as done on success
close(done)
}
}()
// skip if a rooted sub-DAG exists
exists, err := dst.Exists(ctx, desc)
if err != nil {
return err
}
if exists {
if opts.OnCopySkipped != nil {
if err := opts.OnCopySkipped(ctx, desc); err != nil {
return err
}
}
return nil
}
// find successors while non-leaf nodes will be fetched and cached
successors, err := opts.FindSuccessors(ctx, proxy, desc)
if err != nil {
return err
}
successors = removeForeignLayers(successors)
if len(successors) != 0 {
// for non-leaf nodes, process successors and wait for them to complete
region.End()
if err := syncutil.Go(ctx, limiter, fn, successors...); err != nil {
return err
}
for _, node := range successors {
done, committed := tracker.TryCommit(node)
if committed {
return fmt.Errorf("%s: %s: successor not committed", desc.Digest, node.Digest)
}
select {
case <-done:
case <-ctx.Done():
return ctx.Err()
}
}
if err := region.Start(); err != nil {
return err
}
}
exists, err = proxy.Cache.Exists(ctx, desc)
if err != nil {
return err
}
if exists {
return copyNode(ctx, proxy.Cache, dst, desc, opts)
}
return copyNode(ctx, src, dst, desc, opts)
}
return syncutil.Go(ctx, limiter, fn, root)
}
// doCopyNode copies a single content from the source CAS to the destination CAS.
func doCopyNode(ctx context.Context, src content.ReadOnlyStorage, dst content.Storage, desc ocispec.Descriptor) error {
rc, err := src.Fetch(ctx, desc)
if err != nil {
return err
}
defer rc.Close()
err = dst.Push(ctx, desc, rc)
if err != nil && !errors.Is(err, errdef.ErrAlreadyExists) {
return err
}
return nil
}
// copyNode copies a single content from the source CAS to the destination CAS,
// and apply the given options.
func copyNode(ctx context.Context, src content.ReadOnlyStorage, dst content.Storage, desc ocispec.Descriptor, opts CopyGraphOptions) error {
if opts.PreCopy != nil {
if err := opts.PreCopy(ctx, desc); err != nil {
if err == errSkipDesc {
return nil
}
return err
}
}
if err := doCopyNode(ctx, src, dst, desc); err != nil {
return err
}
if opts.PostCopy != nil {
return opts.PostCopy(ctx, desc)
}
return nil
}
// copyCachedNodeWithReference copies a single content with a reference from the
// source cache to the destination ReferencePusher.
func copyCachedNodeWithReference(ctx context.Context, src *cas.Proxy, dst registry.ReferencePusher, desc ocispec.Descriptor, dstRef string) error {
rc, err := src.FetchCached(ctx, desc)
if err != nil {
return err
}
defer rc.Close()
err = dst.PushReference(ctx, desc, rc, dstRef)
if err != nil && !errors.Is(err, errdef.ErrAlreadyExists) {
return err
}
return nil
}
// resolveRoot resolves the source reference to the root node.
func resolveRoot(ctx context.Context, src ReadOnlyTarget, srcRef string, proxy *cas.Proxy) (ocispec.Descriptor, error) {
refFetcher, ok := src.(registry.ReferenceFetcher)
if !ok {
return src.Resolve(ctx, srcRef)
}
// optimize performance for ReferenceFetcher targets
refProxy := &registryutil.Proxy{
ReferenceFetcher: refFetcher,
Proxy: proxy,
}
root, rc, err := refProxy.FetchReference(ctx, srcRef)
if err != nil {
return ocispec.Descriptor{}, err
}
defer rc.Close()
// cache root if it is a non-leaf node
fetcher := content.FetcherFunc(func(ctx context.Context, target ocispec.Descriptor) (io.ReadCloser, error) {
if content.Equal(target, root) {
return rc, nil
}
return nil, errors.New("fetching only root node expected")
})
if _, err = content.Successors(ctx, fetcher, root); err != nil {
return ocispec.Descriptor{}, err
}
// TODO: optimize special case where root is a leaf node (i.e. a blob)
// and dst is a ReferencePusher.
return root, nil
}
// prepareCopy prepares the hooks for copy.
func prepareCopy(ctx context.Context, dst Target, dstRef string, proxy *cas.Proxy, root ocispec.Descriptor, opts *CopyOptions) error {
if refPusher, ok := dst.(registry.ReferencePusher); ok {
// optimize performance for ReferencePusher targets
preCopy := opts.PreCopy
opts.PreCopy = func(ctx context.Context, desc ocispec.Descriptor) error {
if preCopy != nil {
if err := preCopy(ctx, desc); err != nil {
return err
}
}
if !content.Equal(desc, root) {
// for non-root node, do nothing
return nil
}
// for root node, prepare optimized copy
if err := copyCachedNodeWithReference(ctx, proxy, refPusher, desc, dstRef); err != nil {
return err
}
if opts.PostCopy != nil {
if err := opts.PostCopy(ctx, desc); err != nil {
return err
}
}
// skip the regular copy workflow
return errSkipDesc
}
} else {
postCopy := opts.PostCopy
opts.PostCopy = func(ctx context.Context, desc ocispec.Descriptor) error {
if content.Equal(desc, root) {
// for root node, tag it after copying it
if err := dst.Tag(ctx, root, dstRef); err != nil {
return err
}
}
if postCopy != nil {
return postCopy(ctx, desc)
}
return nil
}
}
onCopySkipped := opts.OnCopySkipped
opts.OnCopySkipped = func(ctx context.Context, desc ocispec.Descriptor) error {
if onCopySkipped != nil {
if err := onCopySkipped(ctx, desc); err != nil {
return err
}
}
if !content.Equal(desc, root) {
return nil
}
// enforce tagging when root is skipped
if refPusher, ok := dst.(registry.ReferencePusher); ok {
return copyCachedNodeWithReference(ctx, proxy, refPusher, desc, dstRef)
}
return dst.Tag(ctx, root, dstRef)
}
return nil
}
// removeForeignLayers in-place removes all foreign layers in the given slice.
func removeForeignLayers(descs []ocispec.Descriptor) []ocispec.Descriptor {
var j int
for i, desc := range descs {
if !descriptor.IsForeignLayer(desc) {
if i != j {
descs[j] = desc
}
j++
}
}
return descs[:j]
}

View File

@@ -0,0 +1,30 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package errdef
import "errors"
// Common errors used in ORAS
var (
ErrAlreadyExists = errors.New("already exists")
ErrInvalidDigest = errors.New("invalid digest")
ErrInvalidReference = errors.New("invalid reference")
ErrMissingReference = errors.New("missing reference")
ErrNotFound = errors.New("not found")
ErrSizeExceedsLimit = errors.New("size exceeds limit")
ErrUnsupported = errors.New("unsupported")
ErrUnsupportedVersion = errors.New("unsupported version")
)

View File

@@ -0,0 +1,388 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package oras
import (
"context"
"encoding/json"
"errors"
"regexp"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"golang.org/x/sync/semaphore"
"oras.land/oras-go/v2/content"
"oras.land/oras-go/v2/internal/cas"
"oras.land/oras-go/v2/internal/container/set"
"oras.land/oras-go/v2/internal/copyutil"
"oras.land/oras-go/v2/internal/descriptor"
"oras.land/oras-go/v2/internal/docker"
"oras.land/oras-go/v2/internal/status"
"oras.land/oras-go/v2/internal/syncutil"
"oras.land/oras-go/v2/registry"
)
// DefaultExtendedCopyOptions provides the default ExtendedCopyOptions.
var DefaultExtendedCopyOptions ExtendedCopyOptions = ExtendedCopyOptions{
ExtendedCopyGraphOptions: DefaultExtendedCopyGraphOptions,
}
// ExtendedCopyOptions contains parameters for [oras.ExtendedCopy].
type ExtendedCopyOptions struct {
ExtendedCopyGraphOptions
}
// DefaultExtendedCopyGraphOptions provides the default ExtendedCopyGraphOptions.
var DefaultExtendedCopyGraphOptions ExtendedCopyGraphOptions = ExtendedCopyGraphOptions{
CopyGraphOptions: DefaultCopyGraphOptions,
}
// ExtendedCopyGraphOptions contains parameters for [oras.ExtendedCopyGraph].
type ExtendedCopyGraphOptions struct {
CopyGraphOptions
// Depth limits the maximum depth of the directed acyclic graph (DAG) that
// will be extended-copied.
// If Depth is no specified, or the specified value is less than or
// equal to 0, the depth limit will be considered as infinity.
Depth int
// FindPredecessors finds the predecessors of the current node.
// If FindPredecessors is nil, src.Predecessors will be adapted and used.
FindPredecessors func(ctx context.Context, src content.ReadOnlyGraphStorage, desc ocispec.Descriptor) ([]ocispec.Descriptor, error)
}
// ExtendedCopy copies the directed acyclic graph (DAG) that are reachable from
// the given tagged node from the source GraphTarget to the destination Target.
// The destination reference will be the same as the source reference if the
// destination reference is left blank.
//
// Returns the descriptor of the tagged node on successful copy.
func ExtendedCopy(ctx context.Context, src ReadOnlyGraphTarget, srcRef string, dst Target, dstRef string, opts ExtendedCopyOptions) (ocispec.Descriptor, error) {
if src == nil {
return ocispec.Descriptor{}, errors.New("nil source graph target")
}
if dst == nil {
return ocispec.Descriptor{}, errors.New("nil destination target")
}
if dstRef == "" {
dstRef = srcRef
}
node, err := src.Resolve(ctx, srcRef)
if err != nil {
return ocispec.Descriptor{}, err
}
if err := ExtendedCopyGraph(ctx, src, dst, node, opts.ExtendedCopyGraphOptions); err != nil {
return ocispec.Descriptor{}, err
}
if err := dst.Tag(ctx, node, dstRef); err != nil {
return ocispec.Descriptor{}, err
}
return node, nil
}
// ExtendedCopyGraph copies the directed acyclic graph (DAG) that are reachable
// from the given node from the source GraphStorage to the destination Storage.
func ExtendedCopyGraph(ctx context.Context, src content.ReadOnlyGraphStorage, dst content.Storage, node ocispec.Descriptor, opts ExtendedCopyGraphOptions) error {
roots, err := findRoots(ctx, src, node, opts)
if err != nil {
return err
}
// if Concurrency is not set or invalid, use the default concurrency
if opts.Concurrency <= 0 {
opts.Concurrency = defaultConcurrency
}
limiter := semaphore.NewWeighted(int64(opts.Concurrency))
// use caching proxy on non-leaf nodes
if opts.MaxMetadataBytes <= 0 {
opts.MaxMetadataBytes = defaultCopyMaxMetadataBytes
}
proxy := cas.NewProxyWithLimit(src, cas.NewMemory(), opts.MaxMetadataBytes)
// track content status
tracker := status.NewTracker()
// copy the sub-DAGs rooted by the root nodes
return syncutil.Go(ctx, limiter, func(ctx context.Context, region *syncutil.LimitedRegion, root ocispec.Descriptor) error {
// As a root can be a predecessor of other roots, release the limit here
// for dispatching, to avoid dead locks where predecessor roots are
// handled first and are waiting for its successors to complete.
region.End()
if err := copyGraph(ctx, src, dst, root, proxy, limiter, tracker, opts.CopyGraphOptions); err != nil {
return err
}
return region.Start()
}, roots...)
}
// findRoots finds the root nodes reachable from the given node through a
// depth-first search.
func findRoots(ctx context.Context, storage content.ReadOnlyGraphStorage, node ocispec.Descriptor, opts ExtendedCopyGraphOptions) ([]ocispec.Descriptor, error) {
visited := set.New[descriptor.Descriptor]()
rootMap := make(map[descriptor.Descriptor]ocispec.Descriptor)
addRoot := func(key descriptor.Descriptor, val ocispec.Descriptor) {
if _, exists := rootMap[key]; !exists {
rootMap[key] = val
}
}
// if FindPredecessors is not provided, use the default one
if opts.FindPredecessors == nil {
opts.FindPredecessors = func(ctx context.Context, src content.ReadOnlyGraphStorage, desc ocispec.Descriptor) ([]ocispec.Descriptor, error) {
return src.Predecessors(ctx, desc)
}
}
var stack copyutil.Stack
// push the initial node to the stack, set the depth to 0
stack.Push(copyutil.NodeInfo{Node: node, Depth: 0})
for {
current, ok := stack.Pop()
if !ok {
// empty stack
break
}
currentNode := current.Node
currentKey := descriptor.FromOCI(currentNode)
if visited.Contains(currentKey) {
// skip the current node if it has been visited
continue
}
visited.Add(currentKey)
// stop finding predecessors if the target depth is reached
if opts.Depth > 0 && current.Depth == opts.Depth {
addRoot(currentKey, currentNode)
continue
}
predecessors, err := opts.FindPredecessors(ctx, storage, currentNode)
if err != nil {
return nil, err
}
// The current node has no predecessor node,
// which means it is a root node of a sub-DAG.
if len(predecessors) == 0 {
addRoot(currentKey, currentNode)
continue
}
// The current node has predecessor nodes, which means it is NOT a root node.
// Push the predecessor nodes to the stack and keep finding from there.
for _, predecessor := range predecessors {
predecessorKey := descriptor.FromOCI(predecessor)
if !visited.Contains(predecessorKey) {
// push the predecessor node with increased depth
stack.Push(copyutil.NodeInfo{Node: predecessor, Depth: current.Depth + 1})
}
}
}
roots := make([]ocispec.Descriptor, 0, len(rootMap))
for _, root := range rootMap {
roots = append(roots, root)
}
return roots, nil
}
// FilterAnnotation configures opts.FindPredecessors to filter the predecessors
// whose annotation matches a given regex pattern.
//
// A predecessor is kept if key is in its annotations and the annotation value
// matches regex.
// If regex is nil, predecessors whose annotations contain key will be kept,
// no matter of the annotation value.
//
// For performance consideration, when using both FilterArtifactType and
// FilterAnnotation, it's recommended to call FilterArtifactType first.
func (opts *ExtendedCopyGraphOptions) FilterAnnotation(key string, regex *regexp.Regexp) {
keep := func(desc ocispec.Descriptor) bool {
value, ok := desc.Annotations[key]
return ok && (regex == nil || regex.MatchString(value))
}
fp := opts.FindPredecessors
opts.FindPredecessors = func(ctx context.Context, src content.ReadOnlyGraphStorage, desc ocispec.Descriptor) ([]ocispec.Descriptor, error) {
var predecessors []ocispec.Descriptor
var err error
if fp == nil {
if rf, ok := src.(registry.ReferrerLister); ok {
// if src is a ReferrerLister, use Referrers() for possible memory saving
if err := rf.Referrers(ctx, desc, "", func(referrers []ocispec.Descriptor) error {
// for each page of the results, filter the referrers
for _, r := range referrers {
if keep(r) {
predecessors = append(predecessors, r)
}
}
return nil
}); err != nil {
return nil, err
}
return predecessors, nil
}
predecessors, err = src.Predecessors(ctx, desc)
} else {
predecessors, err = fp(ctx, src, desc)
}
if err != nil {
return nil, err
}
// Predecessor descriptors that are not from Referrers API are not
// guaranteed to include the annotations of the corresponding manifests.
var kept []ocispec.Descriptor
for _, p := range predecessors {
if p.Annotations == nil {
// If the annotations are not present in the descriptors,
// fetch it from the manifest content.
switch p.MediaType {
case docker.MediaTypeManifest, ocispec.MediaTypeImageManifest,
docker.MediaTypeManifestList, ocispec.MediaTypeImageIndex,
ocispec.MediaTypeArtifactManifest:
annotations, err := fetchAnnotations(ctx, src, p)
if err != nil {
return nil, err
}
p.Annotations = annotations
}
}
if keep(p) {
kept = append(kept, p)
}
}
return kept, nil
}
}
// fetchAnnotations fetches the annotations of the manifest described by desc.
func fetchAnnotations(ctx context.Context, src content.ReadOnlyGraphStorage, desc ocispec.Descriptor) (map[string]string, error) {
rc, err := src.Fetch(ctx, desc)
if err != nil {
return nil, err
}
defer rc.Close()
var manifest struct {
Annotations map[string]string `json:"annotations"`
}
if err := json.NewDecoder(rc).Decode(&manifest); err != nil {
return nil, err
}
if manifest.Annotations == nil {
// to differentiate with nil
return make(map[string]string), nil
}
return manifest.Annotations, nil
}
// FilterArtifactType configures opts.FindPredecessors to filter the
// predecessors whose artifact type matches a given regex pattern.
//
// A predecessor is kept if its artifact type matches regex.
// If regex is nil, all predecessors will be kept.
//
// For performance consideration, when using both FilterArtifactType and
// FilterAnnotation, it's recommended to call FilterArtifactType first.
func (opts *ExtendedCopyGraphOptions) FilterArtifactType(regex *regexp.Regexp) {
if regex == nil {
return
}
keep := func(desc ocispec.Descriptor) bool {
return regex.MatchString(desc.ArtifactType)
}
fp := opts.FindPredecessors
opts.FindPredecessors = func(ctx context.Context, src content.ReadOnlyGraphStorage, desc ocispec.Descriptor) ([]ocispec.Descriptor, error) {
var predecessors []ocispec.Descriptor
var err error
if fp == nil {
if rf, ok := src.(registry.ReferrerLister); ok {
// if src is a ReferrerLister, use Referrers() for possible memory saving
if err := rf.Referrers(ctx, desc, "", func(referrers []ocispec.Descriptor) error {
// for each page of the results, filter the referrers
for _, r := range referrers {
if keep(r) {
predecessors = append(predecessors, r)
}
}
return nil
}); err != nil {
return nil, err
}
return predecessors, nil
}
predecessors, err = src.Predecessors(ctx, desc)
} else {
predecessors, err = fp(ctx, src, desc)
}
if err != nil {
return nil, err
}
// predecessor descriptors that are not from Referrers API are not
// guaranteed to include the artifact type of the corresponding
// manifests.
var kept []ocispec.Descriptor
for _, p := range predecessors {
if p.ArtifactType == "" {
// if the artifact type is not present in the descriptors,
// fetch it from the manifest content.
switch p.MediaType {
case ocispec.MediaTypeArtifactManifest, ocispec.MediaTypeImageManifest:
artifactType, err := fetchArtifactType(ctx, src, p)
if err != nil {
return nil, err
}
p.ArtifactType = artifactType
}
}
if keep(p) {
kept = append(kept, p)
}
}
return kept, nil
}
}
// fetchArtifactType fetches the artifact type of the manifest described by desc.
func fetchArtifactType(ctx context.Context, src content.ReadOnlyGraphStorage, desc ocispec.Descriptor) (string, error) {
rc, err := src.Fetch(ctx, desc)
if err != nil {
return "", err
}
defer rc.Close()
switch desc.MediaType {
case ocispec.MediaTypeArtifactManifest:
var manifest ocispec.Artifact
if err := json.NewDecoder(rc).Decode(&manifest); err != nil {
return "", err
}
return manifest.ArtifactType, nil
case ocispec.MediaTypeImageManifest:
var manifest ocispec.Manifest
if err := json.NewDecoder(rc).Decode(&manifest); err != nil {
return "", err
}
return manifest.Config.MediaType, nil
default:
return "", nil
}
}

View File

@@ -0,0 +1,88 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package cas
import (
"bytes"
"context"
"fmt"
"io"
"sync"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
contentpkg "oras.land/oras-go/v2/content"
"oras.land/oras-go/v2/errdef"
"oras.land/oras-go/v2/internal/descriptor"
)
// Memory is a memory based CAS.
type Memory struct {
content sync.Map // map[descriptor.Descriptor][]byte
}
// NewMemory creates a new Memory CAS.
func NewMemory() *Memory {
return &Memory{}
}
// Fetch fetches the content identified by the descriptor.
func (m *Memory) Fetch(_ context.Context, target ocispec.Descriptor) (io.ReadCloser, error) {
key := descriptor.FromOCI(target)
content, exists := m.content.Load(key)
if !exists {
return nil, fmt.Errorf("%s: %s: %w", key.Digest, key.MediaType, errdef.ErrNotFound)
}
return io.NopCloser(bytes.NewReader(content.([]byte))), nil
}
// Push pushes the content, matching the expected descriptor.
func (m *Memory) Push(_ context.Context, expected ocispec.Descriptor, content io.Reader) error {
key := descriptor.FromOCI(expected)
// check if the content exists in advance to avoid reading from the content.
if _, exists := m.content.Load(key); exists {
return fmt.Errorf("%s: %s: %w", key.Digest, key.MediaType, errdef.ErrAlreadyExists)
}
// read and try to store the content.
value, err := contentpkg.ReadAll(content, expected)
if err != nil {
return err
}
if _, exists := m.content.LoadOrStore(key, value); exists {
return fmt.Errorf("%s: %s: %w", key.Digest, key.MediaType, errdef.ErrAlreadyExists)
}
return nil
}
// Exists returns true if the described content exists.
func (m *Memory) Exists(_ context.Context, target ocispec.Descriptor) (bool, error) {
key := descriptor.FromOCI(target)
_, exists := m.content.Load(key)
return exists, nil
}
// Map dumps the memory into a built-in map structure.
// Like other operations, calling Map() is go-routine safe. However, it does not
// necessarily correspond to any consistent snapshot of the storage contents.
func (m *Memory) Map() map[descriptor.Descriptor][]byte {
res := make(map[descriptor.Descriptor][]byte)
m.content.Range(func(key, value interface{}) bool {
res[key.(descriptor.Descriptor)] = value.([]byte)
return true
})
return res
}

View File

@@ -0,0 +1,125 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package cas
import (
"context"
"io"
"sync"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2/content"
"oras.land/oras-go/v2/internal/ioutil"
)
// Proxy is a caching proxy for the storage.
// The first fetch call of a described content will read from the remote and
// cache the fetched content.
// The subsequent fetch call will read from the local cache.
type Proxy struct {
content.ReadOnlyStorage
Cache content.Storage
StopCaching bool
}
// NewProxy creates a proxy for the `base` storage, using the `cache` storage as
// the cache.
func NewProxy(base content.ReadOnlyStorage, cache content.Storage) *Proxy {
return &Proxy{
ReadOnlyStorage: base,
Cache: cache,
}
}
// NewProxyWithLimit creates a proxy for the `base` storage, using the `cache`
// storage with a push size limit as the cache.
func NewProxyWithLimit(base content.ReadOnlyStorage, cache content.Storage, pushLimit int64) *Proxy {
limitedCache := content.LimitStorage(cache, pushLimit)
return &Proxy{
ReadOnlyStorage: base,
Cache: limitedCache,
}
}
// Fetch fetches the content identified by the descriptor.
func (p *Proxy) Fetch(ctx context.Context, target ocispec.Descriptor) (io.ReadCloser, error) {
if p.StopCaching {
return p.FetchCached(ctx, target)
}
rc, err := p.Cache.Fetch(ctx, target)
if err == nil {
return rc, nil
}
rc, err = p.ReadOnlyStorage.Fetch(ctx, target)
if err != nil {
return nil, err
}
pr, pw := io.Pipe()
var wg sync.WaitGroup
wg.Add(1)
var pushErr error
go func() {
defer wg.Done()
pushErr = p.Cache.Push(ctx, target, pr)
if pushErr != nil {
pr.CloseWithError(pushErr)
}
}()
closer := ioutil.CloserFunc(func() error {
rcErr := rc.Close()
if err := pw.Close(); err != nil {
return err
}
wg.Wait()
if pushErr != nil {
return pushErr
}
return rcErr
})
return struct {
io.Reader
io.Closer
}{
Reader: io.TeeReader(rc, pw),
Closer: closer,
}, nil
}
// FetchCached fetches the content identified by the descriptor.
// If the content is not cached, it will be fetched from the remote without
// caching.
func (p *Proxy) FetchCached(ctx context.Context, target ocispec.Descriptor) (io.ReadCloser, error) {
exists, err := p.Cache.Exists(ctx, target)
if err != nil {
return nil, err
}
if exists {
return p.Cache.Fetch(ctx, target)
}
return p.ReadOnlyStorage.Fetch(ctx, target)
}
// Exists returns true if the described content exists.
func (p *Proxy) Exists(ctx context.Context, target ocispec.Descriptor) (bool, error) {
exists, err := p.Cache.Exists(ctx, target)
if err == nil && exists {
return true, nil
}
return p.ReadOnlyStorage.Exists(ctx, target)
}

View File

@@ -0,0 +1,35 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package set
// Set represents a set data structure.
type Set[T comparable] map[T]struct{}
// New returns an initialized set.
func New[T comparable]() Set[T] {
return make(Set[T])
}
// Add adds item into the set s.
func (s Set[T]) Add(item T) {
s[item] = struct{}{}
}
// Contains returns true if the set s contains item.
func (s Set[T]) Contains(item T) bool {
_, ok := s[item]
return ok
}

View File

@@ -0,0 +1,55 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package copyutil
import (
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
)
// NodeInfo represents information of a node that is being visited in
// ExtendedCopy.
type NodeInfo struct {
// Node represents a node in the graph.
Node ocispec.Descriptor
// Depth represents the depth of the node in the graph.
Depth int
}
// Stack represents a stack data structure that is used in ExtendedCopy for
// storing node information.
type Stack []NodeInfo
// IsEmpty returns true if the stack is empty, otherwise returns false.
func (s *Stack) IsEmpty() bool {
return len(*s) == 0
}
// Push pushes an item to the stack.
func (s *Stack) Push(i NodeInfo) {
*s = append(*s, i)
}
// Pop pops the top item out of the stack.
func (s *Stack) Pop() (NodeInfo, bool) {
if s.IsEmpty() {
return NodeInfo{}, false
}
last := len(*s) - 1
top := (*s)[last]
*s = (*s)[:last]
return top, true
}

View File

@@ -0,0 +1,88 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package descriptor
import (
"github.com/opencontainers/go-digest"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2/internal/docker"
)
// DefaultMediaType is the media type used when no media type is specified.
const DefaultMediaType string = "application/octet-stream"
// Descriptor contains the minimun information to describe the disposition of
// targeted content.
// Since it only has strings and integers, Descriptor is a comparable struct.
type Descriptor struct {
// MediaType is the media type of the object this schema refers to.
MediaType string `json:"mediaType,omitempty"`
// Digest is the digest of the targeted content.
Digest digest.Digest `json:"digest"`
// Size specifies the size in bytes of the blob.
Size int64 `json:"size"`
}
// Empty is an empty descriptor
var Empty Descriptor
// FromOCI shrinks the OCI descriptor to the minimum.
func FromOCI(desc ocispec.Descriptor) Descriptor {
return Descriptor{
MediaType: desc.MediaType,
Digest: desc.Digest,
Size: desc.Size,
}
}
// IsForeignLayer checks if a descriptor describes a foreign layer.
func IsForeignLayer(desc ocispec.Descriptor) bool {
switch desc.MediaType {
case ocispec.MediaTypeImageLayerNonDistributable,
ocispec.MediaTypeImageLayerNonDistributableGzip,
ocispec.MediaTypeImageLayerNonDistributableZstd,
docker.MediaTypeForeignLayer:
return true
default:
return false
}
}
// IsManifest checks if a descriptor describes a manifest.
func IsManifest(desc ocispec.Descriptor) bool {
switch desc.MediaType {
case docker.MediaTypeManifest,
docker.MediaTypeManifestList,
ocispec.MediaTypeImageManifest,
ocispec.MediaTypeImageIndex,
ocispec.MediaTypeArtifactManifest:
return true
default:
return false
}
}
// Plain returns a plain descriptor that contains only MediaType, Digest and
// Size.
func Plain(desc ocispec.Descriptor) ocispec.Descriptor {
return ocispec.Descriptor{
MediaType: desc.MediaType,
Digest: desc.Digest,
Size: desc.Size,
}
}

View File

@@ -0,0 +1,24 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package docker
// docker media types
const (
MediaTypeConfig = "application/vnd.docker.container.image.v1+json"
MediaTypeManifestList = "application/vnd.docker.distribution.manifest.list.v2+json"
MediaTypeManifest = "application/vnd.docker.distribution.manifest.v2+json"
MediaTypeForeignLayer = "application/vnd.docker.image.rootfs.foreign.diff.tar.gzip"
)

View File

@@ -0,0 +1,134 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package graph
import (
"context"
"errors"
"sync"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2/content"
"oras.land/oras-go/v2/errdef"
"oras.land/oras-go/v2/internal/descriptor"
"oras.land/oras-go/v2/internal/status"
"oras.land/oras-go/v2/internal/syncutil"
)
// Memory is a memory based PredecessorFinder.
type Memory struct {
predecessors sync.Map // map[descriptor.Descriptor]map[descriptor.Descriptor]ocispec.Descriptor
indexed sync.Map // map[descriptor.Descriptor]any
}
// NewMemory creates a new memory PredecessorFinder.
func NewMemory() *Memory {
return &Memory{}
}
// Index indexes predecessors for each direct successor of the given node.
// There is no data consistency issue as long as deletion is not implemented
// for the underlying storage.
func (m *Memory) Index(ctx context.Context, fetcher content.Fetcher, node ocispec.Descriptor) error {
successors, err := content.Successors(ctx, fetcher, node)
if err != nil {
return err
}
m.index(ctx, node, successors)
return nil
}
// Index indexes predecessors for all the successors of the given node.
// There is no data consistency issue as long as deletion is not implemented
// for the underlying storage.
func (m *Memory) IndexAll(ctx context.Context, fetcher content.Fetcher, node ocispec.Descriptor) error {
// track content status
tracker := status.NewTracker()
var fn syncutil.GoFunc[ocispec.Descriptor]
fn = func(ctx context.Context, region *syncutil.LimitedRegion, desc ocispec.Descriptor) error {
// skip the node if other go routine is working on it
_, committed := tracker.TryCommit(desc)
if !committed {
return nil
}
// skip the node if it has been indexed
key := descriptor.FromOCI(desc)
_, exists := m.indexed.Load(key)
if exists {
return nil
}
successors, err := content.Successors(ctx, fetcher, desc)
if err != nil {
if errors.Is(err, errdef.ErrNotFound) {
// skip the node if it does not exist
return nil
}
return err
}
m.index(ctx, desc, successors)
m.indexed.Store(key, nil)
if len(successors) > 0 {
// traverse and index successors
return syncutil.Go(ctx, nil, fn, successors...)
}
return nil
}
return syncutil.Go(ctx, nil, fn, node)
}
// Predecessors returns the nodes directly pointing to the current node.
// Predecessors returns nil without error if the node does not exists in the
// store.
// Like other operations, calling Predecessors() is go-routine safe. However,
// it does not necessarily correspond to any consistent snapshot of the stored
// contents.
func (m *Memory) Predecessors(_ context.Context, node ocispec.Descriptor) ([]ocispec.Descriptor, error) {
key := descriptor.FromOCI(node)
value, exists := m.predecessors.Load(key)
if !exists {
return nil, nil
}
predecessors := value.(*sync.Map)
var res []ocispec.Descriptor
predecessors.Range(func(key, value interface{}) bool {
res = append(res, value.(ocispec.Descriptor))
return true
})
return res, nil
}
// index indexes predecessors for each direct successor of the given node.
// There is no data consistency issue as long as deletion is not implemented
// for the underlying storage.
func (m *Memory) index(ctx context.Context, node ocispec.Descriptor, successors []ocispec.Descriptor) {
if len(successors) == 0 {
return
}
predecessorKey := descriptor.FromOCI(node)
for _, successor := range successors {
successorKey := descriptor.FromOCI(successor)
value, _ := m.predecessors.LoadOrStore(successorKey, &sync.Map{})
predecessors := value.(*sync.Map)
predecessors.Store(predecessorKey, node)
}
}

View File

@@ -0,0 +1,116 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package httputil
import (
"errors"
"fmt"
"io"
"net/http"
)
// Client is an interface for a HTTP client.
// This interface is defined inside this package to prevent potential import
// loop.
type Client interface {
// Do sends an HTTP request and returns an HTTP response.
Do(*http.Request) (*http.Response, error)
}
// readSeekCloser seeks http body by starting new connections.
type readSeekCloser struct {
client Client
req *http.Request
rc io.ReadCloser
size int64
offset int64
closed bool
}
// NewReadSeekCloser returns a seeker to make the HTTP response seekable.
// Callers should ensure that the server supports Range request.
func NewReadSeekCloser(client Client, req *http.Request, respBody io.ReadCloser, size int64) io.ReadSeekCloser {
return &readSeekCloser{
client: client,
req: req,
rc: respBody,
size: size,
}
}
// Read reads the content body and counts offset.
func (rsc *readSeekCloser) Read(p []byte) (n int, err error) {
if rsc.closed {
return 0, errors.New("read: already closed")
}
n, err = rsc.rc.Read(p)
rsc.offset += int64(n)
return
}
// Seek starts a new connection to the remote for reading if position changes.
func (rsc *readSeekCloser) Seek(offset int64, whence int) (int64, error) {
if rsc.closed {
return 0, errors.New("seek: already closed")
}
switch whence {
case io.SeekCurrent:
offset += rsc.offset
case io.SeekStart:
// no-op
case io.SeekEnd:
offset += rsc.size
default:
return 0, errors.New("seek: invalid whence")
}
if offset < 0 {
return 0, errors.New("seek: an attempt was made to move the pointer before the beginning of the content")
}
if offset == rsc.offset {
return offset, nil
}
if offset >= rsc.size {
rsc.rc.Close()
rsc.rc = http.NoBody
rsc.offset = offset
return offset, nil
}
req := rsc.req.Clone(rsc.req.Context())
req.Header.Set("Range", fmt.Sprintf("bytes=%d-%d", offset, rsc.size-1))
resp, err := rsc.client.Do(req)
if err != nil {
return 0, fmt.Errorf("seek: %s %q: %w", req.Method, req.URL, err)
}
if resp.StatusCode != http.StatusPartialContent {
resp.Body.Close()
return 0, fmt.Errorf("seek: %s %q: unexpected status code %d", resp.Request.Method, resp.Request.URL, resp.StatusCode)
}
rsc.rc.Close()
rsc.rc = resp.Body
rsc.offset = offset
return offset, nil
}
// Close closes the content body.
func (rsc *readSeekCloser) Close() error {
if rsc.closed {
return nil
}
rsc.closed = true
return rsc.rc.Close()
}

View File

@@ -0,0 +1,24 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package interfaces
import "oras.land/oras-go/v2/registry"
// ReferenceParser provides reference parsing.
type ReferenceParser interface {
// ParseReference parses a reference to a fully qualified reference.
ParseReference(reference string) (registry.Reference, error)
}

View File

@@ -0,0 +1,58 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package ioutil
import (
"fmt"
"io"
"reflect"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2/content"
)
// CloserFunc is the basic Close method defined in io.Closer.
type CloserFunc func() error
// Close performs close operation by the CloserFunc.
func (fn CloserFunc) Close() error {
return fn()
}
// CopyBuffer copies from src to dst through the provided buffer
// until either EOF is reached on src, or an error occurs.
// The copied content is verified against the size and the digest.
func CopyBuffer(dst io.Writer, src io.Reader, buf []byte, desc ocispec.Descriptor) error {
// verify while copying
vr := content.NewVerifyReader(src, desc)
if _, err := io.CopyBuffer(dst, vr, buf); err != nil {
return fmt.Errorf("copy failed: %w", err)
}
return vr.Verify()
}
// nopCloserType is the type of `io.NopCloser()`.
var nopCloserType = reflect.TypeOf(io.NopCloser(nil))
// UnwrapNopCloser unwraps the reader wrapped by `io.NopCloser()`.
// Similar implementation can be found in the built-in package `net/http`.
// Reference: https://github.com/golang/go/blob/go1.17.6/src/net/http/transfer.go#L423-L425
func UnwrapNopCloser(rc io.Reader) io.Reader {
if reflect.TypeOf(rc) == nopCloserType {
return reflect.ValueOf(rc).Field(0).Interface().(io.Reader)
}
return rc
}

View File

@@ -0,0 +1,136 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package platform
import (
"context"
"encoding/json"
"fmt"
"io"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2/content"
"oras.land/oras-go/v2/errdef"
"oras.land/oras-go/v2/internal/docker"
)
// Match checks whether the current platform matches the target platform.
// Match will return true if all of the following conditions are met.
// - Architecture and OS exactly match.
// - Variant and OSVersion exactly match if target platform provided.
// - OSFeatures of the target platform are the subsets of the OSFeatures
// array of the current platform.
//
// Note: Variant, OSVersion and OSFeatures are optional fields, will skip
// the comparison if the target platform does not provide specfic value.
func Match(got *ocispec.Platform, want *ocispec.Platform) bool {
if got.Architecture != want.Architecture || got.OS != want.OS {
return false
}
if want.OSVersion != "" && got.OSVersion != want.OSVersion {
return false
}
if want.Variant != "" && got.Variant != want.Variant {
return false
}
if len(want.OSFeatures) != 0 && !isSubset(want.OSFeatures, got.OSFeatures) {
return false
}
return true
}
// isSubset returns true if all items in slice A are present in slice B.
func isSubset(a, b []string) bool {
set := make(map[string]bool, len(b))
for _, v := range b {
set[v] = true
}
for _, v := range a {
if _, ok := set[v]; !ok {
return false
}
}
return true
}
// SelectManifest implements platform filter and returns the descriptor of the
// first matched manifest if the root is a manifest list. If the root is a
// manifest, then return the root descriptor if platform matches.
func SelectManifest(ctx context.Context, src content.ReadOnlyStorage, root ocispec.Descriptor, p *ocispec.Platform) (ocispec.Descriptor, error) {
switch root.MediaType {
case docker.MediaTypeManifestList, ocispec.MediaTypeImageIndex:
manifests, err := content.Successors(ctx, src, root)
if err != nil {
return ocispec.Descriptor{}, err
}
// platform filter
for _, m := range manifests {
if Match(m.Platform, p) {
return m, nil
}
}
return ocispec.Descriptor{}, fmt.Errorf("%s: %w: no matching manifest was found in the manifest list", root.Digest, errdef.ErrNotFound)
case docker.MediaTypeManifest, ocispec.MediaTypeImageManifest:
descs, err := content.Successors(ctx, src, root)
if err != nil {
return ocispec.Descriptor{}, err
}
configMediaType := docker.MediaTypeConfig
if root.MediaType == ocispec.MediaTypeImageManifest {
configMediaType = ocispec.MediaTypeImageConfig
}
cfgPlatform, err := getPlatformFromConfig(ctx, src, descs[0], configMediaType)
if err != nil {
return ocispec.Descriptor{}, err
}
if Match(cfgPlatform, p) {
return root, nil
}
return ocispec.Descriptor{}, fmt.Errorf("%s: %w: platform in manifest does not match target platform", root.Digest, errdef.ErrNotFound)
default:
return ocispec.Descriptor{}, fmt.Errorf("%s: %s: %w", root.Digest, root.MediaType, errdef.ErrUnsupported)
}
}
// getPlatformFromConfig returns a platform object which is made up from the
// fields in config blob.
func getPlatformFromConfig(ctx context.Context, src content.ReadOnlyStorage, desc ocispec.Descriptor, targetConfigMediaType string) (*ocispec.Platform, error) {
if desc.MediaType != targetConfigMediaType {
return nil, fmt.Errorf("fail to recognize platform from unknown config %s: expect %s", desc.MediaType, targetConfigMediaType)
}
rc, err := src.Fetch(ctx, desc)
if err != nil {
return nil, err
}
defer rc.Close()
var platform ocispec.Platform
if err = json.NewDecoder(rc).Decode(&platform); err != nil && err != io.EOF {
return nil, err
}
return &platform, nil
}

View File

@@ -0,0 +1,29 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package registryutil
import (
"context"
"oras.land/oras-go/v2/registry"
"oras.land/oras-go/v2/registry/remote/auth"
)
// WithScopeHint adds a hinted scope to the context.
func WithScopeHint(ctx context.Context, ref registry.Reference, actions ...string) context.Context {
scope := auth.ScopeRepository(ref.Repository, actions...)
return auth.AppendScopes(ctx, scope)
}

View File

@@ -0,0 +1,102 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package registryutil
import (
"context"
"io"
"sync"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2/content"
"oras.land/oras-go/v2/internal/cas"
"oras.land/oras-go/v2/internal/ioutil"
"oras.land/oras-go/v2/registry"
)
// ReferenceStorage represents a CAS that supports registry.ReferenceFetcher.
type ReferenceStorage interface {
content.ReadOnlyStorage
registry.ReferenceFetcher
}
// Proxy is a caching proxy dedicated for registry.ReferenceFetcher.
// The first fetch call of a described content will read from the remote and
// cache the fetched content.
// The subsequent fetch call will read from the local cache.
type Proxy struct {
registry.ReferenceFetcher
*cas.Proxy
}
// NewProxy creates a proxy for the `base` ReferenceStorage, using the `cache`
// storage as the cache.
func NewProxy(base ReferenceStorage, cache content.Storage) *Proxy {
return &Proxy{
ReferenceFetcher: base,
Proxy: cas.NewProxy(base, cache),
}
}
// FetchReference fetches the content identified by the reference from the
// remote and cache the fetched content.
func (p *Proxy) FetchReference(ctx context.Context, reference string) (ocispec.Descriptor, io.ReadCloser, error) {
target, rc, err := p.ReferenceFetcher.FetchReference(ctx, reference)
if err != nil {
return ocispec.Descriptor{}, nil, err
}
// skip caching if the content already exists in cache
exists, err := p.Cache.Exists(ctx, target)
if err != nil {
return ocispec.Descriptor{}, nil, err
}
if exists {
return target, rc, nil
}
// cache content while reading
pr, pw := io.Pipe()
var wg sync.WaitGroup
wg.Add(1)
var pushErr error
go func() {
defer wg.Done()
pushErr = p.Cache.Push(ctx, target, pr)
if pushErr != nil {
pr.CloseWithError(pushErr)
}
}()
closer := ioutil.CloserFunc(func() error {
rcErr := rc.Close()
if err := pw.Close(); err != nil {
return err
}
wg.Wait()
if pushErr != nil {
return pushErr
}
return rcErr
})
return target, struct {
io.Reader
io.Closer
}{
Reader: io.TeeReader(rc, pw),
Closer: closer,
}, nil
}

View File

@@ -0,0 +1,61 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package resolver
import (
"context"
"sync"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2/errdef"
)
// Memory is a memory based resolver.
type Memory struct {
index sync.Map // map[string]ocispec.Descriptor
}
// NewMemory creates a new Memory resolver.
func NewMemory() *Memory {
return &Memory{}
}
// Resolve resolves a reference to a descriptor.
func (m *Memory) Resolve(_ context.Context, reference string) (ocispec.Descriptor, error) {
desc, ok := m.index.Load(reference)
if !ok {
return ocispec.Descriptor{}, errdef.ErrNotFound
}
return desc.(ocispec.Descriptor), nil
}
// Tag tags a descriptor with a reference string.
func (m *Memory) Tag(_ context.Context, desc ocispec.Descriptor, reference string) error {
m.index.Store(reference, desc)
return nil
}
// Map dumps the memory into a built-in map structure.
// Like other operations, calling Map() is go-routine safe. However, it does not
// necessarily correspond to any consistent snapshot of the storage contents.
func (m *Memory) Map() map[string]ocispec.Descriptor {
res := make(map[string]ocispec.Descriptor)
m.index.Range(func(key, value interface{}) bool {
res[key.(string)] = value.(ocispec.Descriptor)
return true
})
return res
}

View File

@@ -0,0 +1,24 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package slices
// Clone returns a shallow copy of the slice.
func Clone[S ~[]E, E any](s S) S {
if s == nil {
return nil
}
return append(make(S, 0, len(s)), s...)
}

View File

@@ -0,0 +1,43 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package status
import (
"sync"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2/internal/descriptor"
)
// Tracker tracks content status described by a descriptor.
type Tracker struct {
status sync.Map // map[descriptor.Descriptor]chan struct{}
}
// NewTracker creates a new content status tracker.
func NewTracker() *Tracker {
return &Tracker{}
}
// TryCommit tries to commit the work for the target descriptor.
// Returns true if committed. A channel is also returned for sending
// notifications. Once the work is done, the channel should be closed.
// Returns false if the work is done or still in progress.
func (t *Tracker) TryCommit(target ocispec.Descriptor) (chan struct{}, bool) {
key := descriptor.FromOCI(target)
status, exists := t.status.LoadOrStore(key, make(chan struct{}))
return status.(chan struct{}), !exists
}

View File

@@ -0,0 +1,84 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package syncutil
import (
"context"
"golang.org/x/sync/errgroup"
"golang.org/x/sync/semaphore"
)
// LimitedRegion provides a way to bound concurrent access to a code block.
type LimitedRegion struct {
ctx context.Context
limiter *semaphore.Weighted
ended bool
}
// LimitRegion creates a new LimitedRegion.
func LimitRegion(ctx context.Context, limiter *semaphore.Weighted) *LimitedRegion {
if limiter == nil {
return nil
}
return &LimitedRegion{
ctx: ctx,
limiter: limiter,
ended: true,
}
}
// Start starts the region with concurrency limit.
func (lr *LimitedRegion) Start() error {
if lr == nil || !lr.ended {
return nil
}
if err := lr.limiter.Acquire(lr.ctx, 1); err != nil {
return err
}
lr.ended = false
return nil
}
// End ends the region with concurrency limit.
func (lr *LimitedRegion) End() {
if lr == nil || lr.ended {
return
}
lr.limiter.Release(1)
lr.ended = true
}
// GoFunc represents a function that can be invoked by Go.
type GoFunc[T any] func(ctx context.Context, region *LimitedRegion, t T) error
// Go concurrently invokes fn on items.
func Go[T any](ctx context.Context, limiter *semaphore.Weighted, fn GoFunc[T], items ...T) error {
eg, egCtx := errgroup.WithContext(ctx)
for _, item := range items {
region := LimitRegion(ctx, limiter)
if err := region.Start(); err != nil {
return err
}
eg.Go(func(t T) func() error {
return func() error {
defer region.End()
return fn(egCtx, region, t)
}
}(item))
}
return eg.Wait()
}

View File

@@ -0,0 +1,67 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package syncutil
import (
"context"
"golang.org/x/sync/errgroup"
)
// A LimitedGroup is a collection of goroutines working on subtasks that are part of
// the same overall task.
type LimitedGroup struct {
grp *errgroup.Group
ctx context.Context
}
// LimitGroup returns a new LimitedGroup and an associated Context derived from ctx.
//
// The number of active goroutines in this group is limited to the given limit.
// A negative value indicates no limit.
//
// The derived Context is canceled the first time a function passed to Go
// returns a non-nil error or the first time Wait returns, whichever occurs
// first.
func LimitGroup(ctx context.Context, limit int) (*LimitedGroup, context.Context) {
grp, ctx := errgroup.WithContext(ctx)
grp.SetLimit(limit)
return &LimitedGroup{grp: grp, ctx: ctx}, ctx
}
// Go calls the given function in a new goroutine.
// It blocks until the new goroutine can be added without the number of
// active goroutines in the group exceeding the configured limit.
//
// The first call to return a non-nil error cancels the group's context.
// After which, any subsequent calls to Go will not execute their given function.
// The error will be returned by Wait.
func (g *LimitedGroup) Go(f func() error) {
g.grp.Go(func() error {
select {
case <-g.ctx.Done():
return g.ctx.Err()
default:
return f()
}
})
}
// Wait blocks until all function calls from the Go method have returned, then
// returns the first non-nil error (if any) from them.
func (g *LimitedGroup) Wait() error {
return g.grp.Wait()
}

View File

@@ -0,0 +1,140 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package syncutil
import "sync"
// mergeStatus represents the merge status of an item.
type mergeStatus struct {
// main indicates if items are being merged by the current go-routine.
main bool
// err represents the error of the merge operation.
err error
}
// Merge represents merge operations on items.
// The state transfer is shown as below:
//
// +----------+
// | Start +--------+-------------+
// +----+-----+ | |
// | | |
// v v v
// +----+-----+ +----+----+ +----+----+
// +-------+ Prepare +<--+ Pending +-->+ Waiting |
// | +----+-----+ +---------+ +----+----+
// | | |
// | v |
// | + ---+---- + |
// On Error | Resolve | |
// | + ---+---- + |
// | | |
// | v |
// | +----+-----+ |
// +------>+ Complete +<---------------------+
// +----+-----+
// |
// v
// +----+-----+
// | End |
// +----------+
type Merge[T any] struct {
lock sync.Mutex
committed bool
items []T
status chan mergeStatus
pending []T
pendingStatus chan mergeStatus
}
// Do merges concurrent operations of items into a single call of prepare and
// resolve.
// If Do is called multiple times concurrently, only one of the calls will be
// selected to invoke prepare and resolve.
func (m *Merge[T]) Do(item T, prepare func() error, resolve func(items []T) error) error {
status := <-m.assign(item)
if status.main {
err := prepare()
items := m.commit()
if err == nil {
err = resolve(items)
}
m.complete(err)
return err
}
return status.err
}
// assign adds a new item into the item list.
func (m *Merge[T]) assign(item T) <-chan mergeStatus {
m.lock.Lock()
defer m.lock.Unlock()
if m.committed {
if m.pendingStatus == nil {
m.pendingStatus = make(chan mergeStatus, 1)
}
m.pending = append(m.pending, item)
return m.pendingStatus
}
if m.status == nil {
m.status = make(chan mergeStatus, 1)
m.status <- mergeStatus{main: true}
}
m.items = append(m.items, item)
return m.status
}
// commit closes the assignment window, and the assigned items will be ready
// for resolve.
func (m *Merge[T]) commit() []T {
m.lock.Lock()
defer m.lock.Unlock()
m.committed = true
return m.items
}
// complete completes the previous merge, and moves the pending items to the
// stage for the next merge.
func (m *Merge[T]) complete(err error) {
// notify results
if err == nil {
close(m.status)
} else {
remaining := len(m.items) - 1
status := m.status
for remaining > 0 {
status <- mergeStatus{err: err}
remaining--
}
}
// move pending items to the stage
m.lock.Lock()
defer m.lock.Unlock()
m.committed = false
m.items = m.pending
m.status = m.pendingStatus
m.pending = nil
m.pendingStatus = nil
if m.status != nil {
m.status <- mergeStatus{main: true}
}
}

View File

@@ -0,0 +1,70 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package syncutil
import "context"
// Once is an object that will perform exactly one action.
// Unlike sync.Once, this Once allowes the action to have return values.
type Once struct {
result interface{}
err error
status chan bool
}
// NewOnce creates a new Once instance.
func NewOnce() *Once {
status := make(chan bool, 1)
status <- true
return &Once{
status: status,
}
}
// Do calls the function f if and only if Do is being called first time or all
// previous function calls are cancelled, deadline exceeded, or panicking.
// When `once.Do(ctx, f)` is called multiple times, the return value of the
// first call of the function f is stored, and is directly returned for other
// calls.
// Besides the return value of the function f, including the error, Do returns
// true if the function f passed is called first and is not cancelled, deadline
// exceeded, or panicking. Otherwise, returns false.
func (o *Once) Do(ctx context.Context, f func() (interface{}, error)) (bool, interface{}, error) {
defer func() {
if r := recover(); r != nil {
o.status <- true
panic(r)
}
}()
for {
select {
case inProgress := <-o.status:
if !inProgress {
return false, o.result, o.err
}
result, err := f()
if err == context.Canceled || err == context.DeadlineExceeded {
o.status <- true
return false, nil, err
}
o.result, o.err = result, err
close(o.status)
return true, result, err
case <-ctx.Done():
return false, nil, ctx.Err()
}
}
}

View File

@@ -0,0 +1,64 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package syncutil
import "sync"
// poolItem represents an item in Pool.
type poolItem[T any] struct {
value T
refCount int
}
// Pool is a scalable pool with items identified by keys.
type Pool[T any] struct {
// New optionally specifies a function to generate a value when Get would
// otherwise return nil.
// It may not be changed concurrently with calls to Get.
New func() T
lock sync.Mutex
items map[any]*poolItem[T]
}
// Get gets the value identified by key.
// The caller should invoke the returned function after using the returned item.
func (p *Pool[T]) Get(key any) (*T, func()) {
p.lock.Lock()
defer p.lock.Unlock()
item, ok := p.items[key]
if !ok {
if p.items == nil {
p.items = make(map[any]*poolItem[T])
}
item = &poolItem[T]{}
if p.New != nil {
item.value = p.New()
}
p.items[key] = item
}
item.refCount++
return &item.value, func() {
p.lock.Lock()
defer p.lock.Unlock()
item.refCount--
if item.refCount <= 0 {
delete(p.items, key)
}
}
}

202
vendor/oras.land/oras-go/v2/pack.go vendored Normal file
View File

@@ -0,0 +1,202 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package oras
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"time"
specs "github.com/opencontainers/image-spec/specs-go"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2/content"
"oras.land/oras-go/v2/errdef"
)
const (
// MediaTypeUnknownConfig is the default mediaType used when no
// config media type is specified.
MediaTypeUnknownConfig = "application/vnd.unknown.config.v1+json"
// MediaTypeUnknownArtifact is the default artifactType used when no
// artifact type is specified.
MediaTypeUnknownArtifact = "application/vnd.unknown.artifact.v1"
)
// ErrInvalidDateTimeFormat is returned by Pack() when
// AnnotationArtifactCreated or AnnotationCreated is provided, but its value
// is not in RFC 3339 format.
// Reference: https://www.rfc-editor.org/rfc/rfc3339#section-5.6
var ErrInvalidDateTimeFormat = errors.New("invalid date and time format")
// PackOptions contains parameters for [oras.Pack].
type PackOptions struct {
// Subject is the subject of the manifest.
Subject *ocispec.Descriptor
// ManifestAnnotations is the annotation map of the manifest.
ManifestAnnotations map[string]string
// PackImageManifest controls whether to pack an image manifest or not.
// - If true, pack an image manifest; artifactType will be used as the
// the config descriptor mediaType of the image manifest.
// - If false, pack an artifact manifest.
// Default: false.
PackImageManifest bool
// ConfigDescriptor is a pointer to the descriptor of the config blob.
// If not nil, artifactType will be implied by the mediaType of the
// specified ConfigDescriptor, and ConfigAnnotations will be ignored.
// This option is valid only when PackImageManifest is true.
ConfigDescriptor *ocispec.Descriptor
// ConfigAnnotations is the annotation map of the config descriptor.
// This option is valid only when PackImageManifest is true
// and ConfigDescriptor is nil.
ConfigAnnotations map[string]string
}
// Pack packs the given blobs, generates a manifest for the pack,
// and pushes it to a content storage.
//
// When opts.PackImageManifest is true, artifactType will be used as the
// the config descriptor mediaType of the image manifest.
// If succeeded, returns a descriptor of the manifest.
func Pack(ctx context.Context, pusher content.Pusher, artifactType string, blobs []ocispec.Descriptor, opts PackOptions) (ocispec.Descriptor, error) {
if opts.PackImageManifest {
return packImage(ctx, pusher, artifactType, blobs, opts)
}
return packArtifact(ctx, pusher, artifactType, blobs, opts)
}
// packArtifact packs the given blobs, generates an artifact manifest for the
// pack, and pushes it to a content storage.
// If succeeded, returns a descriptor of the manifest.
func packArtifact(ctx context.Context, pusher content.Pusher, artifactType string, blobs []ocispec.Descriptor, opts PackOptions) (ocispec.Descriptor, error) {
if artifactType == "" {
artifactType = MediaTypeUnknownArtifact
}
annotations, err := ensureAnnotationCreated(opts.ManifestAnnotations, ocispec.AnnotationArtifactCreated)
if err != nil {
return ocispec.Descriptor{}, err
}
manifest := ocispec.Artifact{
MediaType: ocispec.MediaTypeArtifactManifest,
ArtifactType: artifactType,
Blobs: blobs,
Subject: opts.Subject,
Annotations: annotations,
}
manifestJSON, err := json.Marshal(manifest)
if err != nil {
return ocispec.Descriptor{}, fmt.Errorf("failed to marshal manifest: %w", err)
}
manifestDesc := content.NewDescriptorFromBytes(ocispec.MediaTypeArtifactManifest, manifestJSON)
// populate ArtifactType and Annotations of the manifest into manifestDesc
manifestDesc.ArtifactType = manifest.ArtifactType
manifestDesc.Annotations = manifest.Annotations
// push manifest
if err := pusher.Push(ctx, manifestDesc, bytes.NewReader(manifestJSON)); err != nil && !errors.Is(err, errdef.ErrAlreadyExists) {
return ocispec.Descriptor{}, fmt.Errorf("failed to push manifest: %w", err)
}
return manifestDesc, nil
}
// packImage packs the given blobs, generates an image manifest for the pack,
// and pushes it to a content storage. artifactType will be used as the config
// descriptor mediaType of the image manifest.
// If succeeded, returns a descriptor of the manifest.
func packImage(ctx context.Context, pusher content.Pusher, configMediaType string, layers []ocispec.Descriptor, opts PackOptions) (ocispec.Descriptor, error) {
if configMediaType == "" {
configMediaType = MediaTypeUnknownConfig
}
var configDesc ocispec.Descriptor
if opts.ConfigDescriptor != nil {
configDesc = *opts.ConfigDescriptor
} else {
// Use an empty JSON object here, because some registries may not accept
// empty config blob.
// As of September 2022, GAR is known to return 400 on empty blob upload.
// See https://github.com/oras-project/oras-go/issues/294 for details.
configBytes := []byte("{}")
configDesc = content.NewDescriptorFromBytes(configMediaType, configBytes)
configDesc.Annotations = opts.ConfigAnnotations
// push config
if err := pusher.Push(ctx, configDesc, bytes.NewReader(configBytes)); err != nil && !errors.Is(err, errdef.ErrAlreadyExists) {
return ocispec.Descriptor{}, fmt.Errorf("failed to push config: %w", err)
}
}
annotations, err := ensureAnnotationCreated(opts.ManifestAnnotations, ocispec.AnnotationCreated)
if err != nil {
return ocispec.Descriptor{}, err
}
if layers == nil {
layers = []ocispec.Descriptor{} // make it an empty array to prevent potential server-side bugs
}
manifest := ocispec.Manifest{
Versioned: specs.Versioned{
SchemaVersion: 2, // historical value. does not pertain to OCI or docker version
},
Config: configDesc,
MediaType: ocispec.MediaTypeImageManifest,
Layers: layers,
Subject: opts.Subject,
Annotations: annotations,
}
manifestJSON, err := json.Marshal(manifest)
if err != nil {
return ocispec.Descriptor{}, fmt.Errorf("failed to marshal manifest: %w", err)
}
manifestDesc := content.NewDescriptorFromBytes(ocispec.MediaTypeImageManifest, manifestJSON)
// populate ArtifactType and Annotations of the manifest into manifestDesc
manifestDesc.ArtifactType = manifest.Config.MediaType
manifestDesc.Annotations = manifest.Annotations
// push manifest
if err := pusher.Push(ctx, manifestDesc, bytes.NewReader(manifestJSON)); err != nil && !errors.Is(err, errdef.ErrAlreadyExists) {
return ocispec.Descriptor{}, fmt.Errorf("failed to push manifest: %w", err)
}
return manifestDesc, nil
}
// ensureAnnotationCreated ensures that annotationCreatedKey is in annotations,
// and that its value conforms to RFC 3339. Otherwise returns a new annotation
// map with annotationCreatedKey created.
func ensureAnnotationCreated(annotations map[string]string, annotationCreatedKey string) (map[string]string, error) {
if createdTime, ok := annotations[annotationCreatedKey]; ok {
// if annotationCreatedKey is provided, validate its format
if _, err := time.Parse(time.RFC3339, createdTime); err != nil {
return nil, fmt.Errorf("%w: %v", ErrInvalidDateTimeFormat, err)
}
return annotations, nil
}
// copy the original annotation map
copied := make(map[string]string, len(annotations)+1)
for k, v := range annotations {
copied[k] = v
}
// set creation time in RFC 3339 format
// reference: https://github.com/opencontainers/image-spec/blob/v1.1.0-rc2/annotations.md#pre-defined-annotation-keys
now := time.Now().UTC()
copied[annotationCreatedKey] = now.Format(time.RFC3339)
return copied, nil
}

View File

@@ -0,0 +1,269 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package registry
import (
"fmt"
"net/url"
"regexp"
"strings"
"github.com/opencontainers/go-digest"
"oras.land/oras-go/v2/errdef"
)
// regular expressions for components.
var (
// repositoryRegexp is adapted from the distribution implementation. The
// repository name set under OCI distribution spec is a subset of the docker
// spec. For maximum compatability, the docker spec is verified client-side.
// Further checks are left to the server-side.
// References:
// - https://github.com/distribution/distribution/blob/v2.7.1/reference/regexp.go#L53
// - https://github.com/opencontainers/distribution-spec/blob/v1.1.0-rc1/spec.md#pulling-manifests
repositoryRegexp = regexp.MustCompile(`^[a-z0-9]+(?:(?:[._]|__|[-]*)[a-z0-9]+)*(?:/[a-z0-9]+(?:(?:[._]|__|[-]*)[a-z0-9]+)*)*$`)
// tagRegexp checks the tag name.
// The docker and OCI spec have the same regular expression.
// Reference: https://github.com/opencontainers/distribution-spec/blob/v1.1.0-rc1/spec.md#pulling-manifests
tagRegexp = regexp.MustCompile(`^[\w][\w.-]{0,127}$`)
)
// Reference references either a resource descriptor (where Reference.Reference
// is a tag or a digest), or a resource repository (where Reference.Reference
// is the empty string).
type Reference struct {
// Registry is the name of the registry. It is usually the domain name of
// the registry optionally with a port.
Registry string
// Repository is the name of the repository.
Repository string
// Reference is the reference of the object in the repository. This field
// can take any one of the four valid forms (see ParseReference). In the
// case where it's the empty string, it necessarily implies valid form D,
// and where it is non-empty, then it is either a tag, or a digest
// (implying one of valid forms A, B, or C).
Reference string
}
// ParseReference parses a string (artifact) into an `artifact reference`.
//
// Note: An "image" is an "artifact", however, an "artifact" is not necessarily
// an "image".
//
// The token `artifact` is composed of other tokens, and those in turn are
// composed of others. This definition recursivity requires a notation capable
// of recursion, thus the following two forms have been adopted:
//
// 1. BackusNaur Form (BNF) has been adopted to address the recursive nature
// of the definition.
// 2. Token opacity is revealed via its label letter-casing. That is, "opaque"
// tokens (i.e., tokens that are not final, and must therefore be further
// broken down into their constituents) are denoted in *lowercase*, while
// final tokens (i.e., leaf-node tokens that are final) are denoted in
// *uppercase*.
//
// Finally, note that a number of the opaque tokens are polymorphic in nature;
// that is, they can take on one of numerous forms, not restricted to a single
// defining form.
//
// The top-level token, `artifact`, is composed of two (opaque) tokens; namely
// `socketaddr` and `path`:
//
// <artifact> ::= <socketaddr> "/" <path>
//
// The former is described as follows:
//
// <socketaddr> ::= <host> | <host> ":" <PORT>
// <host> ::= <ip> | <FQDN>
// <ip> ::= <IPV4-ADDR> | <IPV6-ADDR>
//
// The latter, which is of greater interest here, is described as follows:
//
// <path> ::= <REPOSITORY> | <REPOSITORY> <reference>
// <reference> ::= "@" <digest> | ":" <TAG> "@" <DIGEST> | ":" <TAG>
// <digest> ::= <ALGO> ":" <HASH>
//
// This second token--`path`--can take on exactly four forms, each of which will
// now be illustrated:
//
// <--- path --------------------------------------------> | - Decode `path`
// <=== REPOSITORY ===> <--- reference ------------------> | - Decode `reference`
// <=== REPOSITORY ===> @ <=================== digest ===> | - Valid Form A
// <=== REPOSITORY ===> : <!!! TAG !!!> @ <=== digest ===> | - Valid Form B (tag is dropped)
// <=== REPOSITORY ===> : <=== TAG ======================> | - Valid Form C
// <=== REPOSITORY ======================================> | - Valid Form D
//
// Note: In the case of Valid Form B, TAG is dropped without any validation or
// further consideration.
func ParseReference(artifact string) (Reference, error) {
parts := strings.SplitN(artifact, "/", 2)
if len(parts) == 1 {
// Invalid Form
return Reference{}, fmt.Errorf("%w: missing repository", errdef.ErrInvalidReference)
}
registry, path := parts[0], parts[1]
var isTag bool
var repository string
var reference string
if index := strings.Index(path, "@"); index != -1 {
// `digest` found; Valid Form A (if not B)
isTag = false
repository = path[:index]
reference = path[index+1:]
if index = strings.Index(repository, ":"); index != -1 {
// `tag` found (and now dropped without validation) since `the
// `digest` already present; Valid Form B
repository = repository[:index]
}
} else if index = strings.Index(path, ":"); index != -1 {
// `tag` found; Valid Form C
isTag = true
repository = path[:index]
reference = path[index+1:]
} else {
// empty `reference`; Valid Form D
repository = path
}
ref := Reference{
Registry: registry,
Repository: repository,
Reference: reference,
}
if err := ref.ValidateRegistry(); err != nil {
return Reference{}, err
}
if err := ref.ValidateRepository(); err != nil {
return Reference{}, err
}
if len(ref.Reference) == 0 {
return ref, nil
}
validator := ref.ValidateReferenceAsDigest
if isTag {
validator = ref.ValidateReferenceAsTag
}
if err := validator(); err != nil {
return Reference{}, err
}
return ref, nil
}
// Validate the entire reference object; the registry, the repository, and the
// reference.
func (r Reference) Validate() error {
if err := r.ValidateRegistry(); err != nil {
return err
}
if err := r.ValidateRepository(); err != nil {
return err
}
return r.ValidateReference()
}
// ValidateRegistry validates the registry.
func (r Reference) ValidateRegistry() error {
if uri, err := url.ParseRequestURI("dummy://" + r.Registry); err != nil || uri.Host != r.Registry {
return fmt.Errorf("%w: invalid registry", errdef.ErrInvalidReference)
}
return nil
}
// ValidateRepository validates the repository.
func (r Reference) ValidateRepository() error {
if !repositoryRegexp.MatchString(r.Repository) {
return fmt.Errorf("%w: invalid repository", errdef.ErrInvalidReference)
}
return nil
}
// ValidateReferenceAsTag validates the reference as a tag.
func (r Reference) ValidateReferenceAsTag() error {
if !tagRegexp.MatchString(r.Reference) {
return fmt.Errorf("%w: invalid tag", errdef.ErrInvalidReference)
}
return nil
}
// ValidateReferenceAsDigest validates the reference as a digest.
func (r Reference) ValidateReferenceAsDigest() error {
if _, err := r.Digest(); err != nil {
return fmt.Errorf("%w: invalid digest; %v", errdef.ErrInvalidReference, err)
}
return nil
}
// ValidateReference where the reference is first tried as an ampty string, then
// as a digest, and if that fails, as a tag.
func (r Reference) ValidateReference() error {
if len(r.Reference) == 0 {
return nil
}
if index := strings.IndexByte(r.Reference, ':'); index != -1 {
return r.ValidateReferenceAsDigest()
}
return r.ValidateReferenceAsTag()
}
// Host returns the host name of the registry.
func (r Reference) Host() string {
if r.Registry == "docker.io" {
return "registry-1.docker.io"
}
return r.Registry
}
// ReferenceOrDefault returns the reference or the default reference if empty.
func (r Reference) ReferenceOrDefault() string {
if r.Reference == "" {
return "latest"
}
return r.Reference
}
// Digest returns the reference as a digest.
func (r Reference) Digest() (digest.Digest, error) {
return digest.Parse(r.Reference)
}
// String implements `fmt.Stringer` and returns the reference string.
// The resulted string is meaningful only if the reference is valid.
func (r Reference) String() string {
if r.Repository == "" {
return r.Registry
}
ref := r.Registry + "/" + r.Repository
if r.Reference == "" {
return ref
}
if d, err := r.Digest(); err == nil {
return ref + "@" + d.String()
}
return ref + ":" + r.Reference
}

View File

@@ -0,0 +1,52 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package registry provides high-level operations to manage registries.
package registry
import "context"
// Registry represents a collection of repositories.
type Registry interface {
// Repositories lists the name of repositories available in the registry.
// Since the returned repositories may be paginated by the underlying
// implementation, a function should be passed in to process the paginated
// repository list.
// `last` argument is the `last` parameter when invoking the catalog API.
// If `last` is NOT empty, the entries in the response start after the
// repo specified by `last`. Otherwise, the response starts from the top
// of the Repositories list.
// Note: When implemented by a remote registry, the catalog API is called.
// However, not all registries supports pagination or conforms the
// specification.
// Reference: https://docs.docker.com/registry/spec/api/#catalog
// See also `Repositories()` in this package.
Repositories(ctx context.Context, last string, fn func(repos []string) error) error
// Repository returns a repository reference by the given name.
Repository(ctx context.Context, name string) (Repository, error)
}
// Repositories lists the name of repositories available in the registry.
func Repositories(ctx context.Context, reg Registry) ([]string, error) {
var res []string
if err := reg.Repositories(ctx, "", func(repos []string) error {
res = append(res, repos...)
return nil
}); err != nil {
return nil, err
}
return res, nil
}

View File

@@ -0,0 +1,159 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package auth
import (
"context"
"strings"
"sync"
"oras.land/oras-go/v2/errdef"
"oras.land/oras-go/v2/internal/syncutil"
)
// DefaultCache is the sharable cache used by DefaultClient.
var DefaultCache Cache = NewCache()
// Cache caches the auth-scheme and auth-token for the "Authorization" header in
// accessing the remote registry.
// Precisely, the header is `Authorization: auth-scheme auth-token`.
// The `auth-token` is a generic term as `token68` in RFC 7235 section 2.1.
type Cache interface {
// GetScheme returns the auth-scheme part cached for the given registry.
// A single registry is assumed to have a consistent scheme.
// If a registry has different schemes per path, the auth client is still
// workable. However, the cache may not be effective as the cache cannot
// correctly guess the scheme.
GetScheme(ctx context.Context, registry string) (Scheme, error)
// GetToken returns the auth-token part cached for the given registry of a
// given scheme.
// The underlying implementation MAY cache the token for all schemes for the
// given registry.
GetToken(ctx context.Context, registry string, scheme Scheme, key string) (string, error)
// Set fetches the token using the given fetch function and caches the token
// for the given scheme with the given key for the given registry.
// The return values of the fetch function is returned by this function.
// The underlying implementation MAY combine the fetch operation if the Set
// function is invoked multiple times at the same time.
Set(ctx context.Context, registry string, scheme Scheme, key string, fetch func(context.Context) (string, error)) (string, error)
}
// cacheEntry is a cache entry for a single registry.
type cacheEntry struct {
scheme Scheme
tokens sync.Map // map[string]string
}
// concurrentCache is a cache suitable for concurrent invocation.
type concurrentCache struct {
status sync.Map // map[string]*syncutil.Once
cache sync.Map // map[string]*cacheEntry
}
// NewCache creates a new go-routine safe cache instance.
func NewCache() Cache {
return &concurrentCache{}
}
// GetScheme returns the auth-scheme part cached for the given registry.
func (cc *concurrentCache) GetScheme(ctx context.Context, registry string) (Scheme, error) {
entry, ok := cc.cache.Load(registry)
if !ok {
return SchemeUnknown, errdef.ErrNotFound
}
return entry.(*cacheEntry).scheme, nil
}
// GetToken returns the auth-token part cached for the given registry of a given
// scheme.
func (cc *concurrentCache) GetToken(ctx context.Context, registry string, scheme Scheme, key string) (string, error) {
entryValue, ok := cc.cache.Load(registry)
if !ok {
return "", errdef.ErrNotFound
}
entry := entryValue.(*cacheEntry)
if entry.scheme != scheme {
return "", errdef.ErrNotFound
}
if token, ok := entry.tokens.Load(key); ok {
return token.(string), nil
}
return "", errdef.ErrNotFound
}
// Set fetches the token using the given fetch function and caches the token
// for the given scheme with the given key for the given registry.
// Set combines the fetch operation if the Set is invoked multiple times at the
// same time.
func (cc *concurrentCache) Set(ctx context.Context, registry string, scheme Scheme, key string, fetch func(context.Context) (string, error)) (string, error) {
// fetch token
statusKey := strings.Join([]string{
registry,
scheme.String(),
key,
}, " ")
statusValue, _ := cc.status.LoadOrStore(statusKey, syncutil.NewOnce())
fetchOnce := statusValue.(*syncutil.Once)
fetchedFirst, result, err := fetchOnce.Do(ctx, func() (interface{}, error) {
return fetch(ctx)
})
if fetchedFirst {
cc.status.Delete(statusKey)
}
if err != nil {
return "", err
}
token := result.(string)
if !fetchedFirst {
return token, nil
}
// cache token
newEntry := &cacheEntry{
scheme: scheme,
}
entryValue, exists := cc.cache.LoadOrStore(registry, newEntry)
entry := entryValue.(*cacheEntry)
if exists && entry.scheme != scheme {
// there is a scheme change, which is not expected in most scenarios.
// force invalidating all previous cache.
entry = newEntry
cc.cache.Store(registry, entry)
}
entry.tokens.Store(key, token)
return token, nil
}
// noCache is a cache implementation that does not do cache at all.
type noCache struct{}
// GetScheme always returns not found error as it has no cache.
func (noCache) GetScheme(ctx context.Context, registry string) (Scheme, error) {
return SchemeUnknown, errdef.ErrNotFound
}
// GetToken always returns not found error as it has no cache.
func (noCache) GetToken(ctx context.Context, registry string, scheme Scheme, key string) (string, error) {
return "", errdef.ErrNotFound
}
// Set calls fetch directly without caching.
func (noCache) Set(ctx context.Context, registry string, scheme Scheme, key string, fetch func(context.Context) (string, error)) (string, error) {
return fetch(ctx)
}

View File

@@ -0,0 +1,167 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package auth
import (
"strconv"
"strings"
)
// Scheme define the authentication method.
type Scheme byte
const (
// SchemeUnknown represents unknown or unsupported schemes
SchemeUnknown Scheme = iota
// SchemeBasic represents the "Basic" HTTP authentication scheme.
// Reference: https://tools.ietf.org/html/rfc7617
SchemeBasic
// SchemeBearer represents the Bearer token in OAuth 2.0.
// Reference: https://tools.ietf.org/html/rfc6750
SchemeBearer
)
// parseScheme parse the authentication scheme from the given string
// case-insensitively.
func parseScheme(scheme string) Scheme {
switch {
case strings.EqualFold(scheme, "basic"):
return SchemeBasic
case strings.EqualFold(scheme, "bearer"):
return SchemeBearer
}
return SchemeUnknown
}
// String return the string for the scheme.
func (s Scheme) String() string {
switch s {
case SchemeBasic:
return "Basic"
case SchemeBearer:
return "Bearer"
}
return "Unknown"
}
// parseChallenge parses the "WWW-Authenticate" header returned by the remote
// registry, and extracts parameters if scheme is Bearer.
// References:
// - https://docs.docker.com/registry/spec/auth/token/#how-to-authenticate
// - https://tools.ietf.org/html/rfc7235#section-2.1
func parseChallenge(header string) (scheme Scheme, params map[string]string) {
// as defined in RFC 7235 section 2.1, we have
// challenge = auth-scheme [ 1*SP ( token68 / #auth-param ) ]
// auth-scheme = token
// auth-param = token BWS "=" BWS ( token / quoted-string )
//
// since we focus parameters only on Bearer, we have
// challenge = auth-scheme [ 1*SP #auth-param ]
schemeString, rest := parseToken(header)
scheme = parseScheme(schemeString)
// fast path for non bearer challenge
if scheme != SchemeBearer {
return
}
// parse params for bearer auth.
// combining RFC 7235 section 2.1 with RFC 7230 section 7, we have
// #auth-param => auth-param *( OWS "," OWS auth-param )
var key, value string
for {
key, rest = parseToken(skipSpace(rest))
if key == "" {
return
}
rest = skipSpace(rest)
if rest == "" || rest[0] != '=' {
return
}
rest = skipSpace(rest[1:])
if rest == "" {
return
}
if rest[0] == '"' {
prefix, err := strconv.QuotedPrefix(rest)
if err != nil {
return
}
value, err = strconv.Unquote(prefix)
if err != nil {
return
}
rest = rest[len(prefix):]
} else {
value, rest = parseToken(rest)
if value == "" {
return
}
}
if params == nil {
params = map[string]string{
key: value,
}
} else {
params[key] = value
}
rest = skipSpace(rest)
if rest == "" || rest[0] != ',' {
return
}
rest = rest[1:]
}
}
// isNotTokenChar reports whether rune is not a `tchar` defined in RFC 7230
// section 3.2.6.
func isNotTokenChar(r rune) bool {
// tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*"
// / "+" / "-" / "." / "^" / "_" / "`" / "|" / "~"
// / DIGIT / ALPHA
// ; any VCHAR, except delimiters
return (r < 'A' || r > 'Z') && (r < 'a' || r > 'z') &&
(r < '0' || r > '9') && !strings.ContainsRune("!#$%&'*+-.^_`|~", r)
}
// parseToken finds the next token from the given string. If no token found,
// an empty token is returned and the whole of the input is returned in rest.
// Note: Since token = 1*tchar, empty string is not a valid token.
func parseToken(s string) (token, rest string) {
if i := strings.IndexFunc(s, isNotTokenChar); i != -1 {
return s[:i], s[i:]
}
return s, ""
}
// skipSpace skips "bad" whitespace (BWS) defined in RFC 7230 section 3.2.3.
func skipSpace(s string) string {
// OWS = *( SP / HTAB )
// ; optional whitespace
// BWS = OWS
// ; "bad" whitespace
if i := strings.IndexFunc(s, func(r rune) bool {
return r != ' ' && r != '\t'
}); i != -1 {
return s[i:]
}
return s
}

View File

@@ -0,0 +1,413 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package auth provides authentication for a client to a remote registry.
package auth
import (
"context"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"strings"
"oras.land/oras-go/v2/registry/remote/internal/errutil"
"oras.land/oras-go/v2/registry/remote/retry"
)
// DefaultClient is the default auth-decorated client.
var DefaultClient = &Client{
Client: retry.DefaultClient,
Header: http.Header{
"User-Agent": {"oras-go"},
},
Cache: DefaultCache,
}
// maxResponseBytes specifies the default limit on how many response bytes are
// allowed in the server's response from authorization service servers.
// A typical response message from authorization service servers is around 1 to
// 4 KiB. Since the size of a token must be smaller than the HTTP header size
// limit, which is usually 16 KiB. As specified by the distribution, the
// response may contain 2 identical tokens, that is, 16 x 2 = 32 KiB.
// Hence, 128 KiB should be sufficient.
// References: https://docs.docker.com/registry/spec/auth/token/
var maxResponseBytes int64 = 128 * 1024 // 128 KiB
// defaultClientID specifies the default client ID used in OAuth2.
// See also ClientID.
var defaultClientID = "oras-go"
// StaticCredential specifies static credentials for the given host.
func StaticCredential(registry string, cred Credential) func(context.Context, string) (Credential, error) {
return func(_ context.Context, target string) (Credential, error) {
if target == registry {
return cred, nil
}
return EmptyCredential, nil
}
}
// Client is an auth-decorated HTTP client.
// Its zero value is a usable client that uses http.DefaultClient with no cache.
type Client struct {
// Client is the underlying HTTP client used to access the remote
// server.
// If nil, http.DefaultClient is used.
// It is possible to use the default retry client from the package
// `oras.land/oras-go/v2/registry/remote/retry`. That client is already available
// in the DefaultClient.
// It is also possible to use a custom client. For example, github.com/hashicorp/go-retryablehttp
// is a popular HTTP client that supports retries.
Client *http.Client
// Header contains the custom headers to be added to each request.
Header http.Header
// Credential specifies the function for resolving the credential for the
// given registry (i.e. host:port).
// `EmptyCredential` is a valid return value and should not be considered as
// an error.
// If nil, the credential is always resolved to `EmptyCredential`.
Credential func(context.Context, string) (Credential, error)
// Cache caches credentials for direct accessing the remote registry.
// If nil, no cache is used.
Cache Cache
// ClientID used in fetching OAuth2 token as a required field.
// If empty, a default client ID is used.
// Reference: https://docs.docker.com/registry/spec/auth/oauth/#getting-a-token
ClientID string
// ForceAttemptOAuth2 controls whether to follow OAuth2 with password grant
// instead the distribution spec when authenticating using username and
// password.
// References:
// - https://docs.docker.com/registry/spec/auth/jwt/
// - https://docs.docker.com/registry/spec/auth/oauth/
ForceAttemptOAuth2 bool
}
// client returns an HTTP client used to access the remote registry.
// http.DefaultClient is return if the client is not configured.
func (c *Client) client() *http.Client {
if c.Client == nil {
return http.DefaultClient
}
return c.Client
}
// send adds headers to the request and sends the request to the remote server.
func (c *Client) send(req *http.Request) (*http.Response, error) {
for key, values := range c.Header {
req.Header[key] = append(req.Header[key], values...)
}
return c.client().Do(req)
}
// credential resolves the credential for the given registry.
func (c *Client) credential(ctx context.Context, reg string) (Credential, error) {
if c.Credential == nil {
return EmptyCredential, nil
}
return c.Credential(ctx, reg)
}
// cache resolves the cache.
// noCache is return if the cache is not configured.
func (c *Client) cache() Cache {
if c.Cache == nil {
return noCache{}
}
return c.Cache
}
// SetUserAgent sets the user agent for all out-going requests.
func (c *Client) SetUserAgent(userAgent string) {
if c.Header == nil {
c.Header = http.Header{}
}
c.Header.Set("User-Agent", userAgent)
}
// Do sends the request to the remote server, attempting to resolve
// authentication if 'Authorization' header is not set.
//
// On authentication failure due to bad credential,
// - Do returns error if it fails to fetch token for bearer auth.
// - Do returns the registry response without error for basic auth.
func (c *Client) Do(originalReq *http.Request) (*http.Response, error) {
if auth := originalReq.Header.Get("Authorization"); auth != "" {
return c.send(originalReq)
}
ctx := originalReq.Context()
req := originalReq.Clone(ctx)
// attempt cached auth token
var attemptedKey string
cache := c.cache()
registry := originalReq.Host
scheme, err := cache.GetScheme(ctx, registry)
if err == nil {
switch scheme {
case SchemeBasic:
token, err := cache.GetToken(ctx, registry, SchemeBasic, "")
if err == nil {
req.Header.Set("Authorization", "Basic "+token)
}
case SchemeBearer:
scopes := GetScopes(ctx)
attemptedKey = strings.Join(scopes, " ")
token, err := cache.GetToken(ctx, registry, SchemeBearer, attemptedKey)
if err == nil {
req.Header.Set("Authorization", "Bearer "+token)
}
}
}
resp, err := c.send(req)
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusUnauthorized {
return resp, nil
}
// attempt again with credentials for recognized schemes
challenge := resp.Header.Get("Www-Authenticate")
scheme, params := parseChallenge(challenge)
switch scheme {
case SchemeBasic:
resp.Body.Close()
token, err := cache.Set(ctx, registry, SchemeBasic, "", func(ctx context.Context) (string, error) {
return c.fetchBasicAuth(ctx, registry)
})
if err != nil {
return nil, fmt.Errorf("%s %q: %w", resp.Request.Method, resp.Request.URL, err)
}
req = originalReq.Clone(ctx)
req.Header.Set("Authorization", "Basic "+token)
case SchemeBearer:
resp.Body.Close()
// merge hinted scopes with challenged scopes
scopes := GetScopes(ctx)
if scope := params["scope"]; scope != "" {
scopes = append(scopes, strings.Split(scope, " ")...)
scopes = CleanScopes(scopes)
}
key := strings.Join(scopes, " ")
// attempt the cache again if there is a scope change
if key != attemptedKey {
if token, err := cache.GetToken(ctx, registry, SchemeBearer, key); err == nil {
req = originalReq.Clone(ctx)
req.Header.Set("Authorization", "Bearer "+token)
if err := rewindRequestBody(req); err != nil {
return nil, err
}
resp, err := c.send(req)
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusUnauthorized {
return resp, nil
}
resp.Body.Close()
}
}
// attempt with credentials
realm := params["realm"]
service := params["service"]
token, err := cache.Set(ctx, registry, SchemeBearer, key, func(ctx context.Context) (string, error) {
return c.fetchBearerToken(ctx, registry, realm, service, scopes)
})
if err != nil {
return nil, fmt.Errorf("%s %q: %w", resp.Request.Method, resp.Request.URL, err)
}
req = originalReq.Clone(ctx)
req.Header.Set("Authorization", "Bearer "+token)
default:
return resp, nil
}
if err := rewindRequestBody(req); err != nil {
return nil, err
}
return c.send(req)
}
// fetchBasicAuth fetches a basic auth token for the basic challenge.
func (c *Client) fetchBasicAuth(ctx context.Context, registry string) (string, error) {
cred, err := c.credential(ctx, registry)
if err != nil {
return "", fmt.Errorf("failed to resolve credential: %w", err)
}
if cred == EmptyCredential {
return "", errors.New("credential required for basic auth")
}
if cred.Username == "" || cred.Password == "" {
return "", errors.New("missing username or password for basic auth")
}
auth := cred.Username + ":" + cred.Password
return base64.StdEncoding.EncodeToString([]byte(auth)), nil
}
// fetchBearerToken fetches an access token for the bearer challenge.
func (c *Client) fetchBearerToken(ctx context.Context, registry, realm, service string, scopes []string) (string, error) {
cred, err := c.credential(ctx, registry)
if err != nil {
return "", err
}
if cred.AccessToken != "" {
return cred.AccessToken, nil
}
if cred == EmptyCredential || (cred.RefreshToken == "" && !c.ForceAttemptOAuth2) {
return c.fetchDistributionToken(ctx, realm, service, scopes, cred.Username, cred.Password)
}
return c.fetchOAuth2Token(ctx, realm, service, scopes, cred)
}
// fetchDistributionToken fetches an access token as defined by the distribution
// specification.
// It fetches anonymous tokens if no credential is provided.
// References:
// - https://docs.docker.com/registry/spec/auth/jwt/
// - https://docs.docker.com/registry/spec/auth/token/
func (c *Client) fetchDistributionToken(ctx context.Context, realm, service string, scopes []string, username, password string) (string, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, realm, nil)
if err != nil {
return "", err
}
if username != "" || password != "" {
req.SetBasicAuth(username, password)
}
q := req.URL.Query()
if service != "" {
q.Set("service", service)
}
for _, scope := range scopes {
q.Add("scope", scope)
}
req.URL.RawQuery = q.Encode()
resp, err := c.send(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return "", errutil.ParseErrorResponse(resp)
}
// As specified in https://docs.docker.com/registry/spec/auth/token/ section
// "Token Response Fields", the token is either in `token` or
// `access_token`. If both present, they are identical.
var result struct {
Token string `json:"token"`
AccessToken string `json:"access_token"`
}
lr := io.LimitReader(resp.Body, maxResponseBytes)
if err := json.NewDecoder(lr).Decode(&result); err != nil {
return "", fmt.Errorf("%s %q: failed to decode response: %w", resp.Request.Method, resp.Request.URL, err)
}
if result.AccessToken != "" {
return result.AccessToken, nil
}
if result.Token != "" {
return result.Token, nil
}
return "", fmt.Errorf("%s %q: empty token returned", resp.Request.Method, resp.Request.URL)
}
// fetchOAuth2Token fetches an OAuth2 access token.
// Reference: https://docs.docker.com/registry/spec/auth/oauth/
func (c *Client) fetchOAuth2Token(ctx context.Context, realm, service string, scopes []string, cred Credential) (string, error) {
form := url.Values{}
if cred.RefreshToken != "" {
form.Set("grant_type", "refresh_token")
form.Set("refresh_token", cred.RefreshToken)
} else if cred.Username != "" && cred.Password != "" {
form.Set("grant_type", "password")
form.Set("username", cred.Username)
form.Set("password", cred.Password)
} else {
return "", errors.New("missing username or password for bearer auth")
}
form.Set("service", service)
clientID := c.ClientID
if clientID == "" {
clientID = defaultClientID
}
form.Set("client_id", clientID)
if len(scopes) != 0 {
form.Set("scope", strings.Join(scopes, " "))
}
body := strings.NewReader(form.Encode())
req, err := http.NewRequestWithContext(ctx, http.MethodPost, realm, body)
if err != nil {
return "", err
}
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
resp, err := c.send(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return "", errutil.ParseErrorResponse(resp)
}
var result struct {
AccessToken string `json:"access_token"`
}
lr := io.LimitReader(resp.Body, maxResponseBytes)
if err := json.NewDecoder(lr).Decode(&result); err != nil {
return "", fmt.Errorf("%s %q: failed to decode response: %w", resp.Request.Method, resp.Request.URL, err)
}
if result.AccessToken != "" {
return result.AccessToken, nil
}
return "", fmt.Errorf("%s %q: empty token returned", resp.Request.Method, resp.Request.URL)
}
// rewindRequestBody tries to rewind the request body if exists.
func rewindRequestBody(req *http.Request) error {
if req.Body == nil || req.Body == http.NoBody {
return nil
}
if req.GetBody == nil {
return fmt.Errorf("%s %q: request body is not rewindable", req.Method, req.URL)
}
body, err := req.GetBody()
if err != nil {
return fmt.Errorf("%s %q: failed to get request body: %w", req.Method, req.URL, err)
}
req.Body = body
return nil
}

View File

@@ -0,0 +1,40 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package auth
// EmptyCredential represents an empty credential.
var EmptyCredential Credential
// Credential contains authentication credentials used to access remote
// registries.
type Credential struct {
// Username is the name of the user for the remote registry.
Username string
// Password is the secret associated with the username.
Password string
// RefreshToken is a bearer token to be sent to the authorization service
// for fetching access tokens.
// A refresh token is often referred as an identity token.
// Reference: https://docs.docker.com/registry/spec/auth/oauth/
RefreshToken string
// AccessToken is a bearer token to be sent to the registry.
// An access token is often referred as a registry token.
// Reference: https://docs.docker.com/registry/spec/auth/token/
AccessToken string
}

View File

@@ -0,0 +1,236 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package auth
import (
"context"
"sort"
"strings"
)
// Actions used in scopes.
// Reference: https://docs.docker.com/registry/spec/auth/scope/
const (
// ActionPull represents generic read access for resources of the repository
// type.
ActionPull = "pull"
// ActionPush represents generic write access for resources of the
// repository type.
ActionPush = "push"
// ActionDelete represents the delete permission for resources of the
// repository type.
ActionDelete = "delete"
)
// ScopeRegistryCatalog is the scope for registry catalog access.
const ScopeRegistryCatalog = "registry:catalog:*"
// ScopeRepository returns a repository scope with given actions.
// Reference: https://docs.docker.com/registry/spec/auth/scope/
func ScopeRepository(repository string, actions ...string) string {
actions = cleanActions(actions)
if repository == "" || len(actions) == 0 {
return ""
}
return strings.Join([]string{
"repository",
repository,
strings.Join(actions, ","),
}, ":")
}
// scopesContextKey is the context key for scopes.
type scopesContextKey struct{}
// WithScopes returns a context with scopes added. Scopes are de-duplicated.
// Scopes are used as hints for the auth client to fetch bearer tokens with
// larger scopes.
//
// For example, uploading blob to the repository "hello-world" does HEAD request
// first then POST and PUT. The HEAD request will return a challenge for scope
// `repository:hello-world:pull`, and the auth client will fetch a token for
// that challenge. Later, the POST request will return a challenge for scope
// `repository:hello-world:push`, and the auth client will fetch a token for
// that challenge again. By invoking `WithScopes()` with the scope
// `repository:hello-world:pull,push`, the auth client with cache is hinted to
// fetch a token via a single token fetch request for all the HEAD, POST, PUT
// requests.
//
// Passing an empty list of scopes will virtually remove the scope hints in the
// context.
//
// Reference: https://docs.docker.com/registry/spec/auth/scope/
func WithScopes(ctx context.Context, scopes ...string) context.Context {
scopes = CleanScopes(scopes)
return context.WithValue(ctx, scopesContextKey{}, scopes)
}
// AppendScopes appends additional scopes to the existing scopes in the context
// and returns a new context. The resulted scopes are de-duplicated.
// The append operation does modify the existing scope in the context passed in.
func AppendScopes(ctx context.Context, scopes ...string) context.Context {
if len(scopes) == 0 {
return ctx
}
return WithScopes(ctx, append(GetScopes(ctx), scopes...)...)
}
// GetScopes returns the scopes in the context.
func GetScopes(ctx context.Context) []string {
if scopes, ok := ctx.Value(scopesContextKey{}).([]string); ok {
return append([]string(nil), scopes...)
}
return nil
}
// CleanScopes merges and sort the actions in ascending order if the scopes have
// the same resource type and name. The final scopes are sorted in ascending
// order. In other words, the scopes passed in are de-duplicated and sorted.
// Therefore, the output of this function is deterministic.
//
// If there is a wildcard `*` in the action, other actions in the same resource
// type and name are ignored.
func CleanScopes(scopes []string) []string {
// fast paths
switch len(scopes) {
case 0:
return nil
case 1:
scope := scopes[0]
i := strings.LastIndex(scope, ":")
if i == -1 {
return []string{scope}
}
actionList := strings.Split(scope[i+1:], ",")
actionList = cleanActions(actionList)
if len(actionList) == 0 {
return nil
}
actions := strings.Join(actionList, ",")
scope = scope[:i+1] + actions
return []string{scope}
}
// slow path
var result []string
// merge recognizable scopes
resourceTypes := make(map[string]map[string]map[string]struct{})
for _, scope := range scopes {
// extract resource type
i := strings.Index(scope, ":")
if i == -1 {
result = append(result, scope)
continue
}
resourceType := scope[:i]
// extract resource name and actions
rest := scope[i+1:]
i = strings.LastIndex(rest, ":")
if i == -1 {
result = append(result, scope)
continue
}
resourceName := rest[:i]
actions := rest[i+1:]
if actions == "" {
// drop scope since no action found
continue
}
// add to the intermediate map for de-duplication
namedActions := resourceTypes[resourceType]
if namedActions == nil {
namedActions = make(map[string]map[string]struct{})
resourceTypes[resourceType] = namedActions
}
actionSet := namedActions[resourceName]
if actionSet == nil {
actionSet = make(map[string]struct{})
namedActions[resourceName] = actionSet
}
for _, action := range strings.Split(actions, ",") {
if action != "" {
actionSet[action] = struct{}{}
}
}
}
// reconstruct scopes
for resourceType, namedActions := range resourceTypes {
for resourceName, actionSet := range namedActions {
if len(actionSet) == 0 {
continue
}
var actions []string
for action := range actionSet {
if action == "*" {
actions = []string{"*"}
break
}
actions = append(actions, action)
}
sort.Strings(actions)
scope := resourceType + ":" + resourceName + ":" + strings.Join(actions, ",")
result = append(result, scope)
}
}
// sort and return
sort.Strings(result)
return result
}
// cleanActions removes the duplicated actions and sort in ascending order.
// If there is a wildcard `*` in the action, other actions are ignored.
func cleanActions(actions []string) []string {
// fast paths
switch len(actions) {
case 0:
return nil
case 1:
if actions[0] == "" {
return nil
}
return actions
}
// slow path
sort.Strings(actions)
n := 0
for i := 0; i < len(actions); i++ {
if actions[i] == "*" {
return []string{"*"}
}
if actions[i] != actions[n] {
n++
if n != i {
actions[n] = actions[i]
}
}
}
n++
if actions[0] == "" {
if n == 1 {
return nil
}
return actions[1:n]
}
return actions[:n]
}

View File

@@ -0,0 +1,128 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package errcode
import (
"fmt"
"net/http"
"net/url"
"strings"
"unicode"
)
// References:
// - https://github.com/opencontainers/distribution-spec/blob/v1.1.0-rc1/spec.md#error-codes
// - https://docs.docker.com/registry/spec/api/#errors-2
const (
ErrorCodeBlobUnknown = "BLOB_UNKNOWN"
ErrorCodeBlobUploadInvalid = "BLOB_UPLOAD_INVALID"
ErrorCodeBlobUploadUnknown = "BLOB_UPLOAD_UNKNOWN"
ErrorCodeDigestInvalid = "DIGEST_INVALID"
ErrorCodeManifestBlobUnknown = "MANIFEST_BLOB_UNKNOWN"
ErrorCodeManifestInvalid = "MANIFEST_INVALID"
ErrorCodeManifestUnknown = "MANIFEST_UNKNOWN"
ErrorCodeNameInvalid = "NAME_INVALID"
ErrorCodeNameUnknown = "NAME_UNKNOWN"
ErrorCodeSizeInvalid = "SIZE_INVALID"
ErrorCodeUnauthorized = "UNAUTHORIZED"
ErrorCodeDenied = "DENIED"
ErrorCodeUnsupported = "UNSUPPORTED"
)
// Error represents a response inner error returned by the remote
// registry.
// References:
// - https://github.com/opencontainers/distribution-spec/blob/v1.1.0-rc1/spec.md#error-codes
// - https://docs.docker.com/registry/spec/api/#errors-2
type Error struct {
Code string `json:"code"`
Message string `json:"message"`
Detail any `json:"detail,omitempty"`
}
// Error returns a error string describing the error.
func (e Error) Error() string {
code := strings.Map(func(r rune) rune {
if r == '_' {
return ' '
}
return unicode.ToLower(r)
}, e.Code)
if e.Message == "" {
return code
}
if e.Detail == nil {
return fmt.Sprintf("%s: %s", code, e.Message)
}
return fmt.Sprintf("%s: %s: %v", code, e.Message, e.Detail)
}
// Errors represents a list of response inner errors returned by the remote
// server.
// References:
// - https://github.com/opencontainers/distribution-spec/blob/v1.1.0-rc1/spec.md#error-codes
// - https://docs.docker.com/registry/spec/api/#errors-2
type Errors []Error
// Error returns a error string describing the error.
func (errs Errors) Error() string {
switch len(errs) {
case 0:
return "<nil>"
case 1:
return errs[0].Error()
}
var errmsgs []string
for _, err := range errs {
errmsgs = append(errmsgs, err.Error())
}
return strings.Join(errmsgs, "; ")
}
// Unwrap returns the inner error only when there is exactly one error.
func (errs Errors) Unwrap() error {
if len(errs) == 1 {
return errs[0]
}
return nil
}
// ErrorResponse represents an error response.
type ErrorResponse struct {
Method string
URL *url.URL
StatusCode int
Errors Errors
}
// Error returns a error string describing the error.
func (err *ErrorResponse) Error() string {
var errmsg string
if len(err.Errors) > 0 {
errmsg = err.Errors.Error()
} else {
errmsg = http.StatusText(err.StatusCode)
}
return fmt.Sprintf("%s %q: response status code %d: %s", err.Method, err.URL, err.StatusCode, errmsg)
}
// Unwrap returns the internal errors of err if any.
func (err *ErrorResponse) Unwrap() error {
if len(err.Errors) == 0 {
return nil
}
return err.Errors
}

View File

@@ -0,0 +1,54 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package errutil
import (
"encoding/json"
"errors"
"io"
"net/http"
"oras.land/oras-go/v2/registry/remote/errcode"
)
// maxErrorBytes specifies the default limit on how many response bytes are
// allowed in the server's error response.
// A typical error message is around 200 bytes. Hence, 8 KiB should be
// sufficient.
const maxErrorBytes int64 = 8 * 1024 // 8 KiB
// ParseErrorResponse parses the error returned by the remote registry.
func ParseErrorResponse(resp *http.Response) error {
resultErr := &errcode.ErrorResponse{
Method: resp.Request.Method,
URL: resp.Request.URL,
StatusCode: resp.StatusCode,
}
var body struct {
Errors errcode.Errors `json:"errors"`
}
lr := io.LimitReader(resp.Body, maxErrorBytes)
if err := json.NewDecoder(lr).Decode(&body); err == nil {
resultErr.Errors = body.Errors
}
return resultErr
}
// IsErrorCode returns true if err is an Error and its Code equals to code.
func IsErrorCode(err error, code string) bool {
var ec errcode.Error
return errors.As(err, &ec) && ec.Code == code
}

View File

@@ -0,0 +1,58 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package remote
import (
"strings"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2/internal/docker"
)
// defaultManifestMediaTypes contains the default set of manifests media types.
var defaultManifestMediaTypes = []string{
docker.MediaTypeManifest,
docker.MediaTypeManifestList,
ocispec.MediaTypeImageManifest,
ocispec.MediaTypeImageIndex,
ocispec.MediaTypeArtifactManifest,
}
// defaultManifestAcceptHeader is the default set in the `Accept` header for
// resolving manifests from tags.
var defaultManifestAcceptHeader = strings.Join(defaultManifestMediaTypes, ", ")
// isManifest determines if the given descriptor points to a manifest.
func isManifest(manifestMediaTypes []string, desc ocispec.Descriptor) bool {
if len(manifestMediaTypes) == 0 {
manifestMediaTypes = defaultManifestMediaTypes
}
for _, mediaType := range manifestMediaTypes {
if desc.MediaType == mediaType {
return true
}
}
return false
}
// manifestAcceptHeader generates the set in the `Accept` header for resolving
// manifests from tags.
func manifestAcceptHeader(manifestMediaTypes []string) string {
if len(manifestMediaTypes) == 0 {
return defaultManifestAcceptHeader
}
return strings.Join(manifestMediaTypes, ", ")
}

View File

@@ -0,0 +1,191 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package remote
import (
"errors"
"strings"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2/content"
"oras.land/oras-go/v2/internal/descriptor"
)
// zeroDigest represents a digest that consists of zeros. zeroDigest is used
// for pinging Referrers API.
const zeroDigest = "sha256:0000000000000000000000000000000000000000000000000000000000000000"
// referrersState represents the state of Referrers API.
type referrersState = int32
const (
// referrersStateUnknown represents an unknown state of Referrers API.
referrersStateUnknown referrersState = iota
// referrersStateSupported represents that the repository is known to
// support Referrers API.
referrersStateSupported
// referrersStateUnsupported represents that the repository is known to
// not support Referrers API.
referrersStateUnsupported
)
// referrerOperation represents an operation on a referrer.
type referrerOperation = int32
const (
// referrerOperationAdd represents an addition operation on a referrer.
referrerOperationAdd referrerOperation = iota
// referrerOperationRemove represents a removal operation on a referrer.
referrerOperationRemove
)
// referrerChange represents a change on a referrer.
type referrerChange struct {
referrer ocispec.Descriptor
operation referrerOperation
}
var (
// ErrReferrersCapabilityAlreadySet is returned by SetReferrersCapability()
// when the Referrers API capability has been already set.
ErrReferrersCapabilityAlreadySet = errors.New("referrers capability cannot be changed once set")
// errNoReferrerUpdate is returned by applyReferrerChanges() when there
// is no any referrer update.
errNoReferrerUpdate = errors.New("no referrer update")
)
// buildReferrersTag builds the referrers tag for the given manifest descriptor.
// Format: <algorithm>-<digest>
// Reference: https://github.com/opencontainers/distribution-spec/blob/v1.1.0-rc1/spec.md#unavailable-referrers-api
func buildReferrersTag(desc ocispec.Descriptor) string {
alg := desc.Digest.Algorithm().String()
encoded := desc.Digest.Encoded()
return alg + "-" + encoded
}
// isReferrersFilterApplied checks annotations to see if requested is in the
// applied filter list.
func isReferrersFilterApplied(annotations map[string]string, requested string) bool {
applied := annotations[ocispec.AnnotationReferrersFiltersApplied]
if applied == "" || requested == "" {
return false
}
filters := strings.Split(applied, ",")
for _, f := range filters {
if f == requested {
return true
}
}
return false
}
// filterReferrers filters a slice of referrers by artifactType in place.
// The returned slice contains matching referrers.
func filterReferrers(refs []ocispec.Descriptor, artifactType string) []ocispec.Descriptor {
if artifactType == "" {
return refs
}
var j int
for i, ref := range refs {
if ref.ArtifactType == artifactType {
if i != j {
refs[j] = ref
}
j++
}
}
return refs[:j]
}
// applyReferrerChanges applies referrerChanges on referrers and returns the
// updated referrers.
// Returns errNoReferrerUpdate if there is no any referrers updates.
func applyReferrerChanges(referrers []ocispec.Descriptor, referrerChanges []referrerChange) ([]ocispec.Descriptor, error) {
referrersMap := make(map[descriptor.Descriptor]int, len(referrers)+len(referrerChanges))
updatedReferrers := make([]ocispec.Descriptor, 0, len(referrers)+len(referrerChanges))
var updateRequired bool
for _, r := range referrers {
if content.Equal(r, ocispec.Descriptor{}) {
// skip bad entry
updateRequired = true
continue
}
key := descriptor.FromOCI(r)
if _, ok := referrersMap[key]; ok {
// skip duplicates
updateRequired = true
continue
}
updatedReferrers = append(updatedReferrers, r)
referrersMap[key] = len(updatedReferrers) - 1
}
// apply changes
for _, change := range referrerChanges {
key := descriptor.FromOCI(change.referrer)
switch change.operation {
case referrerOperationAdd:
if _, ok := referrersMap[key]; !ok {
// add distinct referrers
updatedReferrers = append(updatedReferrers, change.referrer)
referrersMap[key] = len(updatedReferrers) - 1
}
case referrerOperationRemove:
if pos, ok := referrersMap[key]; ok {
// remove referrers that are already in the map
updatedReferrers[pos] = ocispec.Descriptor{}
delete(referrersMap, key)
}
}
}
// skip unnecessary update
if !updateRequired && len(referrersMap) == len(referrers) {
// if the result referrer map contains the same content as the
// original referrers, consider that there is no update on the
// referrers.
for _, r := range referrers {
key := descriptor.FromOCI(r)
if _, ok := referrersMap[key]; !ok {
updateRequired = true
}
}
if !updateRequired {
return nil, errNoReferrerUpdate
}
}
return removeEmptyDescriptors(updatedReferrers, len(referrersMap)), nil
}
// removeEmptyDescriptors in-place removes empty items from descs, given a hint
// of the number of non-empty descriptors.
func removeEmptyDescriptors(descs []ocispec.Descriptor, hint int) []ocispec.Descriptor {
j := 0
for i, r := range descs {
if !content.Equal(r, ocispec.Descriptor{}) {
if i > j {
descs[j] = r
}
j++
}
if j == hint {
break
}
}
return descs[:j]
}

View File

@@ -0,0 +1,175 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package remote provides a client to the remote registry.
// Reference: https://github.com/distribution/distribution
package remote
import (
"context"
"encoding/json"
"fmt"
"net/http"
"strconv"
"oras.land/oras-go/v2/errdef"
"oras.land/oras-go/v2/registry"
"oras.land/oras-go/v2/registry/remote/auth"
"oras.land/oras-go/v2/registry/remote/internal/errutil"
)
// RepositoryOptions is an alias of Repository to avoid name conflicts.
// It also hides all methods associated with Repository.
type RepositoryOptions Repository
// Registry is an HTTP client to a remote registry.
type Registry struct {
// RepositoryOptions contains common options for Registry and Repository.
// It is also used as a template for derived repositories.
RepositoryOptions
// RepositoryListPageSize specifies the page size when invoking the catalog
// API.
// If zero, the page size is determined by the remote registry.
// Reference: https://docs.docker.com/registry/spec/api/#catalog
RepositoryListPageSize int
}
// NewRegistry creates a client to the remote registry with the specified domain
// name.
// Example: localhost:5000
func NewRegistry(name string) (*Registry, error) {
ref := registry.Reference{
Registry: name,
}
if err := ref.ValidateRegistry(); err != nil {
return nil, err
}
return &Registry{
RepositoryOptions: RepositoryOptions{
Reference: ref,
},
}, nil
}
// client returns an HTTP client used to access the remote registry.
// A default HTTP client is return if the client is not configured.
func (r *Registry) client() Client {
if r.Client == nil {
return auth.DefaultClient
}
return r.Client
}
// Ping checks whether or not the registry implement Docker Registry API V2 or
// OCI Distribution Specification.
// Ping can be used to check authentication when an auth client is configured.
//
// References:
// - https://docs.docker.com/registry/spec/api/#base
// - https://github.com/opencontainers/distribution-spec/blob/v1.1.0-rc1/spec.md#api
func (r *Registry) Ping(ctx context.Context) error {
url := buildRegistryBaseURL(r.PlainHTTP, r.Reference)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return err
}
resp, err := r.client().Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
switch resp.StatusCode {
case http.StatusOK:
return nil
case http.StatusNotFound:
return errdef.ErrNotFound
default:
return errutil.ParseErrorResponse(resp)
}
}
// Repositories lists the name of repositories available in the registry.
// See also `RepositoryListPageSize`.
//
// If `last` is NOT empty, the entries in the response start after the
// repo specified by `last`. Otherwise, the response starts from the top
// of the Repositories list.
//
// Reference: https://docs.docker.com/registry/spec/api/#catalog
func (r *Registry) Repositories(ctx context.Context, last string, fn func(repos []string) error) error {
ctx = auth.AppendScopes(ctx, auth.ScopeRegistryCatalog)
url := buildRegistryCatalogURL(r.PlainHTTP, r.Reference)
var err error
for err == nil {
url, err = r.repositories(ctx, last, fn, url)
// clear `last` for subsequent pages
last = ""
}
if err != errNoLink {
return err
}
return nil
}
// repositories returns a single page of repository list with the next link.
func (r *Registry) repositories(ctx context.Context, last string, fn func(repos []string) error, url string) (string, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return "", err
}
if r.RepositoryListPageSize > 0 || last != "" {
q := req.URL.Query()
if r.RepositoryListPageSize > 0 {
q.Set("n", strconv.Itoa(r.RepositoryListPageSize))
}
if last != "" {
q.Set("last", last)
}
req.URL.RawQuery = q.Encode()
}
resp, err := r.client().Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return "", errutil.ParseErrorResponse(resp)
}
var page struct {
Repositories []string `json:"repositories"`
}
lr := limitReader(resp.Body, r.MaxMetadataBytes)
if err := json.NewDecoder(lr).Decode(&page); err != nil {
return "", fmt.Errorf("%s %q: failed to decode response: %w", resp.Request.Method, resp.Request.URL, err)
}
if err := fn(page.Repositories); err != nil {
return "", err
}
return parseLink(resp)
}
// Repository returns a repository reference by the given name.
func (r *Registry) Repository(ctx context.Context, name string) (registry.Repository, error) {
ref := registry.Reference{
Registry: r.Reference.Registry,
Repository: name,
}
return newRepositoryWithOptions(ref, &r.RepositoryOptions)
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,114 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package retry
import (
"net/http"
"time"
)
// DefaultClient is a client with the default retry policy.
var DefaultClient = NewClient()
// NewClient creates an HTTP client with the default retry policy.
func NewClient() *http.Client {
return &http.Client{
Transport: NewTransport(nil),
}
}
// Transport is an HTTP transport with retry policy.
type Transport struct {
// Base is the underlying HTTP transport to use.
// If nil, http.DefaultTransport is used for round trips.
Base http.RoundTripper
// Policy returns a retry Policy to use for the request.
// If nil, DefaultPolicy is used to determine if the request should be retried.
Policy func() Policy
}
// NewTransport creates an HTTP Transport with the default retry policy.
func NewTransport(base http.RoundTripper) *Transport {
return &Transport{
Base: base,
}
}
// RoundTrip executes a single HTTP transaction, returning a Response for the
// provided Request.
// It relies on the configured Policy to determine if the request should be
// retried and to backoff.
func (t *Transport) RoundTrip(req *http.Request) (*http.Response, error) {
ctx := req.Context()
policy := t.policy()
attempt := 0
for {
resp, respErr := t.roundTrip(req)
duration, err := policy.Retry(attempt, resp, respErr)
if err != nil {
if respErr == nil {
resp.Body.Close()
}
return nil, err
}
if duration < 0 {
return resp, respErr
}
// rewind the body if possible
if req.Body != nil {
if req.GetBody == nil {
// body can't be rewound, so we can't retry
return resp, respErr
}
body, err := req.GetBody()
if err != nil {
// failed to rewind the body, so we can't retry
return resp, respErr
}
req.Body = body
}
// close the response body if needed
if respErr == nil {
resp.Body.Close()
}
timer := time.NewTimer(duration)
select {
case <-ctx.Done():
timer.Stop()
return nil, ctx.Err()
case <-timer.C:
}
attempt++
}
}
func (t *Transport) roundTrip(req *http.Request) (*http.Response, error) {
if t.Base == nil {
return http.DefaultTransport.RoundTrip(req)
}
return t.Base.RoundTrip(req)
}
func (t *Transport) policy() Policy {
if t.Policy == nil {
return DefaultPolicy
}
return t.Policy()
}

View File

@@ -0,0 +1,154 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package retry
import (
"hash/maphash"
"math"
"math/rand"
"net"
"net/http"
"strconv"
"time"
)
// headerRetryAfter is the header key for Retry-After.
const headerRetryAfter = "Retry-After"
// DefaultPolicy is a policy with fine-tuned retry parameters.
// It uses an exponential backoff with jitter.
var DefaultPolicy Policy = &GenericPolicy{
Retryable: DefaultPredicate,
Backoff: DefaultBackoff,
MinWait: 200 * time.Millisecond,
MaxWait: 3 * time.Second,
MaxRetry: 5,
}
// DefaultPredicate is a predicate that retries on 5xx errors, 429 Too Many
// Requests, 408 Request Timeout and on network dial timeout.
var DefaultPredicate Predicate = func(resp *http.Response, err error) (bool, error) {
if err != nil {
// retry on Dial timeout
if err, ok := err.(net.Error); ok && err.Timeout() {
return true, nil
}
return false, err
}
if resp.StatusCode == http.StatusRequestTimeout || resp.StatusCode == http.StatusTooManyRequests {
return true, nil
}
if resp.StatusCode == 0 || resp.StatusCode >= 500 {
return true, nil
}
return false, nil
}
// DefaultBackoff is a backoff that uses an exponential backoff with jitter.
// It uses a base of 250ms, a factor of 2 and a jitter of 10%.
var DefaultBackoff Backoff = ExponentialBackoff(250*time.Millisecond, 2, 0.1)
// Policy is a retry policy.
type Policy interface {
// Retry returns the duration to wait before retrying the request.
// It returns a negative value if the request should not be retried.
// The attempt is used to:
// - calculate the backoff duration, the default backoff is an exponential backoff.
// - determine if the request should be retried.
// The attempt starts at 0 and should be less than MaxRetry for the request to
// be retried.
Retry(attempt int, resp *http.Response, err error) (time.Duration, error)
}
// Predicate is a function that returns true if the request should be retried.
type Predicate func(resp *http.Response, err error) (bool, error)
// Backoff is a function that returns the duration to wait before retrying the
// request. The attempt, is the next attempt number. The response is the
// response from the previous request.
type Backoff func(attempt int, resp *http.Response) time.Duration
// ExponentialBackoff returns a Backoff that uses an exponential backoff with
// jitter. The backoff is calculated as:
//
// temp = backoff * factor ^ attempt
// interval = temp * (1 - jitter) + rand.Int63n(2 * jitter * temp)
//
// The HTTP response is checked for a Retry-After header. If it is present, the
// value is used as the backoff duration.
func ExponentialBackoff(backoff time.Duration, factor, jitter float64) Backoff {
return func(attempt int, resp *http.Response) time.Duration {
var h maphash.Hash
h.SetSeed(maphash.MakeSeed())
rand := rand.New(rand.NewSource(int64(h.Sum64())))
// check Retry-After
if resp != nil && resp.StatusCode == http.StatusTooManyRequests {
if v := resp.Header.Get(headerRetryAfter); v != "" {
if retryAfter, _ := strconv.ParseInt(v, 10, 64); retryAfter > 0 {
return time.Duration(retryAfter) * time.Second
}
}
}
// do exponential backoff with jitter
temp := float64(backoff) * math.Pow(factor, float64(attempt))
return time.Duration(temp*(1-jitter)) + time.Duration(rand.Int63n(int64(2*jitter*temp)))
}
}
// GenericPolicy is a generic retry policy.
type GenericPolicy struct {
// Retryable is a predicate that returns true if the request should be
// retried.
Retryable Predicate
// Backoff is a function that returns the duration to wait before retrying.
Backoff Backoff
// MinWait is the minimum duration to wait before retrying.
MinWait time.Duration
// MaxWait is the maximum duration to wait before retrying.
MaxWait time.Duration
// MaxRetry is the maximum number of retries.
MaxRetry int
}
// Retry returns the duration to wait before retrying the request.
// It returns -1 if the request should not be retried.
func (p *GenericPolicy) Retry(attempt int, resp *http.Response, err error) (time.Duration, error) {
if attempt >= p.MaxRetry {
return -1, nil
}
if ok, err := p.Retryable(resp, err); err != nil {
return -1, err
} else if !ok {
return -1, nil
}
backoff := p.Backoff(attempt, resp)
if backoff < p.MinWait {
backoff = p.MinWait
}
if backoff > p.MaxWait {
backoff = p.MaxWait
}
return backoff, nil
}

View File

@@ -0,0 +1,107 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package remote
import (
"fmt"
"net/url"
"strings"
"oras.land/oras-go/v2/registry"
)
// buildScheme returns HTTP scheme used to access the remote registry.
func buildScheme(plainHTTP bool) string {
if plainHTTP {
return "http"
}
return "https"
}
// buildRegistryBaseURL builds the URL for accessing the base API.
// Format: <scheme>://<registry>/v2/
// Reference: https://docs.docker.com/registry/spec/api/#base
func buildRegistryBaseURL(plainHTTP bool, ref registry.Reference) string {
return fmt.Sprintf("%s://%s/v2/", buildScheme(plainHTTP), ref.Host())
}
// buildRegistryCatalogURL builds the URL for accessing the catalog API.
// Format: <scheme>://<registry>/v2/_catalog
// Reference: https://docs.docker.com/registry/spec/api/#catalog
func buildRegistryCatalogURL(plainHTTP bool, ref registry.Reference) string {
return fmt.Sprintf("%s://%s/v2/_catalog", buildScheme(plainHTTP), ref.Host())
}
// buildRepositoryBaseURL builds the base endpoint of the remote repository.
// Format: <scheme>://<registry>/v2/<repository>
func buildRepositoryBaseURL(plainHTTP bool, ref registry.Reference) string {
return fmt.Sprintf("%s://%s/v2/%s", buildScheme(plainHTTP), ref.Host(), ref.Repository)
}
// buildRepositoryTagListURL builds the URL for accessing the tag list API.
// Format: <scheme>://<registry>/v2/<repository>/tags/list
// Reference: https://docs.docker.com/registry/spec/api/#tags
func buildRepositoryTagListURL(plainHTTP bool, ref registry.Reference) string {
return buildRepositoryBaseURL(plainHTTP, ref) + "/tags/list"
}
// buildRepositoryManifestURL builds the URL for accessing the manifest API.
// Format: <scheme>://<registry>/v2/<repository>/manifests/<digest_or_tag>
// Reference: https://docs.docker.com/registry/spec/api/#manifest
func buildRepositoryManifestURL(plainHTTP bool, ref registry.Reference) string {
return strings.Join([]string{
buildRepositoryBaseURL(plainHTTP, ref),
"manifests",
ref.Reference,
}, "/")
}
// buildRepositoryBlobURL builds the URL for accessing the blob API.
// Format: <scheme>://<registry>/v2/<repository>/blobs/<digest>
// Reference: https://docs.docker.com/registry/spec/api/#blob
func buildRepositoryBlobURL(plainHTTP bool, ref registry.Reference) string {
return strings.Join([]string{
buildRepositoryBaseURL(plainHTTP, ref),
"blobs",
ref.Reference,
}, "/")
}
// buildRepositoryBlobUploadURL builds the URL for blob uploading.
// Format: <scheme>://<registry>/v2/<repository>/blobs/uploads/
// Reference: https://docs.docker.com/registry/spec/api/#initiate-blob-upload
func buildRepositoryBlobUploadURL(plainHTTP bool, ref registry.Reference) string {
return buildRepositoryBaseURL(plainHTTP, ref) + "/blobs/uploads/"
}
// buildReferrersURL builds the URL for querying the Referrers API.
// Format: <scheme>://<registry>/v2/<repository>/referrers/<digest>?artifactType=<artifactType>
// Reference: https://github.com/opencontainers/distribution-spec/blob/v1.1.0-rc1/spec.md#listing-referrers
func buildReferrersURL(plainHTTP bool, ref registry.Reference, artifactType string) string {
var query string
if artifactType != "" {
v := url.Values{}
v.Set("artifactType", artifactType)
query = "?" + v.Encode()
}
return fmt.Sprintf(
"%s/referrers/%s%s",
buildRepositoryBaseURL(plainHTTP, ref),
ref.Reference,
query,
)
}

View File

@@ -0,0 +1,94 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package remote
import (
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"strings"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2/content"
"oras.land/oras-go/v2/errdef"
)
// defaultMaxMetadataBytes specifies the default limit on how many response
// bytes are allowed in the server's response to the metadata APIs.
// See also: Repository.MaxMetadataBytes
var defaultMaxMetadataBytes int64 = 4 * 1024 * 1024 // 4 MiB
// errNoLink is returned by parseLink() when no Link header is present.
var errNoLink = errors.New("no Link header in response")
// parseLink returns the URL of the response's "Link" header, if present.
func parseLink(resp *http.Response) (string, error) {
link := resp.Header.Get("Link")
if link == "" {
return "", errNoLink
}
if link[0] != '<' {
return "", fmt.Errorf("invalid next link %q: missing '<'", link)
}
if i := strings.IndexByte(link, '>'); i == -1 {
return "", fmt.Errorf("invalid next link %q: missing '>'", link)
} else {
link = link[1:i]
}
linkURL, err := resp.Request.URL.Parse(link)
if err != nil {
return "", err
}
return linkURL.String(), nil
}
// limitReader returns a Reader that reads from r but stops with EOF after n
// bytes. If n is less than or equal to zero, defaultMaxMetadataBytes is used.
func limitReader(r io.Reader, n int64) io.Reader {
if n <= 0 {
n = defaultMaxMetadataBytes
}
return io.LimitReader(r, n)
}
// limitSize returns ErrSizeExceedsLimit if the size of desc exceeds the limit n.
// If n is less than or equal to zero, defaultMaxMetadataBytes is used.
func limitSize(desc ocispec.Descriptor, n int64) error {
if n <= 0 {
n = defaultMaxMetadataBytes
}
if desc.Size > n {
return fmt.Errorf(
"content size %v exceeds MaxMetadataBytes %v: %w",
desc.Size,
n,
errdef.ErrSizeExceedsLimit)
}
return nil
}
// decodeJSON safely reads the JSON content described by desc, and
// decodes it into v.
func decodeJSON(r io.Reader, desc ocispec.Descriptor, v any) error {
jsonBytes, err := content.ReadAll(r, desc)
if err != nil {
return err
}
return json.Unmarshal(jsonBytes, v)
}

View File

@@ -0,0 +1,120 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package registry
import (
"context"
"io"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2/content"
)
// Repository is an ORAS target and an union of the blob and the manifest CASs.
//
// As specified by https://docs.docker.com/registry/spec/api/, it is natural to
// assume that content.Resolver interface only works for manifests. Tagging a
// blob may be resulted in an `ErrUnsupported` error. However, this interface
// does not restrict tagging blobs.
//
// Since a repository is an union of the blob and the manifest CASs, all
// operations defined in the `BlobStore` are executed depending on the media
// type of the given descriptor accordingly.
//
// Furthermore, this interface also provides the ability to enforce the
// separation of the blob and the manifests CASs.
type Repository interface {
content.Storage
content.Deleter
content.TagResolver
ReferenceFetcher
ReferencePusher
ReferrerLister
TagLister
// Blobs provides access to the blob CAS only, which contains config blobs,
// layers, and other generic blobs.
Blobs() BlobStore
// Manifests provides access to the manifest CAS only.
Manifests() ManifestStore
}
// BlobStore is a CAS with the ability to stat and delete its content.
type BlobStore interface {
content.Storage
content.Deleter
content.Resolver
ReferenceFetcher
}
// ManifestStore is a CAS with the ability to stat and delete its content.
// Besides, ManifestStore provides reference tagging.
type ManifestStore interface {
BlobStore
content.Tagger
ReferencePusher
}
// ReferencePusher provides advanced push with the tag service.
type ReferencePusher interface {
// PushReference pushes the manifest with a reference tag.
PushReference(ctx context.Context, expected ocispec.Descriptor, content io.Reader, reference string) error
}
// ReferenceFetcher provides advanced fetch with the tag service.
type ReferenceFetcher interface {
// FetchReference fetches the content identified by the reference.
FetchReference(ctx context.Context, reference string) (ocispec.Descriptor, io.ReadCloser, error)
}
// ReferrerLister provides the Referrers API.
// Reference: https://github.com/opencontainers/distribution-spec/blob/v1.1.0-rc1/spec.md#listing-referrers
type ReferrerLister interface {
Referrers(ctx context.Context, desc ocispec.Descriptor, artifactType string, fn func(referrers []ocispec.Descriptor) error) error
}
// TagLister lists tags by the tag service.
type TagLister interface {
// Tags lists the tags available in the repository.
// Since the returned tag list may be paginated by the underlying
// implementation, a function should be passed in to process the paginated
// tag list.
// `last` argument is the `last` parameter when invoking the tags API.
// If `last` is NOT empty, the entries in the response start after the
// tag specified by `last`. Otherwise, the response starts from the top
// of the Tags list.
// Note: When implemented by a remote registry, the tags API is called.
// However, not all registries supports pagination or conforms the
// specification.
// References:
// - https://github.com/opencontainers/distribution-spec/blob/v1.1.0-rc1/spec.md#content-discovery
// - https://docs.docker.com/registry/spec/api/#tags
// See also `Tags()` in this package.
Tags(ctx context.Context, last string, fn func(tags []string) error) error
}
// Tags lists the tags available in the repository.
func Tags(ctx context.Context, repo TagLister) ([]string, error) {
var res []string
if err := repo.Tags(ctx, "", func(tags []string) error {
res = append(res, tags...)
return nil
}); err != nil {
return nil, err
}
return res, nil
}

43
vendor/oras.land/oras-go/v2/target.go vendored Normal file
View File

@@ -0,0 +1,43 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package oras
import "oras.land/oras-go/v2/content"
// Target is a CAS with generic tags.
type Target interface {
content.Storage
content.TagResolver
}
// GraphTarget is a CAS with generic tags that supports direct predecessor node
// finding.
type GraphTarget interface {
content.GraphStorage
content.TagResolver
}
// ReadOnlyTarget represents a read-only Target.
type ReadOnlyTarget interface {
content.ReadOnlyStorage
content.Resolver
}
// ReadOnlyGraphTarget represents a read-only GraphTarget.
type ReadOnlyGraphTarget interface {
content.ReadOnlyGraphStorage
content.Resolver
}