Skip to main content

Contributing

How to contribute to Standard Registry. https://github.com/aquaproj/aqua-registry

See also​

Changelog of document and development workflow​

change log.

Should you create an Issue before sending a Pull Request?​

Basically, you don't have to create an Issue before sending a Pull Request. But if the pull request requires the discussion before reviewing, you have to create an Issue in advance.

For example, you don't have to create an Issue in the following cases.

  • Add a package
  • Fix a typo

On the other hand, for example if you want to change the directory structure in pkgs or the workflow adding a package, you have to create an Issue and describe what is changed and why the change is needed.

aqua can't support some tools' plugin mechanism​

Some tools have the plugin mechanism.

e.g.

aqua simply installs commands in PATH (AQUA_ROOT_DIR/bin), but some of these plugins expect to be installed in the other location. If aqua can't support the plugin, we will reject the pull request adding the plugin to aqua-registry.

So if you send a pull request adding a plugin to aqua-registry, please check if aqua can support the plugin. We aren't necessarily familiar with the plugin, so please explain where the plugin expects to be installed and how the plugin works in the pull request description.

If you don't know well, please create a pull request and consult us.

Note of Programing Language Support​

aqua supports several programing languages such as Go and Node.js, but when we support a programing language, we need to be careful about where the programing language installs libraries and commands.

For instance, if the programing language installs commands in the same directory with the programing language itself, aqua can't add them to $PATH, meaning we can't execute them. aqua doesn't support changing $PATH dynamically (We have no plan to support it as it makes aqua more complicated). Node.js's npm i -g installs the same directory with node by default, so we gave up the support of Node.js before (Now aqua supports Node.js again because we can change the install path by NPM_CONFIG_PREFIX). If the language installs libraries in the same directory with it, the language can't refer installed libraries when we change the version of the language.

So before supporting a programing language, we should consider carefully if it really works well. Many programing languages have dedicated version managers, so maybe they are more appropriate.

Commit Signing​

All commits of pull requests must be signed. Please see the document.

Set up​

  1. Please fork aquaproj/aqua-registry.
  2. Checkout the repository
  3. Run aqua i -l in the repository root directory to install tools which are required for contribution.
aqua i -l

Structure of aqua-registry​

Package-related code is located in the pkgs/<package name> directory of https://github.com/aquaproj/aqua-registry. e.g. cli/cli Each package directory contains the following files:

  • pkg.yaml: List of versions installed during testing. This is essentially test data
  • registry.yaml: Configuration. Each tool's registry.yaml is merged to generate the repository root registry.yaml
  • scaffold.yaml: Optional. Configuration file for commands that auto-generate pkg.yaml and registry.yaml. Required when you want to change the auto-generation behavior
note

pkg.yaml is just test data. You can install versions not included in this file.

There is also a registry.yaml at the repository root, which is a huge YAML file merging all registry.yaml files under pkgs. When specifying Standard Registry in aqua.yaml, this repository root registry.yaml is referenced. To modify the repository root registry.yaml, modify the registry.yaml under pkgs and run the cmdx gr command.

registry.yaml Documentation​

Please refer to Registry Config. There is also a JSON Schema. The registry.yaml files under pkgs have JSON Schema comments, so VSCode and similar editors can provide auto-completion.

Additionally, there are abundant examples under pkgs in aqua-registry. By grepping here, you can see how much each configuration item is used and how to write them for reference.

# Search with slsa_provenance
$ git grep -l slsa_provenance pkgs  
pkgs/Zxilly/go-size-analyzer/registry.yaml
pkgs/aquaproj/aqua-registry-updater/registry.yaml
pkgs/aquaproj/example-go-slsa-provenance/registry.yaml
...

Development Tools​

info

Unfortunately, the current development tools depend on Shell Scripts and are unlikely to work on Windows (though they probably work on WSL). There is an issue to rewrite them in Go.

CLIs for developing aqua-registry are provided. These can be installed with aqua. Check out aqua-registry and run aqua i -l.

aqua i -l

We use a task runner called cmdx. You can check tasks with cmdx help.

cmdx help

Using development tools, you can generate files for each package (pkg.yaml, registry.yaml, scaffold.yaml) and test tool installation in containers. Tests are performed in containers using the docker command, so you need the docker command and a compatible container engine. Docker Desktop would work fine of course. We install docker/cli and abiosoft/colima with aqua. Please confirm that the docker command works.

docker version

cmdx s - Scaffold configuration and test it in containers​

cmdx s <package name> generates a configuration file pkgs/<package name>/registry.yaml and a test data file pkgs/<package name>/pkg.yaml, and tests them in containers. It gets data from GitHub Releases by GitHub API. By default, it gets all releases, so it takes a bit long time if the repository has a lot of releases. cmdx s isn't perfect, but you must use it when you add new packages.

cmdx t - Test a package in containers​

cmdx t [<package name>] tests a package in containers. If the branch name is feat/<package name>, you can omit the argument <package name>. cmdx t copies files pkgs/<package name>/{pkg.yaml,registry.yaml} in containers and test them. If the test succeeds, registry.yaml is updated.

cmdx rm - Remove containers​

cmdx rm removes containers. cmdx s and cmdx t reuse containers, but if you want to test packages in clean environment, you can do it by removing containers.

cmdx rmp - Remove an installed package from containers​

cmdx rmp [<package name>] removes an installed package from containers. If the branch name is feat/<package name>, you can omit the argument <package name>. It runs aqua rm <package name> and removes aqua-checksums.json in containers. This task is useful when you want to test packages in clean environment.

cmdx gr - Update registry.yaml​

cmdx gr merges pkgs/**/registry.yaml and updates registry.yaml. Please don't edit registry.yaml directly. When you edit pkgs/**/registry.yaml, please run cmdx gr to reflect the update to registry.yaml in the repository.

cmdx con - Connect to a container​

cmdx con [<os>] [<arch>] connect to a given container. cmdx s and cmdx t tests packages in containers. cmdx con is useful to look into the trouble in containers. By default, <os> is linux and <arch> is CPU architecture of your machine.

Code Auto-generation​

Writing configuration files for each package from scratch is difficult and has quality issues. Therefore, commands for auto-generating code are provided. When adding a new package, always use this command. Code written manually from scratch is not quality assured, so Pull Requests will not be accepted. However, code auto-generation is not perfect and often generates incomplete code. In that case, you need to manually fix the generated code.

aqua supports various package types, but currently auto-generation mainly supports only github_release and cargo. When generating code for other packages like http package, specify -l 1 to generate only a template and write the rest manually.

cmdx s -l 1 "<package name>"

GitHub Access Token​

Development tools execute GitHub API to get lists of GitHub Releases and assets. It works without an access token, but the possibility of hitting API rate limits increases. Hitting API rate limits can prevent normal code generation or cause tests to fail. You can pass an access token through environment variables GITHUB_TOKEN or AQUA_GITHUB_TOKEN. If these environment variables are not set, it will try to get an access token using the gh auth token command. No special permissions are needed as it only reads public repository resources.

Modifying Existing Packages​

When modifying existing packages, you need to modify code under pkgs/<package name>. There are several modification methods:

  1. Manually modify the code
  2. Regenerate the code from scratch with commands
  3. Auto-generate code for the latest version and manually modify based on that

Which method to use depends on the state of the original code. Code auto-generation has been improved many times. Therefore, there is low-quality code generated before improvements. Such code may be better regenerated from scratch rather than manually fixed.

One characteristic to identify if code is old is how version_constraint and version_overrides are written. In the new style, it basically looks like this:

  version_constraint: "false" # Root version_constraint is "false"
  version_overrides:
    - version_constraint: semver("<= 0.1.0") # Version constraints use <, <= not >, >= (basically <=)
      # ...
    # ...
    - version_constraint: "true" # End with "true" for latest version configuration
      # ...

In the old style, version_overrides is often not defined. In this case, it's likely better to regenerate from scratch. However, as mentioned earlier, auto-generation doesn't support package types other than github_release or cargo, so manual modification will be necessary.

Also, aliases and files cannot be auto-generated, so you need to modify the auto-generated code referring to the original code.

3. Auto-generate code for the latest version and manually modify based on that is effective when the package no longer installs with the latest version but you want to reuse existing code (don't want to regenerate from scratch). Running the following command generates code for the latest version:

aqua gr -l 1 "<package name>"

Fix this and add it to the end of version_overrides in the original code and modify version_constraint.

Manual Modification​

When manual modification is necessary, you'll need to look at error messages and fix appropriately.

When Configuration Needs to Change for Specific Versions​

You can change configuration by version using version_overrides and version_constraint.

When Configuration Needs to Change for Specific OS/Arch​

You can change configuration by OS/Arch with overrides.

When Version Cannot Be Found​

Sometimes a released version is deleted and disappears. In that case, delete that version from pkg.yaml. And delete configuration related to that version from registry.yaml (if any). However, no_asset and error_message don't need to be deleted. You may or may not add no_asset and error_message.

When Asset Cannot Be Found​

When an asset cannot be found, either the asset name is wrong or the asset hasn't been released.

Running the cmdx lsa [-r <repository name>] "<version>" command outputs a list of assets, which is convenient.

$ cmdx lsa -repo suzuki-shunsuke/pinact v3.0.0
+ REPO=${REPO#https://github.com/}
repo=$(bash scripts/get_test_pkg.sh "$REPO")

gh release view --json assets --jq ".assets[].name" -R "$repo" "$VERSION"

multiple.intoto.jsonl
pinact_3.0.0_checksums.txt
pinact_3.0.0_checksums.txt.pem
pinact_3.0.0_checksums.txt.sig
pinact_darwin_amd64.tar.gz
pinact_darwin_arm64.tar.gz
pinact_linux_amd64.tar.gz
pinact_linux_arm64.tar.gz
pinact_windows_amd64.zip
pinact_windows_arm64.zip

It's common for new GitHub Releases or tags to be released without assets being released. When there are no assets, the following causes are possible:

  1. Release is simply delayed. It will be released if you wait
  2. CI failed midway and wasn't released
  3. CI skipped the release

These are not problems with aqua or aqua-registry. For example, if such a problem occurs with suzuki-shunsuke/pinact and you want to take action, it would be good to create an issue or PR at https://github.com/suzuki-shunsuke/pinact. As aqua-registry maintainers, we often encounter these problems. Each time, we report problems to various repositories or fix CI.

It's common for specific os/arch not to be supported. In that case, you need to exclude that os/arch from supported_envs.

If the asset name is wrong, the asset naming convention may have changed from a certain version. For example, the GoReleaser configuration was modified and the format became zip, or the version disappeared from the asset name, etc. In that case, you need to modify the asset in registry.yaml. If the name changed due to a mistake on the tool side, it would be kind to report the problem or create a PR to fix it.

When Command Cannot Be Found​

When a command cannot be found, the following possibilities exist:

  1. Command name is wrong
  2. Command name changed
  3. Path is wrong
  4. Target os/arch is excluded by supported_envs

In these cases, you need to modify the files configuration.

files:
  - name: <command name>
    src: <relative path to command executable>

By default, the command name is the last element when splitting the package name by /. So for cli/cli it becomes cli, but the actual command name is gh, so you need to explicitly specify files.

files:
  - name: gh

Note that even on Windows, .exe is not added to the name.

src is the relative path where the command executable is located when extracting assets like tarball or zip. By default, it's the same as name. For gh, since the path is different, you need to specify src.

    files:
      - name: gh
        src: gh_{{trimV .Version}}_{{.OS}}_{{.Arch}}/bin/gh

https://github.com/aquaproj/aqua-registry/blob/dc98ca0c3314ae3cface74556a295a4cb0a95918/pkgs/cli/cli/registry.yaml#L7-L9

The auto-generation tool currently cannot auto-generate files. Therefore, manual modification is necessary.

Adding Support for Specific OS / Architecture​

Sometimes a tool supports new OS/Architecture from a specific version but it's not reflected in registry.yaml and remains uninstallable. In that case, you need to add that OS/Architecture to supported_envs.

When Checksum Cannot Be Extracted from Checksum File​

Please see the document.

When Checksum Verification Fails​

Please see the document.

  1. Checksum written in checksum file is wrong => Disable checksum
checksum:
  enabled: false

Or delete the checksum configuration since it's disabled by default.

info

The checksum enable/disable setting in registry configuration is just a setting for "whether to download checksum file and get checksum". Even if this is disabled, if checksum verification is enabled in aqua.yaml, checksum verification will be performed. In that case, it actually downloads the asset, calculates the checksum, and records it in aqua-checksums.json. There is also an issue for getting checksum via GitHub API.

  1. Extracting wrong string from checksum file

Modify extraction parameters or disable checksum.

  1. Wrong checksum algorithm (sha1, sha256, sha512, md5, etc) => Fix the algorithm

When cosign Verification Fails​

Please see the document.

When SLSA Provenance Verification Fails​

Please see the document.

When GitHub Artifact Attestations Verification Fails​

Please see the document.

signer_workflow might be wrong. If attestations are not generated for a specific version in the first place, delete the github_artifact_attestations configuration.

The github_artifact_attestations configuration cannot be auto-generated currently. Therefore, when adding a new tool, check if attestations are generated and add the configuration if they are.

When Minisign Verification Fails​

Please see the document.

The minisign configuration might be wrong. If minisign signing is not performed for a specific version in the first place, delete the minisign configuration.

Tool Naming Convention​

To avoid name conflicts, tool names must include / (namespace-like meaning).

  • NG: terraform
  • OK: hashicorp/terraform

If the tool code is managed on GitHub, match the repository name. If multiple tools are managed in that repository, change the name for each tool.

e.g. winebarrel/cronplan

  • winebarrel/cronplan/cronmatch
  • winebarrel/cronplan/cronplan
  • winebarrel/cronplan/cronviz

Packages hosted outside GitHub should have naming that distinguishes them from GitHub. cargo packages become crates.io/{crate name}. Platforms other than GitHub like GitLab are not actively supported, but some are supported as http type packages. GitLab uses gitlab.com/<repository name>.

Adding New Tools​

When submitting a Pull Request to add a new tool, there's no need to create an Issue.

Run cmdx s to auto-generate code.

cmdx s "<tool name>"

e.g.

cmdx s cli/cli

For package types other than github_release, specify -l 1.

cmdx s -l 1 "<package name>"

cmdx s generates a branch feat/<package name>, code, and commit, and tests using containers.

caution

cmdx s creates a commit, but please don't edit the commit by git commit --amend, git rebase, or somehow. cmdx s creates a commit to distinguish scaffolded code from manual changes. Please add new commits if you update code.

info

This command may sometimes fail tests and output a large amount of error messages, but don't be overwhelmed by those error messages. Test failures are expected.

Customizing cmdx s with Configuration File​

note

In many cases, this is unnecessary. Also, you should not use this feature carelessly.

Sometimes cmdx s generation doesn't work in one go. For github_release packages, cmdx s gets lists of GitHub Releases and assets via GitHub API and auto-generates configuration based on that. However, sometimes you need to exclude specific versions or assets. For example, if multiple CLIs are published in the same repository, if you don't exclude assets from other CLIs, code might be generated based on asset names from other CLIs. Also, if multiple tools are published in the same repository, versions might have different prefixes for different tools. In that case, if you don't ignore versions from other tools, code likely won't be generated correctly.

In such cases, follow these steps:

  1. Generate a template configuration file aqua-generate-registry.yaml for cmdx s with aqua gr -init <package name>
  2. Modify the configuration file aqua-generate-registry.yaml
  3. Generate code with cmdx s -c "<configuration file>" "<package name>"

You can configure the following:

  • version_filter: Versions not matching this condition are excluded
  • version_prefix: Versions without this prefix are excluded
  • all_assets_filter: Assets not matching this condition are excluded

However, using this feature carelessly can exclude things that shouldn't be excluded, so it shouldn't be used lightly. all_assets_filter in particular requires caution. This is because it can accidentally exclude checksum files like SHA256SUM or checksums.txt, and it's difficult to notice if you've excluded them. Therefore, you should first generate code without exclusion settings, and if unnecessary things are mixed in the generated code, write settings that explicitly exclude only those (without making the scope too broad to avoid excluding extra things).

caution

Note that version_filter is not a feature for dropping support for old versions. version_constraint, no_asset, and error_message are used for dropping support for old versions.

https://github.com/aquaproj/aqua-registry/blob/191f2136c10b1eb962dd43c8f421af417b1b3a16/pkgs/Shopify/ejson/registry.yaml#L8-L10

Retrying cmdx s Until It Works​

note

In many cases, this is unnecessary.

As mentioned earlier, code generation with cmdx s doesn't always work on the first try. Sometimes you need to repeat it several times.

  1. Generate code without configuration file cmdx s
  2. Check the generated code, and if extra versions or assets are included, delete the generated branch

cmdx s generates a branch and commit, but if it's before opening a Pull Request, you can delete them without problems.

git checkout main
git branch -D "feat/<package name>"
  1. Generate configuration file aqua gr -init
  2. Modify configuration file and generate code cmdx s
  3. Repeat 2, 4 until extra versions and assets are excluded

Modifying Manually​

If installation of multiple versions is failing and the log is hard to read, it's good to comment out some versions in pkg.yaml and tackle problems one by one. When modifying configuration, refer to Manual Modification and Style Guide. After modification, run cmdx t to confirm it can be installed correctly. Repeat modification and confirmation until it can be installed.

When you're done with modifications, or if you're not sure how to fix it, submit a Pull Request.

note

The cmdx new command has been removed from the standard procedure. However, the command itself remains and can still be used. This command has large environment dependencies and didn't work well for some users, making troubleshooting and support difficult. Since you can create Pull Requests without using cmdx new, we decided to remove it from the standard procedure. See also changelog.

Use cmdx s definitely​

We don't accept pull requests not following this guide. Especially, we don't accept pull requests not using cmdx s. Standard Registry must support not only the latest version but also almost all versions and various platforms. Many tools have so many versions that people can't check all of them manually. So we can't trust the code not using cmdx s. cmdx s checks all GitHub Releases and generates code supporting all of them (Strictly speaking, if there are too many GitHub Releases we have to restrict the number of GitHub Releases, though cmdx s can still check over 200 versions). cmdx s generates much better code than us.

cmdx s isn't perfect and sometimes cmdx s causes errors and generates invalid code. Then you have to fix the code according to the error message. cmdx s supports only github_release type packages, so for other package types you have to fix the code. Even if so, you must still use cmdx s. cmdx s guarantees the quality of code.

💡 How to ignore some assets and versions​

You can ignore some assets and versions to scaffold better configuration files.

caution

Be careful to use this feature as it may exclude assets and versions unexpectedly. Especially, all_assets_filter may exclude assets such as checksum files. We recommend to scaffold codes without this feature first. Then if cmdx s can't scaffold good codes due to some noisy versions or assets, you should re-scaffold code using this feature. About all_assets_filter, we recommend specifying patterns to exclude assets (deny list) rather than specifying patterns to include assets (allow list).

e.g.

all_assets_filter: not (Asset contains "static")
  1. Create aqua-generate-registry.yaml by aqua gr --init command:
aqua gr --init <package name>
  1. Edit aqua-generate-registry.yaml:

Example 1. Filter assets:

name: argoproj/argo-rollouts
all_assets_filter: not ((Asset matches "rollouts-controller") or (Asset matches "rollout-controller"))

Example 2. Filter versions by version_prefix:

name: grpc/grpc-go/protoc-gen-go-grpc
version_prefix: cmd/protoc-gen-go-grpc/

Example 3. Filter versions by version_filter:

name: crate-ci/typos
version_filter: not (Version startsWith "varcon-")
  1. Run cmdx s with aqua-generate-registry.yaml
cmdx s -c aqua-generate-registry.yaml

How to execute a package in your machine during development​

There are several ways

  1. Execute a package in linux containers via cmdx con
  2. Import pkgs/<package>/pkg.yaml in aqua.yaml
  3. Add aqua-all.yaml in $AQUA_GLOBAL_CONFIG

1. Execute a package in linux containers via cmdx con​

$ cmdx con
+ bash scripts/connect.sh
[INFO] Connecting to the container aqua-registry (linux/arm64)

Then you can execute a package in the container.

2. Import pkgs/<package>/pkg.yaml in aqua.yaml​

packages:
  # ...
  - import: pkgs/<package>/pkg.yaml

Please don't commit this change.

You need to run aqua policy allow to use the local registry.

aqua policy allow

Then you can execute the package.

3. Add aqua-all.yaml in $AQUA_GLOBAL_CONFIG​

export AQUA_GLOBAL_CONFIG=$PWD/aqua-all.yaml:$AQUA_GLOBAL_CONFIG

You need to run aqua policy allow to use the local registry.

aqua policy allow

Then you can execute all packages.

Supported OS and CPU Architecture​

Please consider the following OS and CPU Architecture.

  • OS
    • windows
    • darwin
    • linux
  • CPU Architecture
    • amd64
    • arm64

We test the registry in CI on the above environments by GitHub Actions' build matrix.

Test multiple versions​

If the package has the field version_overrides, please add not only the latest version but also old versions in pkg.yaml to test if old versions can be installed properly.

e.g. pkg.yaml registry.yaml

packages:
  - name: scaleway/scaleway-cli@v2.12.0
  - name: scaleway/scaleway-cli
    version: v2.4.0

⚠️ Don't use the short syntax <package name>@<version> for the old version to prevent Renovate from updating the old version.

👎

packages:
  - name: scaleway/scaleway-cli@v2.12.0
  - name: scaleway/scaleway-cli@v2.12.0

What's pkgs/**/pkg.yaml for?​

pkgs/**/pkg.yaml are test data. pkgs/**/pkg.yaml are used to test if packages can be installed properly.

Note that pkgs/**/pkg.yaml aren't lists of available versions. You can install any versions not listed in pkgs/**/pkg.yaml.

Trouble shooting​

cmdx new fails to push a commit to the origin​

info

We removed cmdx new from the guide. You can still use cmdx new, but if you have any trouble with cmdx new, you can create a pull request without cmdx new. Please see the changelog for details.

If cmdx new can't push a commit to a remote branch, please confirm if origin is not the upstream aquaproj/aqua-registry but your fork. If origin is not your fork, please change it to your fork.

e.g. Fail to push a commit

$ cmdx new pre-commit/pre-commit
# ...
+ git push origin feat/pre-commit/pre-commit
remote: Permission to aquaproj/aqua-registry.git denied to ***.
fatal: unable to access 'https://github.com/aquaproj/aqua-registry/': The requested URL returned error: 403
  1. If you haven't forked aquaproj/aqua-registry, please fork it.
  2. Check remote repositories.
git remote -v
  1. Please fix origin.
git remote set-url origin https://github.com/<your fork>
  1. Please set upstream if necessary.
git remote add upstream https://github.com/aquaproj/aqua-registry