GitHub Packages: Private NuGet Packages via GitHub Actions

Recently after evaluating GitHub Container Registry I also wanted to try using NuGet feed functionality within GitHub Packages to potentially consolidate feeds across sources like Azure DevOps and Proget. I decided to use GitHub Actions to build, test, pack, and push these as private NuGet packages within a GitHub organization.

Project Configuration

Before pushing to the GitHub Packages NuGet feed, NuGet properties for the projects should be reviewed and adjusted. For GitHub packages, the key property is RepositoryUrl which will be checked when pushing to the NuGet feed.

In my case there are multiple related projects in a git repo being packed and pushed together. Rather than duplicating package properties across project files, a common.props file supplies shared NuGet properties.

<Project>
    <PropertyGroup>
        <Authors>Company Contributors</Authors>
        <Product>Company Product</Product>
        <PackageProjectUrl>https://github.com/organization/repo</PackageProjectUrl>
        <PackageIconUrl />
        <RepositoryUrl>https://github.com/organization/repo</RepositoryUrl>
        <RepositoryType>git</RepositoryType>
        <IncludeSymbols>true</IncludeSymbols>
        <SymbolPackageFormat>snupkg</SymbolPackageFormat>
    </PropertyGroup>
</Project>

The projects being packed each import this file which resides in the solution root.

<Project Sdk="Microsoft.NET.Sdk">
    <Import Project="..\common.props" />
    <!-- ... -->
</Project>

Personal Access Tokens and Secrets

Before using the GitHub Package NuGet feed there’s some authentication configuration to consider.

First I created a personal access token for a GitHub action to both restore other NuGet packages from the feed as well as push the repo’s own packages. For that I switched my GitHub user to a service account using a separate Edge browser profile. The PAT needs the write:packages OAuth scope which also currently requires the read:packages and repo scopes as well (they can be unchecked but then it won’t work).

After generating the token and documenting it in a secure location, I then had to authorize the PAT with SSO since my organization uses SAML SSO. Finally I added that token to the secrets of the GitHub repo; an organization level secret may be handy here in cases instead.

Next under my personal GitHub account that I use for the organization, I created a PAT with only read:packages for use in restoring packages on my local dev machine. Sometimes it’s handy to push packages to the feed from my dev box but generally I never need that.

Pack and Push GitHub Action

Build Prep

The action starts out setting some .NET environment variables to turn the noise down some and setting its own variables to be used in later build, test, pack, and push steps.

name: 'Build, Pack, Push'
on:
  workflow_dispatch:
  push:
    branches:
    - develop
  pull_request:
    branches:
    - develop
env:
  DOTNET_CLI_TELEMETRY_OPTOUT: true
  DOTNET_NOLOGO: true
jobs:
  build-push:
    runs-on: ubuntu-latest
    steps:
      -
        name: Prep
        run: |
          echo "::set-env name=BUILD_VER::2.3.$GITHUB_RUN_NUMBER"
          echo "::set-env name=CONFIG::Release"
          echo "::set-env name=NUGET_URL::https://nuget.pkg.github.com/${{ github.repository_owner }}/index.json"
          echo "::set-env name=SLN::MyApp.sln"
          echo "::set-env name=TEST::true"
      -
        name: Checkout Dashboard
        uses: actions/checkout@v2

NuGet Source Setup

There are a few ways to setup the GitHub NuGet package source that I bounced between across different GitHub repositories.

Option 1 – Setup .NET Action

The setup-dotnet action is primarily focused on ensuring the desired .NET SDK version is installed but it can also help setup authentication to private registries like GitHub packages.

# https://github.com/actions/setup-dotnet
-
  name: Setup .NET
  uses: actions/setup-dotnet@v1
  with:
    dotnet-version: '3.1.x'
    source-url: ${{ env.NUGET_URL }}
  env:
    NUGET_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

One thing to note with the above is that ${{ secrets.GITHUB_TOKEN }} is scoped to the repository running the action. In my case I’m restoring other packages in the feed so I’d need to replace it with a token that can read packages in the registry.

I noticed the action always created the NuGet source with a name of Source and I felt I had less control using this. Also from a .NET SDK dependency perspective it isn’t strictly required as the .NET SDK is already installed on GitHub-hosted VMs.

Option 2 – Add NuGet Source Explicitly

Assuming a repo root nuget.config has the GitHub package source…

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="github" value="https://nuget.pkg.github.com/organization/index.json" protocolVersion="3" />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
  </packageSources>
</configuration>

… the source can be added dynamically with the necessary credentials:

- name: Ensure GitHub NuGet Source
  run: |
    dotnet nuget add source ${{ env.NUGET_URL }} \
      -n github \
      -u ${{ secrets.REGISTRY_USER }} \
      -p ${{ secrets.REGISTRY_TOKEN }} \
      --store-password-in-clear-text

If using this approach it might not hurt to remove the source after the packages are pushed but each GitHub actions workflow run will be executed on a fresh instance so it’s not necessary to check to see if it exists and remove it first.

-
  name: Cleanup
  if: always()
  continue-on-error: true
  run: |
    dotnet nuget remove source github

Option 3 – nuget.config Environment Variables

NuGet feed credentials should not be stored in the source controlled nuget.config file but it is possible to use environment variables.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <packageSources>
        <add key="github" value="https://nuget.pkg.github.com/organization/index.json" />
    </packageSources>
    <packageSourceCredentials>
        <github>
            <add key="Username" value="%GH_PKG_USER%" />
            <add key="ClearTextPassword" value="%GH_PKG_TOKEN%" />
        </github>
    </packageSourceCredentials>
</configuration>

That’s fine for GitHub actions where the environment variables can easily be set from GitHub secrets. For local dev machine use though it might be a hassle requiring everyone to ensure those custom environment variables get set before packages are restored.

Option 4 – Dynamically Alter nuget.config

Finally it might be desireable to dynamically create or update nuget.config when the GitHub action runs. There are a few ways to go about that but one approach is using a template – nuget.ci.config in this case:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="github" value="https://nuget.pkg.github.com/organization/index.json" protocolVersion="3" />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
  </packageSources>
  <packageSourceCredentials>
        <github>
            <add key="Username" value="${USER}" />
            <add key="ClearTextPassword" value="${PAT}" />
        </github>
    </packageSourceCredentials>
</configuration>

In the action before restoring NuGet packages or pushing any, the existing config can be replaced with a templated version that is adjusted from GitHub secrets.

-
  name: Adjust NuGet Config Credentials
  run: |
    rm -f ./nuget.config
    mv nuget.ci.config nuget.config
    echo $(sed -e "s/\${USER}/${{ secrets.REGISTRY_USER }}/" -e "s@\${PAT}@${{ secrets.PACKAGE_TOKEN }}@" nuget.config) >nuget.config

That is duplicating the NuGet config of course but generally unless there are many custom NuGet sources that wouldn’t be duplicating much.

Building and Testing

Once the NuGet source has been configured in some manner, builds and unit tests can be executed with the packages getting restored from the GitHub Packages NuGet feed in the process.

-
  name: Build
  run: |
    dotnet build \
      ${{ env.SLN }} \
      /p:VersionPrefix=${{ env.BUILD_VER }} \
      --configuration ${{ env.CONFIG }}
-
  name: Test
  if: ${{ env.TEST == 'true' }}
  run: |
    dotnet test \
      ${{ env.SLN }} \
      --no-build
      --configuration ${{ env.CONFIG }}

Packing the Projects

For the dotnet pack step in this example I’m packing several libraries with the same version number regardless of what’s changed. Typically it’d be ideal to version them separately and only pack/push if something changed for the library. In my cases these are legacy packages being moved that don’t change much and are all tightly related. There’s a loop here as dotnet pack didn’t seem to respect glob patterns (couldn’t find projects) which is a bit curious since dotnet nuget push does.

# Sadly we can't use a glob pattern with dotnet pack currently
# like Azure DevOps Pipelines NuGetCommand@2 supported.
-
  name: NuGet Pack
  run: |
    PROJECTS=./path/**/*.csproj
    shopt -s nullglob
    shopt -s globstar
    for p in $PROJECTS
    do
      echo "Packing $p"
      dotnet pack $p \
        --configuration ${{ env.CONFIG }} \
        --no-build \
        /p:Version=${{ env.BUILD_VER }}
    done

Pushing the Packages

I started with something like the below. Originally I had --api-key as well but apparently it’s ignored in this context and it only suppressed a warning.

-
  name: NuGet Push
  run: |
    dotnet nuget push **/*.nupkg \
      --source ${{ env.NUGET_URL }} \
      --skip-duplicate

The above worked, except when it didn’t :). Sporadically the push would fail with the following error.

An error occurred while sending the request.
The response ended prematurely.
PUT https://nuget.pkg.github.com/organization/

The issue Github package registry not compatible with dotnet nuget client has more details, some sort of sporadic http client issue. I’ve never seen that outside of GitHub packages though so something specific with what they’re doing on the backend I suppose.

As a workaround there’s a gpr tool that seems to resolve it. I’m not entirely sure what it’s doing differently but I suspect it’s just the retry ability it adds. It can be installed as a global tool as follows.

-
  name: GitHub Package Registry Tool Install
  run: |
    dotnet tool install --global --verbosity minimal --no-cache gpr

Afterwards the nuget push appears reliable.

-
  name: NuGet Push with GPR
  run: |
    gpr push \
      --api-key ${{ secrets.PACKAGE_TOKEN }} \
      --repository ${{ github.repository }} \
      --retries 3 \
      **/*.nupkg

After the GitHub action triggers, the packages can be verified at the organization level (https://github.com/orgs/organization/packages). They’re also linked from the repo level, although currently the count that’s shown and the list are inaccurate for me in cases.

Other Potential NuGet Push Issues

Create Package Version Permission

In at least one repository when pushing a NuGet package from an action I ran into the error: *** does not have the correct permissions to execute CreatePackageVersion. In this case the user the token was created under wasn’t a member of the repository but it could also happen with an insufficient role.

Duplicate Package / Symbols

Another potential error is:

[Some.Library.2.3.25.nupkg]: Error: Version 2.3.25 of "Some.Library" has already been pushed.

This may happen when <IncludeSymbols>true</IncludeSymbols> is set in packed project files or specified when packing. The default is .symbols.nupkg which may result in the same package getting pushed. While dotnet nuget push can skip duplicates, GPR cannot currently. One workaround is setting <SymbolPackageFormat>snupkg</SymbolPackageFormat> in the project file.

I’ve not fully tried to verify symbol packages with GitHub Packages. I see symbol packages being created but no evidence they are being pushed correctly. Implicitly they don’t appear to be pushed and explicitly pushing **/*.snupkg results in duplicate package errors. Presumably it’s supported and it’s something simple I’m missing but I’ve not had the need or time yet to investigate.

Dev Machine NuGet Source Setup

I created the following script for setting up the GitHub NuGet feed source on local dev machines for restores, installs etc.

#!/bin/sh
# usage:
#   export MY_PAT=secret
#   echo "$MY_PAT" | ./nuget-github-source.sh -u username
#
# dotnet nuget remove source github
# dotnet nuget list source
NUGET_URL=https://nuget.pkg.github.com/organization/index.json
user=""
usage() {
  cat << EOF >&2
Usage:
    export MY_PAT=secret
    echo "\$MY_PAT" | ./nuget-github-source.sh -u <username>
    <stdin>: GitHub Personal Access Token from https://github.com/settings/tokens
    -u <username>: Github user to authenticate with
EOF
  exit 1
}
while getopts u: o; do
  case $o in
    (u) user=$OPTARG;;
    (*) usage
  esac
done
shift "$((OPTIND - 1))"
read -r -t 2 -d $'\0' token
if [ -z "$token" ]; then
  echo "GitHub PAT is required (pass via stdin)" >&2
  usage
fi
if [ -z "$user" ]; then
  echo "GitHub user is required (-u)" >&2
  usage
fi
dotnet nuget add source $NUGET_URL \
    -n github \
    -u $user \
    -p $token \
    --store-password-in-clear-text
dotnet nuget list source
echo "View config: cat ~/.nuget/NuGet/NuGet.Config"

Thoughts and Going Further

With the exception of sporadic https issues pushing packages and package counts being incorrect in places, the GitHub NuGet experience was mostly positive. I like having the packages within the GitHub org and linked to the repo. Browsing package lists, details, and versions I found intuitive and friendly as well.

One issue I have with GitHub packages in general currently is that there are no retention policies for package versions. There are actions like Delete Package Versions as well as gpr batch package delete functionality where this can be automated somewhat. Still in my opinion this should be more declarative settings within a GitHub repo and organization for handling cleanup automatically based on downloads, date, number of versions and other relevant stats.