Skip to content

Commit 39d4a9f

Browse files
pingsutwpingsutw
authored
Fix release pipeline failures (#3423)
* Fix release pipeline failures in pythonpublish and build_image workflows The v1.16.17 release failed because a transient PyPI connection error during plugin upload killed the entire deploy job, and re-runs failed because already-uploaded packages return 400. The manual build_image workflow also failed because it passed github.sha as the package version instead of the actual semver. Changes: - Upgrade actions/setup-python from v1 to v5 and pin Python to 3.12 (3.x resolved to unstable Python 3.14) - Add --verbose --skip-existing to all twine upload commands so re-runs skip already-uploaded packages instead of failing - Fix build_image.yml to derive VERSION from branch/tag name instead of github.sha, which is not a valid PyPI version - Use safer shell parameter expansion for GITHUB_REF parsing Signed-off-by: Kevin Su <kevin@union.ai> Signed-off-by: Kevin Su <pingsutw@apache.org> * Fail early if build_image.yml is not run from a version tag/branch If triggered from a non-version ref like 'master', error out with a clear message instead of silently using an invalid PyPI version. Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <kevin@union.ai> Signed-off-by: Kevin Su <pingsutw@apache.org>
1 parent 54b82e5 commit 39d4a9f

3 files changed

Lines changed: 38 additions & 13 deletions

File tree

‎.github/workflows/build_image.yml‎

Lines changed: 29 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,28 @@ on:
44
workflow_dispatch:
55

66
jobs:
7+
get-version:
8+
runs-on: ubuntu-latest
9+
outputs:
10+
version: ${{ steps.version.outputs.version }}
11+
steps:
12+
- name: Get version from branch/tag name
13+
id: version
14+
run: |
15+
REF_NAME="${GITHUB_REF_NAME}"
16+
# If ref starts with 'v' followed by a digit (e.g. v1.16.17), strip 'v' to get the PyPI version
17+
if [[ "$REF_NAME" =~ ^v[0-9] ]]; then
18+
VERSION="${REF_NAME#v}"
19+
else
20+
echo "::error::This workflow must be run from a version tag or branch (e.g. v1.16.17), got '$REF_NAME'"
21+
exit 1
22+
fi
23+
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
24+
shell: bash
25+
726
build-and-push-docker-images-manual:
827
runs-on: ubuntu-latest
28+
needs: get-version
929
strategy:
1030
matrix:
1131
python-version: ["3.10", "3.11", "3.12"]
@@ -32,6 +52,7 @@ jobs:
3252
ghcr.io/${{ github.repository_owner }}/flytekit
3353
tags: |
3454
py${{ matrix.python-version }}-${{ github.sha }}
55+
py${{ matrix.python-version }}-${{ needs.get-version.outputs.version }}
3556
- name: Build & Push Flytekit Python${{ matrix.python-version }} Docker Image to Github Registry
3657
uses: docker/build-push-action@v2
3758
with:
@@ -40,15 +61,16 @@ jobs:
4061
push: true
4162
tags: ${{ steps.flytekit-names.outputs.tags }}
4263
build-args: |
43-
VERSION=${{ github.sha }}
44-
DOCKER_IMAGE=ghcr.io/${{ github.repository_owner }}/flytekit:py${{ matrix.python-version }}-${{ github.sha }}
64+
VERSION=${{ needs.get-version.outputs.version }}
65+
DOCKER_IMAGE=ghcr.io/${{ github.repository_owner }}/flytekit:py${{ matrix.python-version }}-${{ needs.get-version.outputs.version }}
4566
PYTHON_VERSION=${{ matrix.python-version }}
4667
file: Dockerfile
4768
cache-from: type=gha
4869
cache-to: type=gha,mode=max
4970

5071
build-and-push-flyteagent-images-manual:
5172
runs-on: ubuntu-latest
73+
needs: get-version
5274
steps:
5375
- uses: actions/checkout@v4
5476
with:
@@ -72,6 +94,7 @@ jobs:
7294
ghcr.io/${{ github.repository_owner }}/flyteagent
7395
tags: |
7496
${{ github.sha }}
97+
${{ needs.get-version.outputs.version }}
7598
- name: Push External Plugin Service Image to GitHub Registry
7699
uses: docker/build-push-action@v2
77100
with:
@@ -80,13 +103,14 @@ jobs:
80103
push: true
81104
tags: ${{ steps.flyteagent-names.outputs.tags }}
82105
build-args: |
83-
VERSION=${{ github.sha }}
106+
VERSION=${{ needs.get-version.outputs.version }}
84107
file: ./Dockerfile.connector
85108
cache-from: type=gha
86109
cache-to: type=gha,mode=max
87110

88111
build-and-push-flyteconnector-images-manual:
89112
runs-on: ubuntu-latest
113+
needs: get-version
90114
steps:
91115
- uses: actions/checkout@v4
92116
with:
@@ -110,6 +134,7 @@ jobs:
110134
ghcr.io/${{ github.repository_owner }}/flyteconnector
111135
tags: |
112136
${{ github.sha }}
137+
${{ needs.get-version.outputs.version }}
113138
- name: Push External Plugin Service Image to GitHub Registry
114139
uses: docker/build-push-action@v2
115140
with:
@@ -118,7 +143,7 @@ jobs:
118143
push: true
119144
tags: ${{ steps.flyteconnector-names.outputs.tags }}
120145
build-args: |
121-
VERSION=${{ github.sha }}
146+
VERSION=${{ needs.get-version.outputs.version }}
122147
file: ./Dockerfile.connector
123148
cache-from: type=gha
124149
cache-to: type=gha,mode=max

‎.github/workflows/pythonpublish.yml‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,9 @@ jobs:
1212
with:
1313
fetch-depth: "0"
1414
- name: Set up Python
15-
uses: actions/setup-python@v1
15+
uses: actions/setup-python@v5
1616
with:
17-
python-version: "3.x"
17+
python-version: "3.12"
1818
- name: Install dependencies
1919
run: |
2020
python -m pip install --upgrade pip
@@ -23,20 +23,20 @@ jobs:
2323
id: bump
2424
run: |
2525
# from 'refs/tags/v1.2.3' get 1.2.3
26-
VERSION=$(echo $GITHUB_REF | sed 's|refs/tags/v||')
27-
echo "version=$VERSION" >> $GITHUB_OUTPUT
26+
VERSION="${GITHUB_REF#refs/tags/v}"
27+
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
2828
shell: bash
2929
- name: Build and publish
3030
env:
3131
TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
3232
TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
3333
run: |
3434
python -m build
35-
twine upload dist/*
35+
twine upload --verbose --skip-existing dist/*
3636
- name: Autobump plugin version
37+
env:
38+
VERSION: ${{ steps.bump.outputs.version }}
3739
run: |
38-
# from 'refs/tags/v1.2.3' get 1.2.3
39-
VERSION=$(echo $GITHUB_REF | sed 's|refs/tags/v||')
4040
VERSION=$VERSION make -C plugins update_all_versions
4141
shell: bash
4242
- name: Build all Plugins and publish

‎plugins/Makefile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,10 @@ build_all_plugins:
88

99
.PHONY: publish_all_plugins
1010
publish_all_plugins:
11-
twine upload */dist/*
11+
twine upload --verbose --skip-existing */dist/*
1212
# We upload community plugins from a separate sub-directory called `community`.
1313
# Check https://github.com/flyteorg/flyte/pull/5610 for more details about.
14-
twine upload community/*/dist/*
14+
twine upload --verbose --skip-existing community/*/dist/*
1515

1616
.PHONY: all_requirements
1717
all_requirements:

0 commit comments

Comments
 (0)