optimize tests #49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Scartography merged 27 commits into main from pytest-xdist

Feb 5, 2026

.github/workflows/python-package.yml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -3,64 +3,75 @@ name: Python package test
  
    on:

      push:

        branches: [main]

        paths-ignore:

          - "uv.lock"

          - "conda/meta.yaml"

          - "docs/**"

      pull_request:

        branches: [main]

        paths-ignore:

          - "uv.lock"

          - "conda/meta.yaml"     

      schedule:

        - cron: '0 8 * * 1,4'

      workflow_call:

    permissions:

      pull-requests: write

      contents: read

    jobs:

      setup:

          name: "🏗️ Build Py${{ matrix.python-version }}"

          runs-on: ubuntu-latest

          strategy:

            matrix:

              python-version: ["3.10", "3.11", "3.12", "3.13"]

          container: ghcr.io/osgeo/gdal:ubuntu-small-latest

          steps:

            - uses: actions/checkout@v4

            - name: ⚡ Install uv

              run: |

                apt-get update && apt-get install -y curl

                curl -LsSf https://astral.sh/uv/install.sh | BINDIR=/usr/local/bin sh

            - name: 📝 Export Locked Requirements

              run: |

                uv export --frozen --all-extras --format requirements-txt > requirements.txt

            - name: 📤 Upload Requirements

              uses: actions/upload-artifact@v4

              with:

                name: frozen-reqs-${{ matrix.python-version }}

                path: requirements.txt

                retention-days: 1

      test:

        runs-on: ${{ matrix.os }}

        needs: setup

        name: "🧪 py${{ matrix.python-version }} [${{ matrix.split }}/4]"

        runs-on: ubuntu-latest

        strategy:

          fail-fast: false

          matrix:

            python-version: ["3.10", "3.11", "3.12", "3.13"]

            os: ["ubuntu-latest"]

            split: [1, 2, 3, 4]

        container: ghcr.io/osgeo/gdal:ubuntu-small-latest

        steps:

          - uses: actions/checkout@v4

          - name: Download updated dependencies

          - name: 📥 Download Requirements

            uses: actions/download-artifact@v4

            continue-on-error: true

            with:

              name: updated-deps

          - name: Set up Python ${{ matrix.python-version }}

            uses: actions/setup-python@v5

            with:

              python-version: ${{ matrix.python-version }}

          - name: Lint

              name: frozen-reqs-${{ matrix.python-version }}

          - name: ⚡ Install uv

            run: |

              python -m pip install pre-commit

              pre-commit run --show-diff-on-failure --all-files

              apt-get update && apt-get install -y curl git gpg

              curl -LsSf https://astral.sh/uv/install.sh | BINDIR=/usr/local/bin sh

          - name: Install GIS Dependencies

          - name: 🏗️ Restore Native Env

            run: |

              sudo apt-add-repository -y ppa:ubuntugis/ubuntugis-unstable

              sudo apt-get -y update

              sudo apt-get install -y gdal-bin libgdal-dev libproj-dev libgeos-dev g++

              uv venv --python ${{ matrix.python-version }}

              uv pip install -r requirements.txt

              uv pip install -e ".[test]" --no-deps

          - name: Install uv

            uses: astral-sh/setup-uv@v5

            with:

              enable-cache: true

          - name: Install dependencies

            env:

              CURL_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt

              GDAL_CONFIG: /usr/bin/gdal-config

            run: |

              # uv sync will prefer the downloaded uv.lock if it exists

              uv sync --all-extras

          - name: run 

          - name: 🧪 Run Split

            env:

              AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}

              AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

    @@ -69,10 +80,56 @@ jobs:
  
              CDSE_S3_ACCESS_KEY: ${{ secrets.CDSE_S3_ACCESS_KEY }}

              CDSE_S3_ACCESS_SECRET: ${{ secrets.CDSE_S3_ACCESS_SECRET }}

              CURL_CA_BUNDLE: /etc/ssl/certs/ca-certificates.crt

            run: uv run pytest -v --cov mapchete_eo --cov-report xml:coverage.xml --cov-report=term-missing:skip-covered --junitxml=pytest.xml

            run: |

                uv run pytest -v \

                --splits 4 \

                --group ${{ matrix.split }} \

                --cov=mapchete_eo \

                --cov-report=xml:coverage.xml \

                --junitxml="pytest-${{ matrix.python-version }}-${{ matrix.split }}.xml"

          - name: Upload coverage reports to Codecov

          - name: 📊 Upload to Codecov

            uses: codecov/codecov-action@v5

            with:

              token: ${{ secrets.CODECOV_TOKEN }}

              slug: mapchete/mapchete-eo
      
              file: ./coverage.xml

              flags: py${{ matrix.python-version }}-split${{ matrix.split }}

              fail_ci_if_error: true

          - name: 📤 Archive JUnit Results

            if: always()

            uses: actions/upload-artifact@v4

            with:

              name: results-${{ matrix.python-version }}-${{ matrix.split }}

              path: pytest-${{ matrix.python-version }}-${{ matrix.split }}.xml

              retention-days: 1

      summary:

        name: "📊 Final Summary"

        needs: test

        if: always()

        runs-on: ubuntu-latest

        steps:

          - name: 📥 Download All Results

            uses: actions/download-artifact@v4

            with:

              path: all-results

              pattern: results-*

              merge-multiple: true

          - name: 📊 Results of the Test Suite

            run: |

              echo "### 🧪 Test Results Summary" >> $GITHUB_STEP_SUMMARY

              # Count XML files

              COUNT=$(ls all-results/*.xml 2>/dev/null | wc -l || echo "0")

              if [[ "${{ needs.test.result }}" == "success" ]]; then

                echo "✅ **All tests passed!**" >> $GITHUB_STEP_SUMMARY

                echo "Successfully collected and verified $COUNT test report files." >> $GITHUB_STEP_SUMMARY

                exit 0

              else

                echo "❌ **Some test splits failed.**" >> $GITHUB_STEP_SUMMARY

                echo "Review the 'test' matrix for specific logs. Collected $COUNT reports before failure." >> $GITHUB_STEP_SUMMARY

                echo "Check the 'Artifacts' section below to download individual JUnit XML reports." >> $GITHUB_STEP_SUMMARY

                exit 1

              fi

.github/workflows/sync-dependencies.yml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -3,41 +3,41 @@ name: sync-dependencies
  
    on:

      schedule:

        - cron: '0 0 * * *'

      pull_request:

        paths:

          - "pyproject.toml"

      workflow_dispatch:

      push:

        branches: [main]

        paths:

          - "pyproject.toml" # ⬅️ Only trigger when the "source" of deps changes

    jobs:

      sync-and-prepare:

        runs-on: ubuntu-latest

        outputs:

          changed: ${{ steps.check.outputs.changed }}

          sync_needed: ${{ steps.check.outputs.changed }}

        steps:

          - uses: actions/checkout@v4

            with:

              fetch-depth: 0

              ref: ${{ github.event.pull_request.head.ref || github.ref }}

          - name: Install uv

          - name: ⚡ Setup uv

            uses: astral-sh/setup-uv@v5

          - name: Sync and Upgrade

          - name: 🛠️ Upgrade & Sync

            run: |

              uv lock || (rm uv.lock && uv lock)

              uv lock --upgrade

          - name: Sync Conda Recipe

            run: |

              uvx pyproject2conda yaml -f pyproject.toml --output conda/meta.yaml --python-include infer

          - name: Check for changes

          - name: 📋 Check for Changes

            id: check

            run: |

              if [ -n "$(git status --porcelain)" ]; then

                echo "changed=true" >> $GITHUB_OUTPUT

              fi

              else

                echo "changed=false" >> $GITHUB_OUTPUT

              fi                

          - name: Upload artifacts for testing

          - name: 📤 Upload Updated Deps

            if: steps.check.outputs.changed == 'true'

            uses: actions/upload-artifact@v4

            with:

    @@ -46,48 +46,41 @@ jobs:
  
                uv.lock

                conda/meta.yaml

      # This job runs the full 16-matrix test suite using the new lockfile

      run-tests:

        needs: sync-and-prepare

        if: needs.sync-and-prepare.outputs.changed == 'true'

        if: needs.sync-and-prepare.outputs.sync_needed == 'true'

        uses: ./.github/workflows/python-package.yml

        secrets: inherit

      finalize:

        needs: [sync-and-prepare, run-tests]

        if: success()

        if: |

          always() && 

          needs.run-tests.result == 'success' &&

          needs.sync-and-prepare.outputs.sync_needed == 'true'

        runs-on: ubuntu-latest

        permissions:

          contents: write

          pull-requests: write

        steps:

          - uses: actions/checkout@v4

            with:

              fetch-depth: 0

              ref: ${{ github.event.pull_request.head.ref || github.ref }}

          - name: Download updated files

          - name: 🛠️ Git Rebase (Clean Tree)

            run: |

              git config user.name "github-actions[bot]"

              git config user.email "github-actions[bot]@users.noreply.github.com"

              git pull --rebase origin ${{ github.event.pull_request.head.ref || github.ref_name }}

          - name: 📥 Apply Verified Artifacts

            uses: actions/download-artifact@v4

            with:

              name: updated-deps

          - name: Push to Existing PR

            if: github.event_name == 'pull_request'

          - name: 🚀 Commit and Push

            run: |

              git config user.name "github-actions[bot]"

              git config user.email "github-actions[bot]@users.noreply.github.com"

              git add uv.lock conda/meta.yaml

              git commit -m "chore: sync uv.lock and conda recipe (tests passed)" || echo "No changes"

              git push origin HEAD:${{ github.event.pull_request.head.ref }}

          - name: Create New Pull Request

            if: github.event_name != 'pull_request'

            uses: peter-evans/create-pull-request@v7

            with:

              token: ${{ secrets.GITHUB_TOKEN }}

              commit-message: "chore: update uv.lock and conda recipe"

              title: "chore(deps): sync uv and conda"

              body: "Automated daily sync."

              branch: "automated-dependency-sync"

              delete-branch: true

              assignees: Scartography

              reviewers: ungarj

              labels: dependencies, automated-pr
      
              git commit -m "chore: automated dependency sync (verified)"

              git push origin HEAD:${{ github.event.pull_request.head.ref || github.ref_name }}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

optimize tests #49

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!