Skip to content

Commit 81a7aa7

Browse files
Copilotkobenguyent
Copilot
and
kobenguyent
committed
Add GitHub workflows demonstrating test sharding functionality
Co-authored-by: kobenguyent <7845001+kobenguyent@users.noreply.github.com>
1 parent f561d29 commit 81a7aa7

4 files changed

Lines changed: 223 additions & 2 deletions

File tree

‎.github/SHARDING_WORKFLOWS.md‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Test Sharding Workflows
2+
3+
This document explains the GitHub Actions workflows that demonstrate the new test sharding functionality in CodeceptJS.
4+
5+
## Updated/Created Workflows
6+
7+
### 1. `acceptance-tests.yml` (Updated)
8+
9+
**Purpose**: Demonstrates sharding with acceptance tests across multiple browser configurations.
10+
11+
**Key Features**:
12+
- Runs traditional docker-compose tests (for backward compatibility)
13+
- Adds new sharded acceptance tests using CodeceptJS directly
14+
- Tests across multiple browser configurations (Puppeteer, Playwright)
15+
- Uses 2x2 matrix: 2 configs × 2 shards = 4 parallel jobs
16+
17+
**Example Output**:
18+
```
19+
- Sharded Tests: codecept.Puppeteer.js (Shard 1/2)
20+
- Sharded Tests: codecept.Puppeteer.js (Shard 2/2)
21+
- Sharded Tests: codecept.Playwright.js (Shard 1/2)
22+
- Sharded Tests: codecept.Playwright.js (Shard 2/2)
23+
```
24+
25+
### 2. `sharding-demo.yml` (New)
26+
27+
**Purpose**: Comprehensive demonstration of sharding features with larger test suite.
28+
29+
**Key Features**:
30+
- Uses sandbox tests (38 test files) for meaningful sharding demonstration
31+
- Shows basic sharding with 4-way split (`1/4`, `2/4`, `3/4`, `4/4`)
32+
- Demonstrates combination of `--shuffle` + `--shard` options
33+
- Comprehensive documentation and examples
34+
35+
### 3. `test.yml` (Updated)
36+
37+
**Purpose**: Clarifies which tests support sharding.
38+
39+
**Changes**:
40+
- Added comment explaining that runner tests are mocha-based and don't support sharding
41+
- Points to sharding-demo.yml for examples of CodeceptJS-based sharding
42+
43+
## Sharding Commands Used
44+
45+
### Basic Sharding
46+
```bash
47+
npx codeceptjs run --config ./codecept.js --shard 1/4
48+
npx codeceptjs run --config ./codecept.js --shard 2/4
49+
# etc.
50+
```
51+
52+
### Combined with Other Options
53+
```bash
54+
npx codeceptjs run --config ./codecept.js --shuffle --shard 1/2 --verbose
55+
```
56+
57+
## Test Distribution
58+
59+
The sharding algorithm distributes tests evenly:
60+
61+
- **38 tests across 4 shards**: ~9-10 tests per shard
62+
- **6 acceptance tests across 2 shards**: 3 tests per shard
63+
- **Uneven splits handled gracefully**: Earlier shards get extra tests when needed
64+
65+
## Benefits Demonstrated
66+
67+
1. **Parallel Execution**: Tests run simultaneously across multiple CI workers
68+
2. **No Manual Configuration**: Automatic test distribution without maintaining test lists
69+
3. **Load Balancing**: Even distribution ensures balanced execution times
70+
4. **Flexibility**: Works with any number of shards and test configurations
71+
5. **Integration**: Compatible with existing CodeceptJS features (`--shuffle`, `--verbose`, etc.)
72+
73+
## CI Matrix Integration
74+
75+
The workflows show practical CI matrix usage:
76+
77+
```yaml
78+
strategy:
79+
matrix:
80+
config: ['codecept.Puppeteer.js', 'codecept.Playwright.js']
81+
shard: ['1/2', '2/2']
82+
```
83+
84+
This creates 4 parallel jobs:
85+
- Config A, Shard 1/2
86+
- Config A, Shard 2/2
87+
- Config B, Shard 1/2
88+
- Config B, Shard 2/2
89+
90+
Perfect for scaling test execution across multiple machines and configurations.

‎.github/workflows/acceptance-tests.yml‎

Lines changed: 42 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Acceptance Tests using docker compose
1+
name: Acceptance Tests with Sharding
22

33
on:
44
push:
@@ -14,8 +14,10 @@ env:
1414
FORCE_COLOR: 1
1515

1616
jobs:
17-
build:
17+
# Traditional docker-compose tests (kept for compatibility)
18+
docker-tests:
1819
runs-on: ubuntu-latest
20+
name: Docker-based Tests
1921

2022
strategy:
2123
matrix:
@@ -46,3 +48,41 @@ jobs:
4648
- name: Run Faker BDD Tests
4749
run: docker-compose run --rm test-bdd.faker
4850
working-directory: test
51+
52+
# New sharded acceptance tests using CodeceptJS directly
53+
sharded-acceptance-tests:
54+
runs-on: ubuntu-latest
55+
name: "Sharded Tests: ${{ matrix.config }} (Shard ${{ matrix.shard }})"
56+
57+
strategy:
58+
fail-fast: false
59+
matrix:
60+
node-version: [20.x]
61+
config: ['codecept.Puppeteer.js', 'codecept.Playwright.js']
62+
shard: ['1/2', '2/2']
63+
64+
steps:
65+
- name: Checkout Repository
66+
uses: actions/checkout@v5
67+
68+
- name: Use Node.js ${{ matrix.node-version }}
69+
uses: actions/setup-node@v4
70+
with:
71+
node-version: ${{ matrix.node-version }}
72+
73+
- name: Install dependencies
74+
run: npm install --ignore-scripts
75+
env:
76+
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: true
77+
PUPPETEER_SKIP_CHROMIUM_DOWNLOAD: true
78+
79+
- name: Run Acceptance Tests with Sharding
80+
run: npx codeceptjs run --config ./test/acceptance/${{ matrix.config }} --shard ${{ matrix.shard }} --verbose
81+
env:
82+
FORCE_COLOR: 1
83+
84+
- name: Display Test Info
85+
run: |
86+
echo "Configuration: ${{ matrix.config }}"
87+
echo "Shard: ${{ matrix.shard }}"
88+
echo "This demonstrates test sharding across different browser configurations"

‎.github/workflows/sharding-demo.yml‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
name: Test Sharding Demonstration
2+
3+
on:
4+
push:
5+
branches:
6+
- '3.x'
7+
pull_request:
8+
branches:
9+
- '**'
10+
11+
env:
12+
CI: true
13+
FORCE_COLOR: 1
14+
15+
jobs:
16+
# Demonstrate sharding with a larger test suite (sandbox tests)
17+
sharded-sandbox-tests:
18+
runs-on: ubuntu-latest
19+
name: "Sandbox Tests (Shard ${{ matrix.shard }})"
20+
21+
strategy:
22+
fail-fast: false
23+
matrix:
24+
node-version: [20.x]
25+
# Split 38 sandbox tests across 4 shards for demonstration
26+
shard: ['1/4', '2/4', '3/4', '4/4']
27+
28+
steps:
29+
- name: Checkout Repository
30+
uses: actions/checkout@v5
31+
32+
- name: Use Node.js ${{ matrix.node-version }}
33+
uses: actions/setup-node@v4
34+
with:
35+
node-version: ${{ matrix.node-version }}
36+
37+
- name: Install dependencies
38+
run: npm install
39+
env:
40+
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: true
41+
PUPPETEER_SKIP_CHROMIUM_DOWNLOAD: true
42+
43+
- name: Run Sandbox Tests with Sharding
44+
run: npx codeceptjs run --config ./test/data/sandbox/codecept.js --shard ${{ matrix.shard }} --verbose
45+
working-directory: test/data/sandbox
46+
env:
47+
FORCE_COLOR: 1
48+
49+
- name: Display Shard Info
50+
run: |
51+
echo "This shard (${{ matrix.shard }}) ran a subset of the total sandbox tests"
52+
echo "All shards together cover the complete test suite without duplication"
53+
54+
# Show combination with shuffle option
55+
sharded-shuffled-tests:
56+
runs-on: ubuntu-latest
57+
name: "Shuffled + Sharded Tests (Shard ${{ matrix.shard }})"
58+
59+
strategy:
60+
fail-fast: false
61+
matrix:
62+
node-version: [20.x]
63+
shard: ['1/2', '2/2']
64+
65+
steps:
66+
- name: Checkout Repository
67+
uses: actions/checkout@v5
68+
69+
- name: Use Node.js ${{ matrix.node-version }}
70+
uses: actions/setup-node@v4
71+
with:
72+
node-version: ${{ matrix.node-version }}
73+
74+
- name: Install dependencies
75+
run: npm install
76+
env:
77+
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: true
78+
PUPPETEER_SKIP_CHROMIUM_DOWNLOAD: true
79+
80+
- name: Run Tests with Shuffle + Sharding
81+
run: npx codeceptjs run --config ./test/data/sandbox/codecept.js --shuffle --shard ${{ matrix.shard }} --verbose
82+
working-directory: test/data/sandbox
83+
env:
84+
FORCE_COLOR: 1
85+
86+
- name: Display Combined Options Info
87+
run: |
88+
echo "This demonstrates sharding combined with shuffle option"
89+
echo "Tests are first shuffled, then sharded for this worker (${{ matrix.shard }})"

‎.github/workflows/test.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,3 +48,5 @@ jobs:
4848
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: true
4949
PUPPETEER_SKIP_CHROMIUM_DOWNLOAD: true
5050
- run: npm run test:runner
51+
# Note: Runner tests are mocha-based, so sharding doesn't apply here.
52+
# Sharding is specifically for CodeceptJS tests (see sharding-demo.yml for examples)

0 commit comments

Comments
 (0)