Skip to content

Run P.808 family of tests on Prolific platform + agent specification#96

Open
babaknaderi wants to merge 29 commits into
masterfrom
babaknaderi/prolific
Open

Run P.808 family of tests on Prolific platform + agent specification#96
babaknaderi wants to merge 29 commits into
masterfrom
babaknaderi/prolific

Conversation

@babaknaderi
Copy link
Copy Markdown
Contributor

@babaknaderi babaknaderi commented May 22, 2026

  • Multiple fixes and updates.
  • Support Prolific Crowdsourcing platform
  • Added Agent support for creating and parsing results.

babaknaderi and others added 28 commits October 7, 2025 13:29
@babaknaderi
update P804 to runon hitapp server
bd228c9
@babaknaderi
fixes on p.804, updated personalized p835 and ACR
9f612e4
@babaknaderi
update for result_parser and utility script to accept/reject hits
2cb01ba
@babaknaderi
add rdp check to p.804
820d332
@babaknaderi
basic support and documentation. Also updates on creating local previ…
86352aa 
…ew for debugging
@babaknaderi
applied changed for prolific support into other templates
cfbf336
@babaknaderi
update documentation
ba37929
@babaknaderi
added script to create gold questions, updated trapping clip script, …
4785d23 
…and added agent instruction to create study.
@babaknaderi
utility to upload files
965239c
@babaknaderi
Update study creation instruction and add utility scripts
4ff4ef4 
- Rewrite .github/create.instruction.md for the P.808 repo with correct
  paths, column names, method mappings, and end-to-end automation steps
- Add src/utils/download_clips.py to download clips from CSV URLs locally
- Add src/utils/select_training_clips.py to select training clips from
  rating clips
- Update AGENTS.md with task-specific instruction file references for
  auto-discovery by agents
- Fix librosa API compatibility in create_trapping_stimuli.py
  (get_duration path= and resample orig_sr=/target_sr= kwargs)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babaknaderi
Add direct upload mode to copy_to_pub_storage.py
e639356 
- Add 'upload' mode that uses 'az storage blob upload-batch' with
  az login credentials for zero-manual-step uploads
- Stages only needed files (not CSVs/txt) via temp directory
- Falls back to 'upload-local' (azcopy command) when az CLI unavailable
- Update create.instruction.md to use direct upload as the default
- Existing upload-local and copy-remote modes are preserved

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babaknaderi
Add AI agent usage section to README.md
fb1c11b 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babaknaderi
Update create.instruction.md: gold/trapping source clip guidance and …
efb2bde 
…P.804 ver assignment

- Split source clip note into gold-specific (requires clean reference audio)
  and trapping-specific (quality does not matter, agent can download directly)
- P.804: replace two independent gold sets with single set approach,
  assigning ver=1 for ovrl_ans=5 and ver=2 for ovrl_ans=1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babaknaderi
Improve agent discoverability and instruction file structure
1acf987 
- Move language.instructions.md to .github/instructions/ for proper CLI discovery
- Add Custom Agents section to copilot-instructions.md
- Add create-study agent definition in .github/agents/
- Update AGENTS.md and README.md

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babaknaderi
Add CLAUDE.md for agent discovery in Claude Code
e1c6c1f 
Points Claude Code users to coding standards and the create-study
agent runbook so the agent is discoverable across both Copilot CLI
and Claude Code.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babaknaderi
Add cross-platform adaptation directive to create-study agent
ce790ab 
Instructs any AI agent to adapt PowerShell commands and Windows paths
to the user's OS and shell, making the runbook usable from Claude Code
on macOS/Linux as well as Copilot CLI on Windows.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babaknaderi
Replace hardcoded repo paths with REPO_ROOT placeholder
13097ec 
All instances of C:\my\repos\P.808 are now REPO_ROOT, which the
agent resolves to the actual Git root at runtime. This makes the
runbook portable across clones, machines, and operating systems.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babaknaderi
Remove redundant quick-reference and known-issues sections
3917211 
The quick-reference was a condensed copy of the main workflow (sections
3-10). All six known issues were already documented inline in their
respective workflow steps. Removing both eliminates maintenance risk
from duplicated information.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babaknaderi
Address agent workflow feedback: dir rename, CSV fix, instruction upd…
9036122 
…ates

- Rename 'src/trapping clips/' to 'src/trapping_clips_assets/' (avoid spaces)
- Keep sample source WAVs (s1-s12.wav) in new location
- Update all config files and docs to reference new directory name
- Add .gitkeep to output/ subdirectory
- Fix CSV double line endings in select_training_clips.py
- Update agent instructions (create-study.agent.md):
  - Add PowerShell execution policy bypass guidance
  - Add training_gold_clips concept for P.804 and pp835
  - Ask for manual training clip selection first, fall back to auto
  - Add [ASK] review step after generating gold/trapping/training clips
  - Make --check_urls and --create_local_test mandatory
  - Use random opaque dir names for rating clip uploads
  - Fix cleanup/review messaging after file deletion
  - Update all trapping directory references

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babaknaderi
Update agent runbook and add SAS token support to download_clips.py
be71557 
- Add --sas_token parameter to download_clips.py for private storage
- Clarify training_gold_clips priority over training_clips for P.804/pp835
- Add URL mapping CSV generation step (step 9b)
- Document check_urls_in_files_exist for validation with SAS token support
- Expand azcopy guidance: prefer SAS token over azcopy login
- Condense agent file to fit within 30,000 char limit

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babaknaderi
Update agent SAS token handling and P.804 template
e7f2d7e 
- Agent: generate SAS tokens with az CLI instead of asking user for them
- Agent: use SAS tokens on both source and destination for azcopy transfers
  (never rely on azcopy login which expires after 90 days)
- Agent: add note about azcopy --recursive preserving directory structure
- P.804 template: remove outdated information

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babaknaderi
Add CCR trapping clips and update TTS script to use Azure AD auth
e53ff6b
@babaknaderi
Add CCR trapping support and update agent workflow
8a2b8bf 
- Add CCR score mappings (-3 to 0) to create_trapping_stimuli.py
- Add trapping_ccr.cfg with adj_ccr_score_ message prefix
- Update agent doc: CCR trapping config in method tables
- Add reuse-confirmation workflow for multi-study sessions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babaknaderi
Clarify DCR/CCR trapping as legacy gold/control question
1c40598 
DCR and CCR do not use traditional trapping clips with overlaid
spoken scores. The TP field is a reference clip used as a gold/control
question (legacy naming). Update agent doc to skip
create_trapping_stimuli.py for these methods and use reference clips
directly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babaknaderi
Add analyze-results agent for result parsing workflow
6692b1b 
New agent runbook for processing crowdsourced subjective test results:
- Runs result_parser.py with config, answers, and optional Prolific data
- Validates inputs, checks rejection rates (>35% alarming)
- Analyzes gold question performance (>20% rejection alarming)
- Provides structured summary and points to output files
- Documents scale suffixes per method and output file patterns

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babaknaderi
Clarify comments on trapping vs gold questions for CCR/DCR
e1b87bc 
Updated docstring in check_tps() and inline comment in data_cleaning()
to note that CCR and DCR use gold questions coded as trapping questions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babaknaderi
Fix spelling, grammar, and punctuation in HTML templates
10a859e 
Corrected user-facing text and accessibility attributes
across all 15 HTML templates.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babaknaderi
Update requirements.txt: remove stdlib, add missing deps, unpin versions
28ac1f7 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@babaknaderi
Copy link
Copy Markdown
Contributor Author

babaknaderi commented May 22, 2026

@copilot resolve the merge conflicts in this pull request

@babaknaderi
Merge origin/master into babaknaderi/prolific, resolve conflicts
0d64953 
Co-authored-by: babaknaderi <15108406+babaknaderi@users.noreply.github.com>
Copy link
Copy Markdown

Copilot AI commented May 22, 2026

@copilot resolve the merge conflicts in this pull request

Merge conflicts resolved in commit 0d64953. Here's a summary of how each conflict was handled:

  • docs/conf_master.md: Kept the PR's grammar fix ("Defines the quantity bonus requirement.")
  • docs/prep_p804.md: Kept master's trailing whitespace cleanup + the PR's "values" (correct plural) and extra create_gold_clips.py content
  • docs/prep_p835.md: Combined both — master's "do not" (correct plural agreement) + the PR's corrections ("proposed" and "based on")
  • src/P808Template/README.md: Kept master's version (no trailing space)
  • src/P808Template/P808_multi.html: Kept the PR's id="gold_warning_checkbox" attribute (required by the existing JS event handler) and fixed the typo value="warining" → value="warning"
  • src/configurations/master.cfg: Kept master's new configuration sections (CommonAccountKeys, DefaultStorage, GoldenSample, etc.)
  • src/requirements.txt: Kept the PR's unpinned/expanded requirements
  • src/result_parser.py: Kept the PR's logger-based logging and simpler save_approved_ones function signature

@babaknaderi babaknaderi requested a review from Copilot May 22, 2026 04:46
@babaknaderi babaknaderi requested a review from andosa May 22, 2026 04:47
Copilot AI reviewed May 22, 2026
View reviewed changes
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@babaknaderi babaknaderi requested a review from Copilot May 22, 2026 06:17
Copilot AI reviewed May 22, 2026
View reviewed changes
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@babaknaderi babaknaderi requested a review from Copilot May 22, 2026 07:08
Copilot AI reviewed May 22, 2026
View reviewed changes
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@andosa andosa Awaiting requested review from andosa

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@babaknaderi