Continuous Alignment Testing (CAT) Framework

A framework for implementing Continuous Alignment Testing (CAT) for LLM-based applications.

Overview

CAT provides the infrastructure needed to:

• Run and track CATs against LLM outputs
• Store and analyze test results over time
• Monitor changes in LLM behavior as prompts/models/data evolve
• Integrate validation into CI/CD pipelines

Example Apps

• team_recommender - Team Recommender
• pythonCatBot - An OpenAI powered Python console chat app

Run Tests

poetry run pytest

Code Quality

poetry run mypy -p src

Publishing Documentation

The Sphinx based documentation is available at https://thisisartium.github.io/continuous-alignment-testing

Wiki

• Archive - To see potentially helpful previous iterations of the code