AVATAR-PIPELINE: AI-Driven Interaction Engine

This repository contains the Avatar-Pipeline, an advanced interactive prototype developed for the NexEra AI Engineer assessment (Test 2). It successfully bridges the gap between unstructured human natural language and real-time 3D WebGL skeletal animation.

Technical Deep Dive: For a comprehensive explanation of the architecture, AI logic, engineering challenges, and scaling plans, please see backend/Architecture.md.

🚀 Project Overview & Assessment Delivery

Avatar-Pipeline acts as the foundational engine for an intelligent AI learning coach. The system solves a critical human-computer interaction challenge: translating ambiguous text commands into rigid, executable 3D animation states without crashing the graphics engine.

What the user does:

• Learner types an unstructured command (e.g., "Show me how to inspect the equipment") or clicks a predefined Quick Command hotkey.
• Uses mouse to orbit and zoom the 3D camera.

What the system does:

1. AI Interpretation: Groq's llama-3.3-70b-versatile interprets the text contextually.
2. Deterministic Mapping: The LLM maps the intent directly to one of 14 strictly validated 3D skeletal tracks (e.g., "Yes", "No", "ThumbsUp", "Dance").
3. Execution: The React Three Fiber frontend crossfades the WebGL character into the new animation seamlessly.
4. Contextualization: The UI displays a teleprompter with an educational rationale generated by the AI, explaining why it chose that action.
5. Auto-Reset: A background useRef timer automatically returns the avatar to an "Idle" breathing state after 5 seconds of inactivity.

🏗️ Tech Stack

Frontend (Avatar-UI)

• Framework: Next.js 16.2 (App Router, Turbopack, React 19) + TypeScript
• 3D Engine: React Three Fiber (@react-three/fiber), @react-three/drei, Three.js
• Assets: Official Three.js RobotExpressive.glb (14 baked animations)
• Styling: Tailwind CSS v4, Lucide React (Icons)

Backend (Avatar-Director)

• Framework: FastAPI (Asynchronous Python), Pydantic v2
• AI Engine: Groq API enforcing strict JSON output mode.
• Containerization: Docker (Python 3.14-slim base image)

⚙️ Setup & Execution Instructions

To maintain a strict Separation of Concerns, the API and the UI must be run concurrently.

1. The Backend (Avatar-Director)

1. Navigate to the backend directory: cd backend
2. Create a .env file and add your AI key:

   GROQ_API_KEY=your_key_here  

3. Run via Docker (Recommended):

   docker build -t avatar-backend .
   docker run -p 7860:7860 -v "${PWD}:/app" --name avatar-director avatar-backend
   

1. Manual Run (Alternative):

python -m venv venv
# Activate: .\venv\Scripts\activate (Win) or source venv/bin/activate (Mac/Linux)
pip install -r requirements.txt
uvicorn app.main:app --host 0.0.0.0 --port 7860 --reload

Health Check: Visit http://localhost:7860/

2. The Frontend (Avatar-UI)

1. Navigate to the frontend directory: cd frontend
2. Create a .env.local file and link the API: NEXT_PUBLIC_API_URL=http://localhost:7860
3. Install dependencies and run Turbopack:

npm install  
npm run dev

Application: Visit http://localhost:3000

🚧 Limitations & Next Steps

• LLM Latency: Currently, requests take ~500ms - 800ms to resolve via Groq. While incredibly fast for an LLM, this latency is noticeable in a real-time 3D environment. Next Step: Implement local Vector routing (Cosine Similarity) to bypass the LLM for common commands, dropping latency to <50ms.
• Baked Animations: The character is limited to its 14 pre-rigged states. It cannot interact with dynamic props (e.g., picking up a randomly placed hard hat). Next Step: Integrate Procedural Inverse Kinematics (IK).