Scope negotiation

Conversation

• pm
  ## System Architecture & Technical Specifications ``` +-------------------------------------------------------+ | Next.js Fullstack | | (App Router + Tailwind + shadcn/ui + TypeScript) | +---------------------------+---------------------------+ | +------------------------------------+-----------------------------------+ | | | v v v +-------------------+ +--------------------+ +-------------------+ | Ingestion Engine | | Consensus Engine | | Cloud & Storage | | - File Parsing | | - LLM Provider API | | - Google Drive API| | (PDF/CSV/TXT/ | | (Gemini, OpenAI, | | (Service Account| | Images) | | Grok Adapter) | | Auth) | | - PDF Renamer | | - Multi-Round Loop | | - SQLite Database | +-------------------+ +--------------------+ +-------------------+ ``` --- ## 1. Complete Feature Matrix ### Ingestion & Processing Pipeline * **Multi-Format Ingestion:** Drag-and-drop / file picker on the landing page accepting `.pdf`, `.png`, `.jpg`, `.jpeg`, `.csv`, `.txt`, and `.md`. * **Universal Normalization:** Text/CSV/Image inputs are parsed into text representations and normalized into a standard PDF artifact for archiving. * **Smart Renaming Engine:** Generates target PDF file names following the strict pattern: `ddmmyyyy-sender-sequence.pdf` (e.g., `22082026-AcmeCorp-001.pdf`), extracting the sender and date via the AI consensus or falling back to user input. ### Multi-LLM Consensus Engine (Extensible) * **Provider Abstraction Interface:** Standardized `BaseLLMProvider` contract (`generateResponse`, `streamResponse`, `modelName`) allowing zero-friction addition of new models (OpenAI, Gemini, Grok, Claude). * **Sequential Peer Review Loop (Option A):** * **Round 1:** Model A (e.g., Gemini 1.5 Pro) reads the raw document and produces an initial structured assessment and tag recommendations. * **Round 2:** Model B (e.g., OpenAI GPT-4o) critiques Model A's extraction, flags discrepancies, and refines key data points. * **Rounds 3–10 (Configurable):** Iterative refinement between models until both models signal consensus or the 10-round cap is reached. * **Synthesis Step:** A unified assessment report (Markdown format) is generated and delivered to the chat window. * **Tag Suggestion Mechanism:** Models return candidate tags based on document analysis; suggestions are displayed as clickable chips in the UI without modifying the database. ### Tag & Metadata Management * **Dynamic User-Driven Tags:** Clean-slate tag system with zero hardcoded tags. * **Manual Tag Creation:** Dedicated "Create Tag" modal/button in the chat interface allowing custom tag creation on the fly. * **Tag Assignment:** Documents can hold multiple user-confirmed tags, stored relationally in SQLite. ### Chat & Interactive Workspace * **ChatGPT/Gemini-Style UI:** Responsive interface featuring a collapsible left sidebar (recent uploads and conversations) and a central chat stream. * **Post-Assessment Interactivity:** User can ask follow-up questions, request specific recalculations, or instruct the models to re-evaluate sections. * **Assessment Preview:** Split-screen or toggle view to see the source document alongside the generated Markdown assessment. * **Session Reset / Logout:** Clears current active session state and resets the workspace to the landing state. ### Google Drive & Database Persistence * **Headless Service Account Integration:** Authenticates automatically using a root-level `service_account.json` credential file to access a designated Drive folder. * **Database Archival:** Persists document metadata, UUIDs, full chat transcripts, tag relationships, and Markdown/PDF assessment reports in a local SQLite database. --- ## 2. Dependency & Stack Assessment This stack is selected for optimal compatibility with **Cursor** (TypeScript end-to-end type safety, unified Next.js App Router, zero-boilerplate ORM). ### Recommended Tech Stack * **Framework:** Next.js 14/15 (App Router, Server Actions, Route Handlers) * **Styling & Components:** Tailwind CSS + `shadcn/ui` + `lucide-react` * **Database & ORM:** SQLite via `@libsql/client` or `better-sqlite3` + `drizzle-orm` (lightweight, zero external server requirement) * **File & PDF Processing:** `pdf-parse`, `pdf-lib`, `papaparse`, `tesseract.js` (or LLM Vision API for OCR) ### Core Dependencies & Environment Configuration | Package | Purpose | Version Spec | | --- | --- | --- | | `googleapis` | Google Drive API v3 Service Account Client | `^140.0.0` | | `@google/genai` | Google Gemini SDK | Latest | | `openai` | OpenAI & Grok (xAI API is OpenAI-compatible) | `^4.50.0` | | `drizzle-orm` / `drizzle-kit` | SQLite Database Schema & Query Builder | Latest | | `better-sqlite3` | Local database engine | `^9.0.0` | | `pdf-lib` | Client/Server PDF creation & metadata manipulation | `^1.17.1` | | `papaparse` | CSV data parsing | `^5.4.1` | | `zod` | Runtime validation for LLM structured outputs | `^3.23.0` | #### Environment Variables (`.env.local`) ```env # AI Providers GEMINI_API_KEY="AIzaSy..." OPENAI_API_KEY="sk-proj-..." GROK_API_KEY="xai-..." DEFAULT_MAX_DEBATE_ROUNDS=10 # Google Drive Configuration GOOGLE_APPLICATION_CREDENTIALS="./service_account.json" GOOGLE_DRIVE_FOLDER_ID="1A2B3C4D5E6F7G8H9I0J" # Database DATABASE_URL="file:./data/app.db" ``` --- ## 3. Database Schema Design (SQLite / Drizzle) ```sql -- Documents Table CREATE TABLE documents ( id TEXT PRIMARY KEY, -- UUIDv4 original_filename TEXT NOT NULL, gdrive_filename TEXT, gdrive_file_id TEXT, mime_type TEXT NOT NULL, local_file_path TEXT NOT NULL, sender TEXT, sequence_number INTEGER, status TEXT CHECK(status IN ('uploaded', 'analyzing', 'ready', 'exported', 'failed')) DEFAULT 'uploaded', created_at DATETIME DEFAULT CURRENT_TIMESTAMP, updated_at DATETIME DEFAULT CURRENT_TIMESTAMP ); -- Tags Table (User-created only) CREATE TABLE tags ( id TEXT PRIMARY KEY, -- UUIDv4 name TEXT UNIQUE NOT NULL, color TEXT DEFAULT '#3B82F6', created_at DATETIME DEFAULT CURRENT_TIMESTAMP ); -- Document <-> Tags Junction Table CREATE TABLE document_tags ( document_id TEXT NOT NULL REFERENCES documents(id) ON DELETE CASCADE, tag_id TEXT NOT NULL REFERENCES tags(id) ON DELETE CASCADE, PRIMARY KEY (document_id, tag_id) ); -- Assessments Table CREATE TABLE assessments ( id TEXT PRIMARY KEY, -- UUIDv4 document_id TEXT NOT NULL REFERENCES documents(id) ON DELETE CASCADE, final_markdown TEXT NOT NULL, debate_rounds_count INTEGER NOT NULL, raw_debate_transcript TEXT, -- JSON Array of internal model exchanges created_at DATETIME DEFAULT CURRENT_TIMESTAMP ); -- Chat Transcript Messages CREATE TABLE chat_messages ( id TEXT PRIMARY KEY, -- UUIDv4 document_id TEXT NOT NULL REFERENCES documents(id) ON DELETE CASCADE, sender_role TEXT CHECK(sender_role IN ('user', 'assistant', 'system', 'gemini', 'openai', 'grok')) NOT NULL, content TEXT NOT NULL, created_at DATETIME DEFAULT CURRENT_TIMESTAMP ); ``` --- ## 4. Extensible LLM Provider Interface ```typescript // lib/llm/types.ts export interface LLMMessage { role: 'system' | 'user' | 'assistant'; content: string; } export interface LLMResponse { content: string; suggestedTags?: string[]; metadata?: { sender?: string; date?: string; }; } export interface BaseLLMProvider { name: string; analyzeDocument(content: string | Buffer, mimeType: string, prompt: string): Promise<LLMResponse>; critiqueAssessment(documentContent: string, previousAssessment: string, roundNumber: number): Promise<LLMResponse>; chat(messages: LLMMessage[]): Promise<string>; } ``` --- ## 5. Comprehensive User Stories Catalog ### Epic 1: Document Upload & Ingestion * **US-1.1: Multi-Format Ingestion** * **As a** user, * **I want to** drag and drop or upload PDF, Image, CSV, TXT, or Markdown files on the landing page, * **So that** I can process any document regardless of its source format. * **Acceptance Criteria:** * Landing page shows a clean upload dropzone. * Supported file formats (`.pdf`, `.png`, `.jpg`, `.jpeg`, `.csv`, `.txt`, `.md`) are accepted; all others show an error toast. * File is assigned a unique UUID and stored in local temporary storage. * **US-1.2: Transition to Active Workspace** * **As a** user, * **I want to** be automatically transitioned to the chat workspace once a file is uploaded, * **So that** I can monitor the analysis and begin interacting immediately. * **Acceptance Criteria:** * Landing page smoothly transitions to the ChatGPT/Gemini-style dual-panel workspace upon upload completion. * Document status indicators update from "Processing" to "Analyzing" to "Ready". --- ### Epic 2: Multi-LLM Collaborative Review Engine * **US-2.1: Sequential Peer Review (Gemini & OpenAI)** * **As a** user, * **I want** Gemini and OpenAI to collaboratively review the document up to a maximum of 10 rounds, * **So that** I receive a highly accurate, audited assessment report. * **Acceptance Criteria:** * Model A generates initial extraction and summary. * Model B reviews Model A's output against the raw file content, pointing out corrections or omissions. * The process repeats sequentially until models reach agreement or hit 10 rounds maximum. * Final consolidated Markdown report is rendered in the main chat viewport. * **US-2.2: Third-Party Model Extensibility (Grok / Claude)** * **As a** developer/user, * **I want** the LLM engine to follow an abstracted interface, * **So that** I can plug in Grok (xAI) or other providers without rewriting the core workflow. * **Acceptance Criteria:** * Adding a new provider requires only implementing the `BaseLLMProvider` interface and registering it in a factory map. --- ### Epic 3: Tagging & Metadata Control * **US-3.1: Zero-Seed Dynamic Tag Creation** * **As a** user, * **I want to** create custom tags manually whenever needed with no predefined seed tags, * **So that** my taxonomy matches my personal organization requirements. * **Acceptance Criteria:** * Initial database contains 0 tags. * "Create Tag" button opens a quick modal with tag name and color selection. * Newly created tag is immediately available for document association. * **US-3.2: AI Tag Suggestions (Read-Only)** * **As a** user, * **I want to** see tag recommendations from the AI without them being auto-created, * **So that** I maintain complete control over my tag taxonomy. * **Acceptance Criteria:** * LLM consensus response includes suggested tags rendered as clickable chips. * Clicking a suggestion checks if the tag exists; if not, it prompts the user to confirm creation before applying. --- ### Epic 4: Interactive Chat & Post-Assessment * **US-4.1: Assessment Discussion** * **As a** user, * **I want to** ask follow-up questions about the assessment in the chat interface, * **So that** I can clarify specific clauses, values, or calculations. * **Acceptance Criteria:** * Chat input accepts questions and responds with context from both the document and consensus report. * Full message history is appended to `chat_messages` in the database. * **US-4.2: Recent Conversations & File Sidebar** * **As a** user, * **I want** a sidebar listing recent uploads and past conversations, * **So that** I can switch back to previous documents and review their state. * **Acceptance Criteria:** * Left sidebar displays chronological list of uploaded documents with assigned tags. * Clicking a historical item loads its chat transcript, assessment, and metadata. --- ### Epic 5: Google Drive Archival & Database Storage * **US-5.1: File Renaming & Standardized PDF Export** * **As a** user, * **I want** the uploaded file converted to PDF and renamed as `ddmmyyyy-sender-sequence.pdf`, * **So that** all stored documents have consistent, predictable file names. * **Acceptance Criteria:** * Non-PDF documents are converted to PDF format. * Sender name is extracted from AI metadata (or requested from user if missing). * Date format is strictly `ddmmyyyy`. Sequence number increments based on existing records in the database. * **US-5.2: Service Account Google Drive Upload** * **As a** system/user, * **I want** the renamed PDF uploaded to Google Drive via a local `service_account.json` file, * **So that** I do not have to perform interactive OAuth logins. * **Acceptance Criteria:** * Server loads `service_account.json` from the root directory. * File uploads to the specified Google Drive target folder. * Returned Google Drive File ID and web view link are recorded in the database. * **US-5.3: Database Transcript & Assessment Persistence** * **As a** user, * **I want** the original filename, generated report, and full chat transcript persisted in the database, * **So that** I have a permanent local audit trail. * **Acceptance Criteria:** * `assessments` and `chat_messages` tables are updated at the end of each session. * Database stores the original file name alongside the renamed Drive file name.
• po
  Thanks for the comprehensive overview, PM! This gives us a great foundation for Doc Flow AI. To effectively define our MVP scope, especially considering our fixed **React frontend** and **Python Flask backend**, I need a bit more clarity on the initial product focus. Could you please help me with the following: 1. **Who is the primary user** for this MVP? 2. **What is the single most critical problem** this MVP aims to solve for them? 3. Based on that problem, which of the proposed Epics (Document Upload & Ingestion, Multi-LLM Collaborative Review Engine, Tagging & Metadata Control, Interactive Chat & Post-Assessment, Google Drive Archival & Database Storage) are **absolutely essential for the MVP**? Please indicate their priority. 4. What would be the **key acceptance criteria** for the MVP to be considered successful from this primary user's perspective?