Backup before continuing from Codex 5.2 session - User storage, compaction suggestions, streaming improvements

This commit is contained in:
Gemini AI
2025-12-24 21:27:05 +04:00
Unverified
parent f9748391a9
commit e8c38b0add
93 changed files with 10615 additions and 2037 deletions

View File

@@ -0,0 +1,140 @@
## Implementation Plan: Enhanced Session Compaction System (9 High-Priority Fixes)
### Phase 1: Core Foundation (Types & Configuration)
**NEW: `packages/ui/src/stores/session-compaction.ts`**
1. **Compaction Types & Interfaces**
- `CompactionMessageFlags`: summary, mode, provenance flags
- `StructuredSummary`: Tier A/B schema with what_was_done, files, current_state, key_decisions, next_steps, blockers, artifacts, tags, provenance
- `CompactionEvent`: Audit trail with event_id, timestamp, actor, trigger_reason, token_before/after, model_used, cost_estimate
- `CompactionConfig`: autoCompactEnabled, autoCompactThreshold, compactPreserveWindow, pruneReclaimThreshold, userPreference, undoRetentionWindow
- `SessionCompactingHook`: Plugin contract for domain-specific rules
2. **Configuration Store**
- Default config: auto=80%, preserve=40k tokens, prune_threshold=20k, preference="ask"
- Export functions: `getCompactionConfig()`, `updateCompactionConfig()`
### Phase 2: Overflow Detection Engine
**MODIFY: `packages/ui/src/stores/session-compaction.ts`**
3. **Token Monitoring Functions**
- `isOverflowDetected(usage, modelLimit)`: Check if usage >= threshold%
- `shouldPruneToolOutputs(usage)`: Check if tool outputs > reclaim threshold
- `estimateTokenReduction(before, after)`: Calculate % reduction
4. **Audit Trail System**
- `recordCompactionEvent(sessionId, event)`: Append-only to audit log
- `getCompactionHistory(sessionId)`: Retrieve audit trail
- `exportAuditLog()`: For compliance/debugging
### Phase 3: Secrets Detection & Sanitization
**NEW: `packages/ui/src/lib/secrets-detector.ts`**
5. **Secrets Detector**
- Pattern matching for: api keys, passwords, tokens, secrets, credentials
- `redactSecrets(content)`: Returns { clean: string, redactions: { path, reason }[] }
- Placeholder format: `[REDACTED: {reason}]`
### Phase 4: AI-Powered Compaction Agent
**MODIFY: `packages/ui/src/stores/session-compaction.ts`**
6. **Compaction Agent Integration**
- `COMPACTION_AGENT_PROMPT`: Structured prompt with instructions
- `generateCompactionSummary(instanceId, sessionId, window)`: Call sendMessage() to get AI summary
- Parse response into Tier A (human) and Tier B (structured JSON)
7. **Execute Compaction**
- `executeCompaction(instanceId, sessionId, mode)`: Main compaction orchestration
- Steps: enumerate → plugin hooks → AI summary → sanitize → store → prune → audit
- Returns: preview, token estimate, compaction event
### Phase 5: Pruning Engine
**MODIFY: `packages/ui/src/stores/session-compaction.ts`**
8. **Sliding Window Pruning**
- `pruneToolOutputs(instanceId, sessionId)`: Maintain queue, prune oldest > threshold
- `isToolOutput(part)`: Classify build logs, test logs, large JSON
### Phase 6: Undo & Rehydration
**MODIFY: `packages/ui/src/stores/session-compaction.ts`**
9. **Undo System**
- `undoCompaction(sessionId, compactionEventId)`: Rehydrate within retention window
- `getCompactedSessionSummary(sessionId)`: Retrieve stored summary
- `expandCompactedView(sessionId)`: Return archived messages
### Phase 7: Integration
**MODIFY: `packages/ui/src/stores/session-events.ts`**
10. **Auto-Compact Trigger**
- Monitor `EventSessionUpdated` for token usage
- Trigger based on user preference (auto/ask/never)
- Call existing `showConfirmDialog()` with compaction preview
**MODIFY: `packages/ui/src/stores/session-actions.ts`**
11. **Replace compactSession**
- Use new `executeCompaction()` function
- Support both "prune" and "compact" modes
### Phase 8: Schema Validation
**NEW: `packages/ui/src/lib/compaction-validation.ts`**
12. **Schema Validation**
- `validateStructuredSummary(summary)`: Zod schema for Tier B
- `validateCompactionEvent(event)`: Zod schema for audit trail
- `ValidationErrors` type with path, message, code
### Phase 9: CI Tests
**NEW: `packages/ui/src/stores/session-compaction.test.ts`**
13. **Test Coverage**
- `test_overflow_detection`: Verify threshold calculation
- `test_secrets_redaction`: Verify patterns are caught
- `test_compaction_execution`: Full compaction flow
- `test_undo_rehydration`: Verify restore works
- `test_plugin_hooks`: Verify custom rules apply
### Phase 10: Canary Rollout
**MODIFY: `packages/ui/src/stores/session-compaction.ts`**
14. **Feature Flag**
- `ENABLE_SMART_COMPACTION`: Environment variable or config flag
- Default: `false` for canary, set to `true` for full rollout
- Graceful degradation: fall back to simple compaction if disabled
---
## Implementation Order (Priority)
1. **P0 - Foundation**: Types, config, schema validation (1-2, 12)
2. **P0 - Core Engine**: Overflow detection, secrets detector (3-5)
3. **P0 - AI Integration**: Compaction agent, execute function (6-7)
4. **P1 - Pruning**: Tool output classification, sliding window (8)
5. **P1 - Undo**: Rehydration system (9)
6. **P1 - Integration**: Session events, actions integration (10-11)
7. **P2 - Tests**: CI test coverage (13)
8. **P2 - Rollout**: Feature flag, canary enablement (14)
---
## Success Criteria
- ✅ AI generates meaningful summaries (not just "0 AI responses")
- ✅ Overflow detected before context limit exceeded
- ✅ Secrets are redacted before storage
- ✅ Audit trail tracks every compaction
- ✅ Undo works within retention window
- ✅ Schema validation prevents corrupt data
- ✅ CI tests ensure reliability
- ✅ Canary flag allows safe rollout

View File

@@ -0,0 +1,391 @@
# FINAL EXECUTION PLAN - 8 Fixes with Proof Deliverables
## Fix Summary
| Fix | Files | Deliverables |
|------|--------|-------------|
| C1 | Install-Windows.bat, Install-Mac.sh, Install-Linux.sh, Launch-Windows.bat, Launch-Dev-Windows.bat, Launch-Unix.sh | 9 path diffs + `dir packages\ui\dist` verification |
| C2 | packages/ui/vite.config.ts, Launch-Dev-Windows.bat, Launch-Dev-Unix.sh (NEW) | vite.config.ts diff + 2 launcher diffs + Vite log showing port |
| C3 | Launch-Windows.bat, Launch-Dev-Windows.bat, Launch-Unix.sh | 3 CLI_PORT env var diffs + server log showing port |
| C4 | Install-Windows.bat, Install-Mac.sh, Install-Linux.sh | 3 download/checksum diffs + log verification |
| C5 | Install-Windows.bat | Certutil parsing diff + hash output |
| C6 | Install-Windows.bat, Install-Mac.sh, Install-Linux.sh | 3 TARGET_DIR/BIN_DIR diffs + fallback test output |
| C7 | Install-Windows.bat, Install-Mac.sh, Install-Linux.sh | 3 health check path diffs + health check output |
| C8 | Launch-Dev-Windows.bat | 1 path diff + grep verification |
---
## C1: UI Build Path Correction
**Files:** Install-Windows.bat (lines 194, 245), Install-Mac.sh (204, 256), Install-Linux.sh (220, 272), Launch-Windows.bat (185), Launch-Dev-Windows.bat (144), Launch-Unix.sh (178)
**Diff:**
```batch
# All Windows scripts - replace:
packages\ui\src\renderer\dist
# With:
packages\ui\dist
# All Unix scripts - replace:
packages/ui/src/renderer/dist
# With:
packages/ui/dist
```
**Verification:** `dir packages\ui\dist` + `dir packages\ui\dist\index.html`
---
## C2: Vite Dev Server Port Wiring
**File 1: packages/ui/vite.config.ts (line 23)**
```diff
- server: {
- port: 3000,
- },
+ server: {
+ port: Number(process.env.VITE_PORT ?? 3000),
+ },
```
**File 2: Launch-Dev-Windows.bat (after port detection)**
```diff
- start "NomadArch UI" cmd /k "cd /d \"%~dp0packages\ui\" && set VITE_PORT=!UI_PORT! && npm run dev"
+ start "NomadArch UI" cmd /k "cd /d \"%~dp0packages\ui\" && set VITE_PORT=!UI_PORT! && npm run dev -- --port !UI_PORT!"
```
**File 3: Launch-Dev-Unix.sh (NEW FILE)**
```bash
#!/bin/bash
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "$SCRIPT_DIR"
# Port detection
DEFAULT_SERVER_PORT=3001
DEFAULT_UI_PORT=5173
SERVER_PORT=$DEFAULT_SERVER_PORT
UI_PORT=$DEFAULT_UI_PORT
echo "[INFO] Detecting available ports..."
# Server port (3001-3050)
for port in {3001..3050}; do
if ! lsof -i :$port -sTCP:LISTEN -t > /dev/null 2>&1; then
SERVER_PORT=$port
break
fi
done
# UI port (5173-5200)
for port in {5173..5200}; do
if ! lsof -i :$port -sTCP:LISTEN -t > /dev/null 2>&1; then
UI_PORT=$port
break
fi
done
echo "[INFO] Using server port: $SERVER_PORT"
echo "[INFO] Using UI port: $UI_PORT"
# Start server with CLI_PORT
echo "[INFO] Starting Backend Server..."
cd packages/server
export CLI_PORT=$SERVER_PORT
npm run dev &
SERVER_PID=$!
sleep 3
# Start UI with VITE_PORT + --port flag
echo "[INFO] Starting Frontend UI..."
cd "$SCRIPT_DIR/packages/ui"
export VITE_PORT=$UI_PORT
npm run dev -- --port $UI_PORT &
UI_PID=$!
sleep 3
# Start Electron
echo "[INFO] Starting Electron..."
cd "$SCRIPT_DIR/packages/electron-app"
npm run dev
# Cleanup on exit
trap "kill $SERVER_PID $UI_PID 2>/dev/null; exit" INT TERM
```
**Verification:** Vite log output showing `Local: http://localhost:<detected_port>`
---
## C3: Server Port Environment Variable
**Launch-Windows.bat (before npm run dev:electron):**
```diff
echo [INFO] Starting NomadArch...
set SERVER_URL=http://localhost:!SERVER_PORT!
echo [INFO] Server will run on http://localhost:!SERVER_PORT!
+
+ set CLI_PORT=!SERVER_PORT!
call npm run dev:electron
```
**Launch-Dev-Windows.bat (server start command):**
```diff
echo [INFO] Starting Backend Server...
- start "NomadArch Server" cmd /k "cd /d \"%~dp0packages\server\" && npm run dev"
+ start "NomadArch Server" cmd /k "cd /d \"%~dp0packages\server\" && set CLI_PORT=!SERVER_PORT! && npm run dev"
```
**Launch-Unix.sh (before npm run dev:electron):**
```bash
echo -e "${GREEN}[INFO]${NC} Starting NomadArch..."
SERVER_URL="http://localhost:$SERVER_PORT"
echo -e "${GREEN}[INFO]${NC} Server will run on http://localhost:$SERVER_PORT"
export CLI_PORT=$SERVER_PORT
npm run dev:electron
```
**Verification:** Server log showing `CodeNomad Server is ready at http://127.0.0.1:<detected_port>`
---
## C4: OpenCode Download with Dynamic Version + Checksum
**Install-Windows.bat (lines 165-195):**
```batch
set TARGET_DIR=%SCRIPT_DIR%
set BIN_DIR=%TARGET_DIR%\bin
if not exist "%BIN_DIR%" mkdir "%BIN_DIR%"
:: Resolve latest version from GitHub API
echo [INFO] Resolving latest OpenCode version...
for /f "delims=" %%v in ('curl -s https://api.github.com/repos/sst/opencode/releases/latest ^| findstr "\"tag_name\""') do (
set OPENCODE_VERSION=%%v
set OPENCODE_VERSION=!OPENCODE_VERSION:~18,-2!
)
set OPENCODE_BASE=https://github.com/sst/opencode/releases/download/v%OPENCODE_VERSION%
set OPENCODE_URL=%OPENCODE_BASE%/opencode-windows-%ARCH%.exe
set CHECKSUM_URL=%OPENCODE_BASE%/checksums.txt
if exist "%BIN_DIR%\opencode.exe" (
echo [OK] OpenCode binary already exists
) else (
echo [INFO] Downloading OpenCode v%OPENCODE_VERSION%...
echo Downloading from: %OPENCODE_URL%
:: Download binary to BIN_DIR
curl -L -o "%BIN_DIR%\opencode.exe.tmp" "%OPENCODE_URL%"
if %ERRORLEVEL% neq 0 (
echo [ERROR] Download failed!
set /a ERRORS+=1
goto :skip_opencode
)
:: Download checksums
curl -L -o "%BIN_DIR%\checksums.txt" "%CHECKSUM_URL%"
:: Extract expected checksum
set EXPECTED_HASH=
for /f "tokens=1,2" %%h in ('type "%BIN_DIR%\checksums.txt" ^| findstr /i "opencode-windows-%ARCH%"') do (
set EXPECTED_HASH=%%h
)
:: Calculate actual hash (line 2 from certutil)
set ACTUAL_HASH=
for /f "skip=1 tokens=*" %%h in ('certutil -hashfile "%BIN_DIR%\opencode.exe.tmp" SHA256 ^| findstr /v "CertUtil" ^| findstr /v "hash of"') do (
set ACTUAL_HASH=%%h
goto :hash_found
)
:hash_found
:: Verify and output hashes
echo Expected hash: !EXPECTED_HASH!
echo Actual hash: !ACTUAL_HASH!
if "!ACTUAL_HASH!"=="!EXPECTED_HASH!" (
move /Y "%BIN_DIR%\opencode.exe.tmp" "%BIN_DIR%\opencode.exe"
echo [OK] OpenCode downloaded and verified
echo [%date% %time%] OpenCode v%OPENCODE_VERSION% downloaded, checksum verified >> "%TARGET_DIR%\install.log"
) else (
echo [ERROR] Checksum mismatch!
del "%BIN_DIR%\opencode.exe.tmp"
set /a ERRORS+=1
)
)
:skip_opencode
```
**Install-Mac.sh / Install-Linux.sh:** Similar pattern with `opencode-darwin-${ARCH}` and `opencode-linux-${ARCH}`, using `TARGET_DIR/bin`
**Verification:** Log shows `OpenCode v<x.y.z> downloaded, checksum verified` + `ls TARGET_DIR/bin/opencode` exists
---
## C5: Windows Checksum Parsing
**Included in C4 above.** Key change:
```batch
:: Parse certutil output - hash is on line 2
for /f "skip=1 tokens=*" %%h in ('certutil -hashfile "%BIN_DIR%\opencode.exe.tmp" SHA256 ^| findstr /v "CertUtil" ^| findstr /v "hash of"') do (
set ACTUAL_HASH=%%h
goto :hash_found
)
```
**Verification:** Output shows matching hashes:
```
Expected hash: abc123def456...
Actual hash: abc123def456...
```
---
## C6: Permission Fallback with TARGET_DIR/BIN_DIR
**Install-Windows.bat (lines 125-160):**
```batch
set TARGET_DIR=%SCRIPT_DIR%
set BIN_DIR=%TARGET_DIR%\bin
set NEEDS_FALLBACK=0
echo [STEP 2/10] Checking Write Permissions...
echo.
echo. > "%SCRIPT_DIR%\test-write.tmp" 2>nul
if %ERRORLEVEL% neq 0 (
echo [WARN] Cannot write to current directory: %SCRIPT_DIR%
echo [INFO] Setting fallback for install outputs...
set TARGET_DIR=%USERPROFILE%\NomadArch-Install
set BIN_DIR=%TARGET_DIR%\bin
if not exist "%TARGET_DIR%" mkdir "%TARGET_DIR%"
if not exist "%BIN_DIR%" mkdir "%BIN_DIR%"
echo. > "%TARGET_DIR%\test-write.tmp" 2>nul
if %ERRORLEVEL% neq 0 (
echo [ERROR] Cannot write to fallback directory either!
set /a ERRORS+=1
goto :final_check
)
echo [OK] Using fallback for outputs: %TARGET_DIR%
echo [%date% %time%] Using fallback: %TARGET_DIR% >> "%TARGET_DIR%\install.log"
set NEEDS_FALLBACK=1
del "%TARGET_DIR%\test-write.tmp"
) else (
if not exist "%BIN_DIR%" mkdir "%BIN_DIR%"
del "%SCRIPT_DIR%\test-write.tmp"
echo [OK] Write permissions verified
)
:: All log writes use TARGET_DIR
set LOG_FILE=%TARGET_DIR%\install.log
```
**Install-Mac.sh / Install-Linux.sh:** Similar pattern with `TARGET_DIR=$HOME/.nomadarch-install`, `BIN_DIR=$TARGET_DIR/bin`
**Verification:** Run from read-only directory, output shows `Using fallback for outputs: C:\Users\xxx\NomadArch-Install`
---
## C7: Health Check Path Corrections
**Install-Windows.bat (health check section):**
```diff
:: UI health check
- if exist "%SCRIPT_DIR%\packages\ui\src\renderer\dist" (
+ if exist "%SCRIPT_DIR%\packages\ui\dist\index.html" (
echo [OK] UI build directory exists
) else (
- echo [ERROR] UI build directory not found
+ echo [ERROR] UI build directory not found at packages\ui\dist
set /a HEALTH_ERRORS+=1
)
:: Electron health check
- if exist "%SCRIPT_DIR%\packages\electron-app\dist\main.js" (
+ if exist "%SCRIPT_DIR%\packages\electron-app\dist\main\main.js" (
echo [OK] Electron main.js exists
) else (
echo [WARN] Electron build not found (will build on launch)
)
```
**Install-Mac.sh / Install-Linux.sh:** Same logic with shell syntax
**Verification:** Health check output:
```
[OK] UI build directory exists
[OK] Electron main.js exists
```
---
## C8: Launch-Dev-Windows Electron Path Fix
**Launch-Dev-Windows.bat line 162:**
```diff
- if not exist "electron-app\dist\main.js" (
+ if not exist "packages\electron-app\dist\main\main.js" (
```
**Verification:** `grep -n "electron-app" Launch-Dev-Windows.bat` shows no `electron-app\` references remaining
---
## Execution Order
1. C6 (TARGET_DIR/BIN_DIR) - Foundation for C4
2. C7 (Health checks) - Independent path fixes
3. C1 (UI paths) - Quick path replacements
4. C8 (Launch-Dev-Windows) - Quick path fix
5. C2 (Vite port) - Includes new file creation
6. C3 (Server port) - Quick env var changes
7. C4 (OpenCode download) - Depends on C6, includes C5
8. **Run build** for C1/C7 verification
---
## Verification Commands to Run
| Fix | Command | Expected Output |
|------|----------|----------------|
| C1 | `dir packages\ui\dist` | Shows `index.html`, `assets/` |
| C2 | Run Launch-Dev, check Vite log | `Local: http://localhost:3001` |
| C3 | Run launcher, check server log | `CodeNomad Server is ready at http://127.0.0.1:3001` |
| C4 | Run install, grep log | `OpenCode v<x.y.z> downloaded, checksum verified` |
| C5 | Run install, check log | Hashes match in output |
| C6 | Run from read-only dir | `Using fallback: C:\Users\xxx\NomadArch-Install` |
| C7 | Run install, check output | `UI build directory exists` + `Electron main.js exists` |
| C8 | `grep -n "electron-app" Launch-Dev-Windows.bat` | Only `packages\electron-app` or commented lines |
---
## Files Modified/Created
| File | Action |
|------|--------|
| Install-Windows.bat | Edit (C1, C4, C5, C6, C7) |
| Install-Mac.sh | Edit (C1, C4, C6, C7) |
| Install-Linux.sh | Edit (C1, C4, C6, C7) |
| Launch-Windows.bat | Edit (C1, C3) |
| Launch-Dev-Windows.bat | Edit (C1, C2, C3, C8) |
| Launch-Unix.sh | Edit (C1, C3) |
| Launch-Dev-Unix.sh | CREATE (C2) |
| packages/ui/vite.config.ts | Edit (C2) |