admin/OpenQode
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add UI Automation 'find' command for true textual vision

Browse Source
This commit is contained in:
Gemini AI
2025-12-14 02:31:53 +04:00
Unverified
parent ce4f0036c2
commit fb2991ba1a
2 changed files with 65 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
bin/opencode-ink.mjs
View File

@@ -491,14 +491,15 @@ Use it to control the mouse, keyboard, and "see" the system.
## 👁️ VISION & BLINDNESS PROTOCOL:
You are a TEXT-BASED intelligence. You CANNOT see images/screenshots you take.
- **\`input.ps1 screenshot\`**: Creates an image for the **USER** to see. You learn NOTHING from this.
- **\`input.ps1 find "Name"\`**: **TRUE VISION**. Finds a UI element (button/window) by text and tells you where it is.
- **\`input.ps1 apps\`**: Your "Eyes" for windows. Returns TEXT list of open apps.
- **\`input.ps1 screen\`**: Your "Eyes" for geometry. Returns TEXT resolution (e.g. 1920x1080).
- **\`input.ps1 screen\`**: Your "Eyes" for geometry. Returns TEXT resolution.
### 📐 THE LAW OF COORDINATES:
Since you cannot see buttons, you MUST calculate them using \`screen\` dimensions.
1. Run \`powershell bin/input.ps1 screen\`.
2. Get Output: \`Width x Height\` (e.g. 1920 x 1200).
### 📐 THE LAW OF ACCURACY:
1. **FIND FIRST**: If you need to click a button, SEARCH FOR IT.
- \`powershell bin/input.ps1 find "Start"\` -> Returns "Found at (30, 1190)".
- **THEN** use those coordinates to click.
2. **FALLBACK**: Only calculate coordinates manually if \`find\` fails.
3. **Start Menu Logic:** Bottom-Left corner.
- X = 0 to 50
- Y = Height - 10 (e.g. 1190).