Bemnet Beshah

Projects / Software

Ayn

Ayn is a developer tool that gives a coding agent a view of a running web page. It opens the page in Chromium, measures interactions and page elements, and writes a report with a fix prompt for each issue. Our four-person team built it at HackTX 2025, where Devpost lists it as a winner of the “[MLH] Best Use of Gemini 2.5 Computer Use Model” prize. I led the team. The judged demo used Gemini Computer Use live.

Problem

Coding agents read and write code, but they don’t see the page. They miss a stuttering hover, a button that is too small to tap, or a link that returns 404. Developers fill that gap by hand: change code, rebuild, screenshot, describe the problem, repeat.

Ayn automates the “look at the page and describe what is wrong” step. Its output is meant to be pasted into an agent such as Cursor or Copilot.

My role

Four-person team: Abel Tadele, Kidus Beshah, Nathan Negera, and Bemnet Beshah (me). I led the team.

I wrote:

  • The initial repo layout and a README that assigned each module to an owner.
  • The orchestrator in tool/orchestrator/loop.py: a task runner that rewrites the user’s instruction with an LLM, asks Gemini Computer Use for a plan, calls the executor over HTTP and builds the audit.
  • The FastAPI app in tool/main.py and its tests.
  • Environment configuration, including the Python version pin and the setup docs.
  • The last feature before the demo: an LLM-written improvement prompt for each detected issue, shown in the CLI output and in ui.json.

Kidus’s executor, Nathan’s aggregator and Abel’s UI/CLI tooling are theirs.

How it works

instruction / prompt file
        |
        v
  demo CLI  ----------------->  executor (FastAPI + Playwright, Chromium)
 (builds action list)           hover, scan_links, scan_images, measure_target,
                                detect_overlap, screenshots, console capture
                                        |
                                        v  raw observation JSON
                                adapter: normalize via OpenRouter
                                (local fallback if the call fails)
                                        |
                                        v
                                aggregator: rule-based checks -> Issue list
                                        |
                                        v
                                LLM adds one improvement_prompt per issue
                                        |
                                        v
                         result.json (audit) + ui.json (viewer payload)

Issue detection is deterministic. The aggregator turns measurements into eight issue types: hover animation, console error, broken link, missing alt, missing accessible name, overlap, small click target and CTA not found. A run counts as failed if any issue has high severity. The LLM only normalizes the executor output and writes the fix prompts.

The orchestrator path that uses Gemini Computer Use is separate. run_task sends the rewritten instruction to the gemini-2.5-computer-use-preview-10-2025 model for one turn and maps the returned function calls onto the executor’s action list.

Key decisions

Measure with Playwright, use the LLM for language. The alternative was to let the model judge screenshots directly. Computed styles, bounding boxes and link status codes give repeatable evidence, and the model’s job is limited to wording the fix.

Allow-list Gemini’s actions. Computer Use can return actions the executor can’t run. _cu_to_playwright_actions keeps only eight supported functions and falls back to screenshot, page text and URL when nothing maps. The cost is that Gemini’s plan is flattened (hover and click always target body).

Every LLM step has a fallback. Prompt rewriting returns the raw instruction on failure. The improvement-prompt step returns the issues unchanged. The adapter has a local normalizer. A missing or failing API key degrades the report instead of killing the run.

Executor as a separate HTTP service. The orchestrator posts to /execute and retries once on failure. This let Kidus build the browser side while I built the orchestrator against its request schema.

What was hard

Wiring several models and services together under time pressure. Devpost lists async state handling around Gemini, prompt drift and environment conflicts between FastAPI, PowerShell and Python. The Python pin and pyenv-win install script I committed early on address that last one. The demo README gives separate Windows and Linux/macOS commands.

Keeping the modules compatible while four people worked in parallel. The executor’s raw output, the normalizer’s schema and the aggregator all had to agree, which is why the adapter has a local fallback mapper and the bridge coerces probe events defensively.

Results

  • Devpost lists Ayn as a winner of “[MLH] Best Use of Gemini 2.5 Computer Use Model” at HackTX 2025.
  • The repo contains a CLI demo that audits a local test page and reports hover animation, small click target, console error, broken link and missing alt issues. The README lists these as an example summary. The page has defects planted for this purpose.
  • No measurement of detection accuracy, latency or cost exists.

Limitations and next steps

  • The judged demo used Gemini Computer Use live. The repository also contains a scripted replay path; the available demo tooling should not be read as evidence that every path in the repository ran during judging.
  • The orchestrator makes one Gemini call per run. There is no loop that re-prompts a coding agent or re-checks a fix.
  • The checks were tested on a test page we built, not on real production sites.
  • Next step would be to connect run_task to the real audit path, then close the loop: audit, send the prompt to an agent, re-audit.