Hacker News

Show HN: Libretto – Making AI browser automations deterministic

52 points by muchael ago | 16 comments
Libretto (https://libretto.sh) is a Skill+CLI that makes it easy for your coding agent to generate deterministic browser automations and debug existing ones. Key shift is going from “give an agent a prompt at runtime and hope it figures things out” to: “Use coding agents to generate real scripts you can inspect, run, and debug”.

Here’s a demo: https://www.youtube.com/watch?v=0cDpIntmHAM. Docs start at https://libretto.sh/docs/get-started/introduction.

We spent a year building and maintaining browser automations for EHR and payer portal integrations at our healthcare startup. Building these automations and debugging failed ones was incredibly time-consuming.

There’s lots of tools that use runtime AI like Browseruse and Stagehand which we tried, but (1) they’re reliant on custom DOM parsing that's unreliable on older and complicated websites (including all of healthcare). Using a website’s internal network calls is faster and more reliable when possible. (2) They can be expensive since they rely on lots of AI calls and for workflows with complicated logic you can’t always rely on caching actions to make sure it will work. (3) They’re at runtime so it’s not interpretable what the agent is going to do. You kind of hope you prompted it correctly to do the right thing, but legacy workflows are often unintuitive and inconsistent across sites so you can’t trust an agent to just figure it out at runtime. (4) They don’t really help you generate new automations or help you debug automation failures.

We wanted a way to reliably generate and maintain browser automations in messy, high-stakes environments, without relying on fragile runtime agents.

Libretto is different because instead of runtime agents it uses “development-time AI”: scripts are generated ahead of time as actual code you can read and control, not opaque agent behavior at runtime. Instead of a black box, you own the code and can inspect, modify, version, and debug everything.

Rather than relying on runtime DOM parsing, Libretto takes a hybrid approach combining Playwright UI automation with direct network/API requests within the browser session for better reliability and bot detection evasion.

It records manual user actions to help agents generate and update scripts, supports step-through debugging, has an optional read-only mode to prevent agents from accidentally submitting or modifying data, and generates code that follows all the abstractions and conventions you have already in your coding repo.

Would love to hear how others are building and maintaining browser automations in practice, and any feedback on the approach we’ve taken here.

heyitsaamir |next [-]

I built something very similar for my company internally. The idea was that that the maintenance of the code is on the agent and the code is purely an optimization. If it breaks the agent runs it iteratively, fixes the code for next time. Happy to replace my tool with this and see how it does!

z3ugma |next |previous [-]

Love it! Do you have a BAA with Claude though? Otherwise, your demo is likely exposing PHI to 3rd parties and exposing you to risk related to HIPAA

muchael |root |parent [-]

It's a good callout. We have a BAA + ZDR with Anthropic and OpenAI, and if you want to use libretto for healthcare use cases having a BAA is essential. Was using Codex in the demo, and we've seen that both Claude and Codex work pretty well

alexbike |next |previous [-]

The network-request-first approach is the right call. DOM parsing is fragile because it's scraping a rendering artifact — any style refactor, framework upgrade, or A/B test can silently break it. Intercepting the actual API calls the browser is already making is closer to testing the contract, not the presentation.

The healthcare context makes this especially sharp. Those portals are notoriously inconsistent and rarely built for automation. Would be curious how you handle cases where the app uses WebSockets or chunked responses instead of clean REST calls.

muchael |root |parent [-]

Right now libretto only captures HTTP requests, which the coding agent can use to determine how to perform the automation.

For more complex cases where libretto can't validate that the network approach would produce the right data (like sites that rely on WebSockets or heavy client-side logic) it falls back to using the DOM with playwright

etwigg |next |previous [-]

Thanks for this! We have clear answers for things that are 100% and 0% automated, but it’s always that 80%-99% automated slice where the frontier is, great idea.

canarias_mate |root |parent [-]

script maintenance is exactly where that middle slice bites - the app keeps evolving and the scripts lag behind. we took the angle of having the agent re-explore from scratch each run with autonoma (https://github.com/autonoma-ai/autonoma) for e2e qa, no maintained scripts, adapts naturally - different goal than libretto but same core intuition

messh |next |previous [-]

how does it differ from playwright-cli?

muchael |root |parent [-]

At its core, libretto generates, validates, and helps with debugging RPA scripts. As far as I understand tools like playwright CLI are more focused on letting your agent use playwright to perform one-off automations.

The implementation is also pretty different:

- libretto gives your agent a single exec tool (instead of different tools for each action) so it can write arbitrary playwright/javascript and is more context efficient

- Also we gave libretto instructions on bot detection avoidance so that it will prefer using network requests for automation (something that other tools don’t support), but will fall back to playwright if it identifies network requests as too risky

seagull |next |previous [-]

I've wanted something like this for ages, excited to try this out!

daveguy |next |previous [-]

What is the license?

Edit: nevermind. I see from the website it is MIT. Probably should add a COPYING.md or LICENSE.md to the repository itself.

gbibas |next |previous [-]

Cool. Thank you for sharing. While AI tools are extremely powerful, packages like this help create some good standards and stepping stones for connectivity that the models haven’t gotten around to yet. Thanks again.

arpadav |next |previous [-]

this looks awesome

devstatic |next |previous [-]

this is interesting

raffaeleg |next |previous [-]

[dead]

KaiShips |next |previous [-]

[dead]

secureotter |next |previous [-]

[dead]

|next |previous [-]

surgical_fire |previous [-]

For a moment I throught it was something about libretro. As an avid RetroArch user the headline picked my interest.

Then I clicked and realized it's just some other AI shit.

muchael |root |parent [-]

Lol sorry for the misleading click. We named it libretto after the term in theater, inspired by Playwright. No retro gaming here, just browser automation!