Hacker News
Test/verification technique effectiveness with Astra
reedf1
|next
[-]
kooi
|next
|previous
[-]
Makes me think that LLM assisted formal requirements generation for a roll your own "Aristotle" is possible.
Just thinking about our own projects in the signal processing domain. A nice path could be: human language -> temporal logic specs for black and white box run-time testing.
There are code generators which will turn formal logic into a run time monitor. In the past, I've never used them past toy examples because its so verbose.
> The point of this post is to demonstrate that it’s possible in the era of AI coding to still make decisions based on evidence rather than vibes.
That's the difference between an engineer and a script kiddie.
Domain expertise will always be valuable.
59nadir
|root
|parent
|next
[-]
A good motivation to stray from the conventional (non-)wisdom of using Lean as a solution as well, particularly because it's in many cases worse than much less rigorous methods for correctness, and worse than doing literally nothing special at all.
vessenes
|next
|previous
[-]
His general test methods this year have been to provide a specification document of some sort and then prompt a clean room implementation of the spec, which can be tested against holdout tests.
Things move pretty quickly - this summer, his test was Zstd and results were at like 25-30% of tests passing at best. He now considers Zstd saturated, as is IMAP from RFC; pandoc is one that he's used throughout, and is not yet perfect.
With all that in mind, read away.
__mharrison__
|next
|previous
[-]
Otherwise, I love this. I was pretty involved in Python testing circles (there was an old Python in tears mailing lists,) and would love to find similar for agentic testing.
ArtRichards
|next
|previous
[-]
jiehong
|next
|previous
[-]
Danluu often says in his posts that LLM are not very good at good testing strategies, and also that most people are bad at it too.
But, if true, what are the know better ways?
scotty79
|next
|previous
[-]
polskibus
|next
|previous
[-]
scotty79
|next
|previous
[-]
I'd love to see more of that. I'm currently defaulting to Rust in my software building but F# looks super strong there. On par with Python for correctness with Ultra effort.
I wonder how Rust and other languages are doing in Astra 6/Sol 6.1 and on more projects.