The difference is accuracy, cost, time and more importantly consistency.
In case you noticed, a year ago, LLMs could not reliably count the number of 'R's in strawberry. Now they all do well. Guess how? Instead of training LLMs to do this, it was easier for them to write a small python script and run that. That solved the problem once and for all.
The same thing here, instead of just using LLMs and keep grilling repeatedly, there is a higher chance of getting a workable solution quickly. The best part about formal methods here is, it is self validating.
It checks in seconds, what would have taken hours or even days with LLM only flow.
The overlapping part of what FizzBee and Kiro is we formalize the requirements first and do formal analysis on them to identify various requirements issues.
However the technique is significantly different. Kiro's post says they use predicate logic. Whereas FizzBee uses Dynamic Logic. So, Kiro's approach cannot find many issues. Let us take the same example from the fizzbee blog.
For example if the requirements are,
```
R1: THE SYSTEM SHALL allow the stylist to set their work schedule.
R2: WHEN a customer requests a slot within the stylist's work schedule,
THE SYSTEM SHALL book the appointment.
R2b: IF a customer requests a slot that is not within the stylist's work schedule,
THEN THE SYSTEM SHALL reject the request.
R3: THE SYSTEM SHALL ensure that all appointments are within the stylist's schedule.
```
Kiro's approach would say, it is both consistent and complete. That is, R2 + R2b => R3 in this case.
But FizzBee found the issue as linked in the blog:
The related https://fizzbee.ai/ tool is pretty neat. It's similar to /grillme but with additional formalism. Not sure if the resulting specs are definitively better, if only because the FizzBee code is harder for me to decipher.
11 comments
[ 7.2 ms ] story [ 44.3 ms ] threadThey typically capture the requirements, design, and implementation plan in Markdown files. But is that Markdown file actually a specification?
This article explores how formal analysis can uncover requirements gaps that are easy to miss.
I'm not sure if this does much more than a grillme skill and then poking an agent to do the work.
In case you noticed, a year ago, LLMs could not reliably count the number of 'R's in strawberry. Now they all do well. Guess how? Instead of training LLMs to do this, it was easier for them to write a small python script and run that. That solved the problem once and for all.
The same thing here, instead of just using LLMs and keep grilling repeatedly, there is a higher chance of getting a workable solution quickly. The best part about formal methods here is, it is self validating. It checks in seconds, what would have taken hours or even days with LLM only flow.
However the technique is significantly different. Kiro's post says they use predicate logic. Whereas FizzBee uses Dynamic Logic. So, Kiro's approach cannot find many issues. Let us take the same example from the fizzbee blog. For example if the requirements are, ``` R1: THE SYSTEM SHALL allow the stylist to set their work schedule. R2: WHEN a customer requests a slot within the stylist's work schedule, THE SYSTEM SHALL book the appointment. R2b: IF a customer requests a slot that is not within the stylist's work schedule, THEN THE SYSTEM SHALL reject the request. R3: THE SYSTEM SHALL ensure that all appointments are within the stylist's schedule. ``` Kiro's approach would say, it is both consistent and complete. That is, R2 + R2b => R3 in this case.
But FizzBee found the issue as linked in the blog:
https://blog.fizzbee.ai/formal-analysis-in-requirements-spec...