Amazing Post! Got intrigued when I read markdown because I’m currently working on a project related to Markdown and I need to understand the agents thing!
Interesting approach, I can see the benefits of such approach where the md is acting like a Jupyter notebook of some sort. You place the content with documentation and it becomes a self standing unit of test.
I wonder if this can be replicated to other notions where the docs/data needed for test add more value to live with the test
TAP is not really about writing tests, it's more about making sure that a system under test emits plain text logs that can easily be consumed/validated
The markdown based tests described is more of an analog to gerken, robotframework, etc.
there's a number of similar efforts. turning english-like specs for a test into executable tests has seens lots of variants over the years but LLMs have reaaly stepped up the possibilities
Anybody remember "cram"? From about 10 years ago, https://bitheap.org/cram/ basically a markdown syntax (making heavy use of code-blocks) for documenting and writing tests as "shell commands and expected output" (with a bunch of the sharp edges filed off, like line endings and partial matches.) Was particularly good for easy-to-write, easy-to-review tests of unix utilities. (It's the kind of thing that you only stumble on if you've been working with doctests but they don't really fit well for shell/unix stuff...)
9 comments
[ 3.1 ms ] story [ 35.5 ms ] threadI wonder if this can be replicated to other notions where the docs/data needed for test add more value to live with the test
I remember using it to implement the test suite for a Shakespeare language interpreter. Fun times.
https://en.wikipedia.org/wiki/Test_Anything_Protocol
The markdown based tests described is more of an analog to gerken, robotframework, etc.
there's a number of similar efforts. turning english-like specs for a test into executable tests has seens lots of variants over the years but LLMs have reaaly stepped up the possibilities
PS: I am currently working on Voiden.
https://repo.autonoma.ca/repo/keenquotes/tree/HEAD/src/test/...
Am curious whether SOTA LLMs can curl the ambiguous cases.