[–] TylerJewell 1mo ago ↗ We created a self-improving, spec-driven delivery harness to test whether frontier models can port complete systems. [–] TylerJewell 1mo ago ↗ There were a few interesting findings:1. We were able to execute complete, unattended rewrites of existing systems that pass the original's unit and integration tests.2. Low effort models were more efficient than high effort models.3. The delivery harness and its structure had a bigger impact on consistency than model capability.4. We saw a LOC or performance improvement on 57 of 65 ports.
[–] TylerJewell 1mo ago ↗ There were a few interesting findings:1. We were able to execute complete, unattended rewrites of existing systems that pass the original's unit and integration tests.2. Low effort models were more efficient than high effort models.3. The delivery harness and its structure had a bigger impact on consistency than model capability.4. We saw a LOC or performance improvement on 57 of 65 ports.
2 comments
[ 4.2 ms ] story [ 21.1 ms ] thread1. We were able to execute complete, unattended rewrites of existing systems that pass the original's unit and integration tests.
2. Low effort models were more efficient than high effort models.
3. The delivery harness and its structure had a bigger impact on consistency than model capability.
4. We saw a LOC or performance improvement on 57 of 65 ports.