The readme has a lot of words but still does not clearly explain what this does or what you'd use it for.
The best I can gather is that this captures telemetry while you're running some computationally-expensive job, and checks that the resource usage roughly matches what it's supposed to be. But what does "Tamper-evident" mean? I assume you sign with your own private key? What exactly prevents the data from being faked?
It didn't sound very convincing. It doesn't really prove anything other than the author signed the report. Unless the auditing is running from a secure enclave/core that the user has no access to, it can't do what is claimed on the tin.
There alternatives to a secure enclave like building in a risc0 VM that produces a mathematical proof of each computational step that is very expensive to generate and very cheap to verify. THAT is proof.
But this seems to just be collecting evidence and signing it?
> "A run of committed code, on attested hardware at time T, reading a model with hash m and a dataset with hash d, produced score S.” If d is a public benchmark, they also confirm it is the real benchmark, without you exposing it. The score is also cross-checked by the compute-cost certificate, which confirms the evaluation consumed the compute a real forward pass over the data requires.
They should really put that front and center. Still not a complete enough description, but something at least.
This is exactly what it is, just slop. I do agree with the intent we need someway to verify and validate agentic coding output that doesn't rely on codereviews; but this project is just sloppy.
Correct me if I am wrong
You are not trying to prove scientific correctness; You are just proposing a way to know that X code(s) produced Y claim(s),not verifying how accurate are X and Y?
Have you tested it on clusters? Most experiments are ran on distributed systems
I have tested it with c++, it had few problems when I combined the compile and execution commands. but it worked when i specify sh -c.
8 comments
[ 0.22 ms ] story [ 5.7 ms ] threadThe best I can gather is that this captures telemetry while you're running some computationally-expensive job, and checks that the resource usage roughly matches what it's supposed to be. But what does "Tamper-evident" mean? I assume you sign with your own private key? What exactly prevents the data from being faked?
But this seems to just be collecting evidence and signing it?
https://kveritas.org/docs/benchmarks - What a verifier learns:
> "A run of committed code, on attested hardware at time T, reading a model with hash m and a dataset with hash d, produced score S.” If d is a public benchmark, they also confirm it is the real benchmark, without you exposing it. The score is also cross-checked by the compute-cost certificate, which confirms the evaluation consumed the compute a real forward pass over the data requires.
They should really put that front and center. Still not a complete enough description, but something at least.
Punchy phrase. It does A, never B. It Xs, Ys and Zs.
These are the simple tells but the general tell is what you describe - a lot of words that say nothing.
What are the scopes of this project?
Have you tested it on clusters? Most experiments are ran on distributed systems I have tested it with c++, it had few problems when I combined the compile and execution commands. but it worked when i specify sh -c.
i might have more questions after testing it more