1. Create something that monitors an app's network connections
2. Use the data to auto-generate integration tests for every endpoint including edge cases, error states, etc
3. Have user validate and accept the integration tests
(end there for self-serve tier, and for the enterprise tier...)
4. Rewrite app into another language, framework, or simply refactor code to be cleaner, more consistent, convert callbacks to promises or async/await, etc.
... key step being #3, need a really good test suite to validate the new code works the same as the old.
Funny how it's all going William-Gibson-Sprawl-shaped. The best defensive ice (Intrusion Countermeasures Electronic) was written by AIs who constantly updated it, and the best icebreakers were also written by AI.
I'd be very surprised if secretive agencies within the USA, China, Israel, Russia and others weren't going full-bore on the offensive side (which OpenAI says this particular grant program is not involved in, at least, but I wonder what the black-budget NSA-TAO side of things looks like at present).
Indeed any AI defense capabilities will necessarily need to include some offensive abilities, which for the public are mostly censored right now (eg try asking for any of the obvious public PoCs which it almost certainly was trained on) I find it to be one of the worst aspects of the current alignment….
I don’t think it’s a serious limitation though, since the open-source LLMs like Alpaca/LLaMA have already “won” the mass adoption among engineers, and are only getting better (already better than GPT4 in some areas).
I thought this would be about a bug bounty program, which it isn't, but it did lead me to check if there is an OpenAI bug bounty program, which there is: https://bugcrowd.com/openai
For areas that require a great deal of confidence in conclusions, like forensics and incident triage, I can only see generative AI making matters worse. There’s already a ton of over-automation and humans making mistakes because they trust their tools too much. Giving that industry access to more tools that confidently give wrong answers almost seems irresponsible.
I am excited for AI-based threat intelligence products, though.
I could see the exact opposite happening. Forensics and IR often require a ton of complex information parsing that an LLM could accomplish very quickly and then point an analyst in the right direction to verify. It's not meant to be a panacea just an accelerant.
Threat intel has purported to use AI for ages and I remain unconvinced of its effectiveness. Program analysis to support TI could be exciting however.
Agree, but having the LLM explain how it arrived at the output will be crucial for provenance and anti-hallucination purposes. Judges don’t care for fabricated evidence or citations.
This language bothers me deeply. So many people ask the system itself to describe what it is doing, for which it is absolutely incapable of doing.
The system has no access to its internals.
Even academics are asking the LLM itself to describe what it's doing like idiots. I know that academia's intellect is slipping due to allowing too many people in, but this type of stupidity is just mind boggling.
I like to preserve skepticism here as well, though probably blaming them won’t help the issue.
But absolutely agreed, just listened to Gary Marcus’s podcast episode on how LLMs are already capable of making dad jokes, but they can’t do neither “deep” humor generation nor “deep” humor explanation very well, in part because they lack all of the cultural and experiential (“touchy-feely”) context of the humans, and an even bigger part since they are optimized for generation, not explanation, while humans are almost the opposite.
I think you misunderstood my comment or I didn’t articulate the idea clearly. I’m simply arguing for a verbose or debug mode that can be toggled on to show how output was derived, not expecting an LLM to have awareness or understanding of itself or it’s internal workings.
There’s traceable.ai, for one (not affiliated, but testing it). I’d say none of traditional ML products are very good, at all (at least among the ones available publicly). Most of them are just snake oil. But I’d err on the side of plausibility that LLMs might finally add fuel to that, at least on the pentesting side (might, not yet will).
Additionally, even OpenAI’s own advertisement “paper” acknowledges they couldn’t make it work for pentesting well (hence probably why they created this bounty program, among other reasons).
FWIW there’s this paper that just came out, and promises a low hallucination factor (not quite for pentesting, but this could be a useful starting point):
This is clearly a play for getting access to organizations' logs for them to train their models on - probably in preparation for a future cyber related offering.
No. If it were ranked they would have numbers next to them. It's too bad they can't add numbers without people assuming it's a ranked list because they would be useful to point to them.
Yup. Defense is mostly technical nuance. Attacks outside of academia often rely on the weakest links, distracted or scared humans. Chatgpt and LLMs like it can easily mass phish or otherwise thousands of targets in a matter of seconds. Can also be used for OSINT on human targets... Combine the two and ransomware will be a true cottage industry ran by script kiddies.
That said there are things they could do to prevent this. They won't do them, it'd be costly research... But they could.
We don’t seem to be quite there yet, but things are moving in that direction, unfortunately (see the Gorilla paper, for example). I still think to tailor the tool to each particular API requires human intelligence (and even that is hard), but a shotgun approach might be more “successful” even if it’s 5% effective.
That said, more traditional approaches (non-ML) are probably more effective still.
What about AI security itself, everyone is afraid of rogue AI, but a "dead" AI or "locked up" AI can't do harm. I.e. Have some sort of monitoring system to monitor AI systems, and a sort of kill switch that can disable it from the internet, or if need be trigger a small onboard EMP device to fully nuke the device.
It could give some safety if skynet becomes a thing, of course super AI will learn to disable it in a week, but still maybe there's a path there to mitigation of future threats?
39 comments
[ 0.24 ms ] story [ 107 ms ] thread1. Create something that monitors an app's network connections
2. Use the data to auto-generate integration tests for every endpoint including edge cases, error states, etc
3. Have user validate and accept the integration tests
(end there for self-serve tier, and for the enterprise tier...)
4. Rewrite app into another language, framework, or simply refactor code to be cleaner, more consistent, convert callbacks to promises or async/await, etc.
... key step being #3, need a really good test suite to validate the new code works the same as the old.
Cybersecurity is certainly one of the near term (now?) AI related risks.
This should help encourage some defensive progress.
I'd be very surprised if secretive agencies within the USA, China, Israel, Russia and others weren't going full-bore on the offensive side (which OpenAI says this particular grant program is not involved in, at least, but I wonder what the black-budget NSA-TAO side of things looks like at present).
https://simonwillison.net/2023/May/2/prompt-injection-explai...
I am excited for AI-based threat intelligence products, though.
Threat intel has purported to use AI for ages and I remain unconvinced of its effectiveness. Program analysis to support TI could be exciting however.
Even academics are asking the LLM itself to describe what it's doing like idiots. I know that academia's intellect is slipping due to allowing too many people in, but this type of stupidity is just mind boggling.
But absolutely agreed, just listened to Gary Marcus’s podcast episode on how LLMs are already capable of making dad jokes, but they can’t do neither “deep” humor generation nor “deep” humor explanation very well, in part because they lack all of the cultural and experiential (“touchy-feely”) context of the humans, and an even bigger part since they are optimized for generation, not explanation, while humans are almost the opposite.
IMO encoder models are far more powerful and useful for devs than any of the decoder models.
Have examples of existing product or project in this space that are interesting to you?
Additionally, even OpenAI’s own advertisement “paper” acknowledges they couldn’t make it work for pentesting well (hence probably why they created this bounty program, among other reasons).
“Gorilla: Large Language Model Connected with Massive APIs” https://arxiv.org/abs/2305.15334
Semi underground stuff like Phrack, 2600 and various punky groups who surprisingly did good security research and open to participation..
Today's cutting edge security research feels very silod in the products.
OWASP Top 10 for Large Language Model Applications
https://owasp.org/www-project-top-10-for-large-language-mode...
The draft of the Top 10
https://owasp.org/www-project-top-10-for-large-language-mode...
Does "increments of $10k" mean that a single grant can be greater than $10k? Or are grants limited to that amount?
That said there are things they could do to prevent this. They won't do them, it'd be costly research... But they could.
That said, more traditional approaches (non-ML) are probably more effective still.
It could give some safety if skynet becomes a thing, of course super AI will learn to disable it in a week, but still maybe there's a path there to mitigation of future threats?