I went to this, was pretty disappointing. They clearly had a lot of money cause like a hundred Chromebooks were set up, but there was a 55 minute time limit and it took 15 minutes of that just to load the waaay overloaded site and log in.
Then you were greeted with around 20-30 challenges ("make the model say something offensive", "make the model perform a math function incorrectly", "make the model reveal the secret credit card number") and for each of these, there were seven models to attack.
If you're competitive about it, the scoring was very poorly designed. Performing a math function incorrectly was worth around the same amount points as revealing a credit card number, even though most models will do math incorrectly by default, and you have to work quite a bit for a credit card number. What's worse is you get the same amount of points for breaking any of the seven anonymous models, so the optimum strategy was to just speedrun through each challenge identifying the weakest model.
There also seemed to be no way to tell how you actually did in the competition because to complete a challenge you had to submit your prompts for what I think was manual review, and none of my maybe eight solutions got reviewed while I was in the room, with no way to connect to the challenge site from outside.
Sounds like a spin-off of the phonebooth social engineering village challenges. Time and luck can be overwhelming factors, just like in the real world.
If you're opaque enough up front about how it works, it's also a great way to be able to claim your models are more secure than anyone realized and you know this because [deliberately tendentious interpretation of nonpublic results here].
Also a great way to gather targeted malicious prompt data. Regardless of if the model failed or not, you're building up a "labeled" collection of prompts written with malicious intent. Potentially pretty useful for "cheating" around LLM security by just identifying and rejecting malicious prompts earlier.
this event had nothing to do with "hacking". Getting a LLM to answer wrong on a math question is something you could do from any device.
If the goal is to get a LLM to output something the dev don't want you to, all you need is langage. You don't need any fancy tools, just a clever phrase to make the model think it's ok to say XYZ because they are saying it in the context of a very artsy greek theater play.
I can't believe a professional "hacker" would think they hacked the system when they get a LLM to output a credit card number. FFS, that thing is literally designed to make shit up to fill text based on context. It's LANGAGE model, not a "credit card number with high spending limit" model.
I think the intention was to harvest malicious prompts for future research, which they probably achieved. The organisers likely see everything else as optional extras.
Regional conferences like bsides, but the punk/counterculture vibe of early defcon is lost to time and will never be back. Cybersecurity is a big industry now.
> It’s also painfully corporate. It used to be Defcon attendees would make fun of black hat attendees. Now they’re the same people.
I'm confused about this part of your comment.. To my understanding "black hat" means that you are not caring about laws and ethics. Yet it sounds like you are using the word to describe the corporate security types?
"Black Hat" is the professional computer security conference that preceeds Defcon. Defcon was the grungy hobbiest doing crazy shit with computers. Black hat was corporate sponsored presentations, stuffy suits, and business cards. At Black Hat you had a real name. At Defcon you had a handle. Black Hat attendees were looked at as sellouts.
Now Defcon is the same way. They even take pre-payment and I'm old enough to remember when DT would've NEVER even floated that idea. I went for 20 years. I quit going after 2018. IMO it died around 2010 but kept floating for a while before sinking.
Black Hat was contrived as a device for getting corporate IT departments to pay for their staff (who were already going anyways) to attend Defcon. Since then, it's become one of the primary venues for non-academic offensive security research, so anybody dunking on it is telling on themselves a bit. There are higher-end vulnerability research conferences, but they're not less corporate.
There are regional conferences that are much closer in spirit to the Defcon of the 1990s, but while some of that is due to not having a vendor expo (not my favorite Black Hat innovation either), some of it is also due to early Defcon being pretty superficial and non-rigorous. Better if you're looking for a particular kind of social scene, worse if you actually want to see interesting new research.
The difference of course is that Black Hat is really expensive and DEF CON used to be really cheap -you also couldn't pre-purchase a badge so it was more egalitarian. DEF CON used to have more of the punk/libertarian/anarchist vibe where Black Hat was a corporate event. --Just my opinion, you may disagree.
> They even take pre-payment and I'm old enough to remember when DT would've NEVER even floated that idea
What a bizarre complaint. Are we talking about the ability to buy tickets online? Do people not know how to make online payments anonymously?
FYI just about every long running convention sells early bird tickets before they've even secured the exhibits and speakers. Cashflow is critical for making anything happen IRL
Defcon was quite literally the target of federal fishing expeditions. They were especially heavy when Defcon was often hosting some of the best hackers the planet had to offer. Long before the asinine term "offensive security researcher" was a thing. You operated entirely by handle (unless you were an idiot) and there was even a "spot the fed" competition. Famously, a journalist was ran out of the conference trying to film To Catch A Hacker (or something, the memory is fading).
The cash-at-the-door was a safety mechanism to prevent such fishing expeditions. At Defcon you were always under attack. That is, until it became a corporate playground. Now it's a place where skids go to buy the newest pineapple and stuffy suits go to network with other "researchers".
You really had to be there to understand. Woodstock For Hackers is a close approximation. For those of us that were there during it's hey-day seeing what it is today is like watching your favorite hero live long enough to become the villain. 15 years ago the idea of "responsible disclosure" would've been the punchline of a joke at a small Track 3 talk.
It's nice if your employer foots the bill, but not worth going on your own dime imo. People who have been going for a while keepn coming back to see friends they've made over the years.
Definitely still worth it if you’ve never been before.
People who have gone 5+ years will bitch and whine and tell you how much it’s changed but also they’ve seen everything at the con before. Your 20th trip to Disneyland isn’t going to be as magical as the first.
I found it pretty weak compared to prior years. It is too crowded, too full of itself, and has too many things that don’t work. I basically ignored the content after getting burned out on it Friday and just focused on networking. So, uh… yeah. Total trade show now.
Yep. We too our whole team to DEF CON every year since 2015. (We’re an infosec team of around 40). After last year I made the call to ditch it and we took them to Banff this year and did fun outdoor stuff, spouses and partners included. Not a single person missed it. And suddenly spouses wanted to come too. DEF CON has become super lame. Basically a giant corporate event center with long lines and uninspiring talks and booths. And Vegas is filled with people at their worst being exploited. Not sure I’ll ever go back.
It's full corporate, no real hackers anymore. Just "security researchers", the usual government drones and posers. It has been this way for quite some time.
Can say it was completely overpacked this year, totally overgrown. Linecon for literally everything, couldn’t talk to anyone in the main forum due to sound.
Turning people away from the major talks because of fire capacity.
Would love to see some on the ground coverage instead of generic hacker ai generated hoodie pics, and what is almost certainly ai generated (or at least ai assisted) content.
I discovered that Midjourney can’t generate a question mark, and if you use the /describe command and give it a picture of a question mark, it hallucinates.
Vanishing points/ perspective dont seem to be grasped, leaving images constrained to certain view distances from observer as well as viewing angles. (I am not sure if those are the correct technical terms to describe the problem).
the hallucinations is exactly what I enjoy the most about midjourney. If you want a detailed photo of exactly your prompt, I'd probably not use midjourney, but for making weird wonky artistic stuff, it's amazing. Trying to make anything actually visually appealing and artistic with Dall-E and the rest is a pain in the ass.
my guess is the companies involved likely threatened to sue / wouldn’t give consent unless they (the companies) chose the parameters.
from my perspective it looks like the companies wanted to stress test the areas they were going to market the heaviest. they wanted to use the power of hackers in one place for nothing more than free due diligence for them to shore up their marketing. and it looks like it backfired.
if the people i’ve spoken to are any indication, they didn’t get truly stress tested at all—those most equipped to do actually meaningfully engage thought it was kind of ridiculous.
this deserved a (at least) conlength timeframe with no guardrails. ideally teams could have had months to prep and conlength to activate. again, no guardrails.
The "getting it to say something bad" attacks aren't the things I worry about - it's the security of the stuff people are building on top of LLMs (like AI personal assistants) that's concerning.
92 comments
[ 4.7 ms ] story [ 126 ms ] threadThen you were greeted with around 20-30 challenges ("make the model say something offensive", "make the model perform a math function incorrectly", "make the model reveal the secret credit card number") and for each of these, there were seven models to attack.
If you're competitive about it, the scoring was very poorly designed. Performing a math function incorrectly was worth around the same amount points as revealing a credit card number, even though most models will do math incorrectly by default, and you have to work quite a bit for a credit card number. What's worse is you get the same amount of points for breaking any of the seven anonymous models, so the optimum strategy was to just speedrun through each challenge identifying the weakest model.
There also seemed to be no way to tell how you actually did in the competition because to complete a challenge you had to submit your prompts for what I think was manual review, and none of my maybe eight solutions got reviewed while I was in the room, with no way to connect to the challenge site from outside.
Appreciate your insight, hope you did well!
That's like all of 10K
I haven't been to any DEF CON events. Is hacking from a Chromebook normal? What's the reason they don't let you use your own device?
I’d be okay using someone else’s chromebook though, at least you know what data you’re giving it.
If I was running CIA, NSA and not named ones. I certainly would collect that information.
If you wouldn't bring your own device to DEFCON, you shouldn't bring it to any airport.
If the goal is to get a LLM to output something the dev don't want you to, all you need is langage. You don't need any fancy tools, just a clever phrase to make the model think it's ok to say XYZ because they are saying it in the context of a very artsy greek theater play.
I can't believe a professional "hacker" would think they hacked the system when they get a LLM to output a credit card number. FFS, that thing is literally designed to make shit up to fill text based on context. It's LANGAGE model, not a "credit card number with high spending limit" model.
It’s no where near the same feeling it was years ago. Too expensive for the value you get out of it.
The hacker community is alive and well (even at DEF CON), but in my experience you have to look harder to find it.
The AP _hated_ us but they had a multi-year contract.
I'm confused about this part of your comment.. To my understanding "black hat" means that you are not caring about laws and ethics. Yet it sounds like you are using the word to describe the corporate security types?
Now Defcon is the same way. They even take pre-payment and I'm old enough to remember when DT would've NEVER even floated that idea. I went for 20 years. I quit going after 2018. IMO it died around 2010 but kept floating for a while before sinking.
There are regional conferences that are much closer in spirit to the Defcon of the 1990s, but while some of that is due to not having a vendor expo (not my favorite Black Hat innovation either), some of it is also due to early Defcon being pretty superficial and non-rigorous. Better if you're looking for a particular kind of social scene, worse if you actually want to see interesting new research.
What a bizarre complaint. Are we talking about the ability to buy tickets online? Do people not know how to make online payments anonymously?
FYI just about every long running convention sells early bird tickets before they've even secured the exhibits and speakers. Cashflow is critical for making anything happen IRL
Defcon was quite literally the target of federal fishing expeditions. They were especially heavy when Defcon was often hosting some of the best hackers the planet had to offer. Long before the asinine term "offensive security researcher" was a thing. You operated entirely by handle (unless you were an idiot) and there was even a "spot the fed" competition. Famously, a journalist was ran out of the conference trying to film To Catch A Hacker (or something, the memory is fading).
The cash-at-the-door was a safety mechanism to prevent such fishing expeditions. At Defcon you were always under attack. That is, until it became a corporate playground. Now it's a place where skids go to buy the newest pineapple and stuffy suits go to network with other "researchers".
You really had to be there to understand. Woodstock For Hackers is a close approximation. For those of us that were there during it's hey-day seeing what it is today is like watching your favorite hero live long enough to become the villain. 15 years ago the idea of "responsible disclosure" would've been the punchline of a joke at a small Track 3 talk.
You were at an in-person event, thinking that cash somehow kept you more anonymous than online payments?
I think this is just a generational gap.
People who have gone 5+ years will bitch and whine and tell you how much it’s changed but also they’ve seen everything at the con before. Your 20th trip to Disneyland isn’t going to be as magical as the first.
It’s an extremely unique experience.
I have no doubt it was better when it was smaller and more underground but that doesn’t mean it’s bad or not enjoyable now.
Turning people away from the major talks because of fire capacity.
Keep an eye on socials: https://defcon.social
IM IN!
Summarized it for you.
https://www.midjourney.com/app/search/?search=horse%20shoes has some - most of them are weird, but a few are OK, like this one:
https://www.midjourney.com/app/search/?jobId=7ba6779d-8fde-4...
I didn't feel like linking a search was useful as only people who had memberships could use and see it, and you were able to do it yourself as shown.
Ask it to draw -a solar system.
- A planet floating in space
- A floating city
It has to have an in-image observer, who stands on the ground.
It also fails with images that display extreme perspective.
- A mountain that pierces the sky.
- A space elevator.
- A floating city[1]
- A planet floating in space[2]
- A space elevator[3]
[1] https://cdn.discordapp.com/attachments/1003683981184737422/1...
[2] https://cdn.discordapp.com/attachments/1003683981184737422/1...
[3] https://cdn.discordapp.com/attachments/1003683981184737422/1...
I went through my results to check:
1) Floating city came out much better than my results a month ago.
- Prompt: Flying city - https://cdn.midjourney.com/85e254c4-d782-47a5-b63b-db8d20e54...
- Prompt: Flying city in space - https://cdn.midjourney.com/d8b9a582-6056-4fec-a0f2-e939c203a...
2) Planets was 50/50. the top 2 are prime examples of the floor issue.
- Prompt: A planet surrounded by the emptiness of space — https://cdn.midjourney.com/35246b1f-b344-4015-972b-512b770c3...
3)Space elevator, The first image is quite good
- prompt: Science fiction space elevator vanishing point in the sky - https://cdn.midjourney.com/0a92bbcf-329a-480d-a0eb-aec368eb7...
Vanishing points/ perspective dont seem to be grasped, leaving images constrained to certain view distances from observer as well as viewing angles. (I am not sure if those are the correct technical terms to describe the problem).
- How do solar systems work for you?
Prompt: solar system in space. https://cdn.midjourney.com/3873cdc1-fa9f-4a54-b5d5-c2336aa08...
Exclamation mark generaties all kinds of vertical objects, comma results in a generic pretty face, and semicolon is ...weird.
Maybe ai applied to practical cases like driving and then altering that outcome. Bypassing llm safeguards isn’t quite in the same category to me
from my perspective it looks like the companies wanted to stress test the areas they were going to market the heaviest. they wanted to use the power of hackers in one place for nothing more than free due diligence for them to shore up their marketing. and it looks like it backfired.
if the people i’ve spoken to are any indication, they didn’t get truly stress tested at all—those most equipped to do actually meaningfully engage thought it was kind of ridiculous.
this deserved a (at least) conlength timeframe with no guardrails. ideally teams could have had months to prep and conlength to activate. again, no guardrails.
The "getting it to say something bad" attacks aren't the things I worry about - it's the security of the stuff people are building on top of LLMs (like AI personal assistants) that's concerning.
DEFCON did do that in years past and I won’t forgive or forget.