Stories like this might appear to hurt Facebook and similar companies, but they actually help create a barrier to entry for the "next Facebook" to not be able to use the same strategy.
If my writing makes it less likely that some would-be Facebook competitor can use mass-scale psychological manipulation to get an edge, I can live with that.
Is that necessarily a bad thing? I feel like social media has destroyed an enormous amount of value by making it harder to have authentic relationships.
If you can name a successful product which does not manipulate users to continue using that product, then it's not a product:
* I've been manipulated to bring an umbrella because I don't like to get rained on.
* I've been manipulated into liking broadband because it provides the internet faster than dialup.
* I've been manipulated into owning a car because it's a fast convenient mode of transportation.
This is the basics of products and services that provide value, this isn't magical or new, and this book is probably being more honest than most industries are.
The point the book was making is that if you don't want a product to be flash in the pan garbage, follow some simple guidelines which ensure that your service is actually providing some sort of value to end users, or your end users will eventually leave and find better things to do with their time and resources (Unless you're selling heroin I guess).
I don't consider Facebook sending me an email when I get a personal message to be exploitative, but I find no value at all when they send me something every time one of my friends fart. I specifically avoid interacting with those "hey someone did something" emails because they are annoying and make me want to stop being on facebook altogether. That's the result of abusing the hook model, users go away because it's annoying and unproductive.
Manipulation had always been there but what's new is that the best minds of a generation are working on exploiting human weaknesses with the help of technology to a new level.
Same for surveillance. It was always there but now it can be done on an absolutely unprecedented level.
In standard / common usage, selling and marketing an umbrella to someone is not "manipulation".
Your definition of "manipulate" is not what most people mean in these contexts. Your use of "manipulate" in such a way muddies the waters and distracts from the core issue.
Putting word choice aside, the core issue here is _not_ about people's preferences (e.g. many people don't like to get rained on). Rather, the core issue is about unknown and unwelcome influences, particularly influences that operate on parts of our minds that are hard to combat due to human nature.
Facebook's manipulation of our attention is conceptually related to how casinos trick people into staying longer than they would otherwise.
You're missing the static vs dynamic distinction. Facebook can guess with eerie accuracy whether you're about to start a relationship with someone, whether you recently got pregnant, whether you're going through a depressive/manic phase...the list is endless. Facebook changes itself by taking advantage of this information. An umbrella can't do this.
I can't afford a butler, but if I could, then I'd get one. He'd live in my house and know all about me and my personal life; knowing about my personal life would help him do his job better. I wouldn't worry that my butler knew I was in a relationship, or pregnant, and that he changed his behaviour accordingly.
On the other hand, I suppose I wouldn't want a butler who sold information about me to third parties, or used the information to advertise to me for kickbacks. "You have been down recently, sir... perhaps a new Bentley would cheer you up?"
This exactly, the problem isn't the hook method which pretty much every service does to one degree or another, it's the fact that they are combining your personal information with the hook method, that's exploitative and terrible behavior which eventually is recognized as such.
The point though is that companies act to maximize 'user engagement', a goal which may be at odds with what users actually want to get out of a product or service. Whatever they can do to get just 5% more views or clicks or time on sight is 5% more ad revenue for their investors. Nevermind if that 5% comes from time I'd rather been spending elsewhere and will end up regretting. They incorporate design choices that cause this, whether by autoplaying the next video, implementing a continuous feed, all while controlling exactly how much content they need to show before you finally get sick of it. They are stealing the surplus the promised you.
And suppose you are of psychological steel. Should those who aren't still be exploited? There are entire industries and schools of design whose goal it is to extract as much money as possible from users, take slot machine design for example. (I'd definitely recommend looking up the work of Natasha Schull). Do you consider that practice immoral? Nir's existence is a testament to the fact that product designers use similar techniques
We didn't solve alcoholism by outlawing alcohol, we are only really recovering from that product by creating a culture of respect and social norms, which means we need to critically think about how we're spending our time and the signs of any manipulation as a part of becoming an adult.
Alcohol is outlawed in many contexts, though they vary from place to place. In some (all?) countries, likewise tobacco. These are combined with large scale government advertising campaigns that start in school, which only differ from propaganda in that I think they are true (but then, if they were fake and I’d fallen for it, how would I know? Lots of people in my original country fear the drug Ecstasy, but that government lied - and continues to lie - about the dangers, and the poster-girl of the anti-Ecstasy talks at school died of a water overdose that only happened because of previous, different, bad anti-drugs advice).
Also, there are many ways to fool people that still work even if the victim knows about them, and the pain of being told one has been fooled means every technique the victim knows can (and often will) be rattled off as a fully generic counter-argument to whoever tells them that they have been fooled.
I just want to be clear that society at large has instituted regulations like "You need to be 21 to drink" and "You cannot drink in a public park", but there are no regulations on how much you are allowed to drink, that's where personal responsibility to social norms comes into play.
So you're saying it's ok for the government to manipulate you, but not a company, because their goal isn't a vague idea of societal good, it's to make more money? I find it much easier to understand why a company does X, rather than why the age for drinking is set at 21. I take that understanding with me every time I sign up for any kind of service.
If anything should come out of any of this, it's that personal/private information should be fully controlled by individuals in the same way health information is handled. It shouldn't be so easy to generate huge databases of people and claim ownership over them, and it definitely shouldn't be so easy to use those social ties to create an addiction.
> So you're saying it's ok for the government to manipulate you, but not a company, because their goal isn't a vague idea of societal good, it's to make more money?
In the right area, sort of (manipulation generally bad but a few things can override the badness) but I would say that governments are (in theory) democratically accountable, whereas business are legally and financially accountable. Yes, people do vote for dumb laws, but in principle that can be undone by another vote, businesses only get in trouble with their shareholders if the government doesn’t limit their behaviour.
While I agree with the point that it would be better if it wasn’t “so easy to generate huge databases of people and claim ownership over them, and it definitely shouldn't be so easy to use those social ties to create an addiction“, I don’t have any long term solutions. At some point the collective footprint of someone’s public statements on the internet is such a database, albeit in the informal sense of the word and spread over dozens of unconnected businesses, and demonstrating that the information has been used to cause addiction is more difficult than demonstrating tax avoidance.
You haven't been manipulated into buying an umbrella at all, because deciding you'd rather stay dry than wet is a straightforward choice; a product that offers you that choice is a straightforward product which provides a tangible service.
To be manipulated, you'd have to exposed to a constant stream of ads telling you how wonderful umbrellas are and what an unattractive loser you are for not owning one; how desirable people with high status use exclusive umbrellas while most people get by under a wet newspaper; and how you should buy a new umbrella every year because they're constantly improving, and this year's umbrella already has more a billion users worldwide and offers 23% more anti-rain features thanks to the exclusive Umbrella Corporation[tm] lifestyle eco-system. (And now here's a video of attractive people smiling, and of a world-famous umbrella designer talking earnestly about engineering, design, and craftsmanship.)
You'd also see paid testimonials and likes on social media sites. They'd appear to be spontaneous and sincere praise, but in fact they'd be factory farmed and produced to order by unhappy people working in open plan offices paid so little they can barely afford the rent.
And you'd have an umbrella that only worked reliably some of the time, so you'd have to keep considering the purchase of an improved model - in the vain hope you could finally end your constant nagging frustration with umbrella technology.
Worst of all, you'd believe you were making a straightforward umbrella-buying decision entirely on your own initiative, using perfectly objective and rational criteria.
Has facebook done any of that? Did you come to facebook because of an ad, or was it because you wanted to keep in contact with your friends and family? The fact that they are trying to use those social ties to make you look at more ads is what's amoral, not the fact that they sent you an email that doesn't apply to you as a "hook".
The explicit description at one of the Facebook developer conferences I went to was to do things which caused users to stay for longer. I think the word they used was some pleasant-to-neutral term like “compelling content”, but the goal was to switch from “more users” to “longer stays” - and to use psychology to achieve that.
Then there is the games sector. I was only in that sector from 2007 to 2010, but when I left many of them were asking how to make their content “addictive”.
I’d say it’s immoral that Facebook et al have wasted subjective lifetimes of their users’ consciousness by addicting them to whichever drug the brain synthesises to reward the minor and unfulfilling social interactions that it provides (and the reverse, the pain from the fear of missing out).
> I've been manipulated into liking broadband because it provides the internet faster than dialup.
You weren't manipulated. Manipulating you into buying broadband would entail creating the artificial perception that broadband is faster. If a company produces a product that performs as it is supposed to, and it solves a problem you have that was not artificially created, then you're not being manipulated. Same for umbrellas. Not the same for cars: the auto industry has pretty clearly lobbied for policies that ensure sustained demand for cars as opposed to public transit.
In the meantime, users will have to judge the yet
unknown consequences for themselves, while creators
will have to live with the moral repercussions of how
they spend their professional lives.
I am glad to see more people caring about long-term effects of their use of social software. More of us are mindful in how we engage with technology.
Now that I've said the positive, let me talk about the negatives. I bet things will get worse before they get better. At present, the commercial pressures on social media platforms to manipulate "users" dwarf human pushback against these "dark art" techniques.
I don't see any easy solutions. As with many collective challenges, we need increased awareness to effect change. This is arguably a political problem; we need social organizing as a counterbalancing force.
We don't need "increased awareness", we need a social decision system (aka "an economy") that rewards positive social outcomes and disincentivises negative social outcomes.
At the moment we have the opposite. No one cares how you make your first billion or two, or how people suffer as a side effect. They only care that you have a billion or two. Toxic side-effects remain invisible and uncounted.
There's a certain irony in attacking EA for building a game economy that's closer to the effort/reward profile of the real economy - for both players and owners - than any other game economy to date.
First, a question for you: What values would you like your "social decision system" to embody?
Second, I advocate for increased awareness because awareness is the first step towards rational thought. Many psychological manipulation techniques work best on the irrational parts of our brains, so often a useful first step is to engage the rational mind. That said, I'm not claiming awareness guarantees systemic change.
Well to have that economy we need transparency (read: the open availability of essential data/information).
On top of that we need awareness that such tools exit.
And on top of both of those we need a universal belief that this "stack" is an enabler and that enpowerment can have impact; that the little guy/ gal do matter.
Currently, best I can tell, we're lacking quite a bit at all three levels.
From many angles there are too many things working to passively shore this edifice up.
The best solution I found in years was after randomly catching up with a good friend. He just gave up on the smart phone.
Hes not a Techie, he used to love video games - but he has a unique quality to be embody Human-ness. Hes dumped social networks, and dropped smart phones.
In short the way out for those who can manage it is cold turkey.
I now suspect that the people who can handle this, are not on HN or social networks anymore
My knee-jerk reaction to the book Hooked (which I've somewhat shamefully avoided reading out of a sort of premeditated disgust) was based on the general arguments laid out in this blog post.
I believe the tech world has a huge problem on its hands here, and that problem doesn't get anywhere near the attention it deserves. Large swaths of the population are turning into zombies and tech is the facilitator. Try looking at the drivers turning through a busy intersection some time. It's obscene how many are staring down at their phones rather than at whatever might be in the road around the corner.
That said, I don't believe the problem is simply that certain products remind or encourage people to use them at times. Rather, it's that there is a class of truly addictive products that don't do much (if anything) to help people manage the resulting dependency. And that approach is celebrated far more than it's scrutinized.
Coincidentally, I wrote the following on Twitter yesterday:
"Any potentially addictive software/service should offer a way for users to lock themselves out of their own account for a predetermined set of hours." [0]
It was meant in my usual tongue-in-cheek style, but I do think that a solution for breaking tech product dependency is sorely needed and hope this discussion gains some steam.
You might want to check out some of the criticism advanced by people such as Tristan Harris on the tension between the user behaviors that tech companies optimize their product design, and the goals users hope to accomplish through their use. I do in some ways support Nir in that we should learn how to harness these types of technologies for our own good, to 'supercharge' our ability to accomplish what we want, but I do not trust any company dependent on ad-revenue (or backed by Wall Street in general) to help with that.
I'd also recommend checking out The Attention Merchants by Tim Wu and Addiction by Design by Natasha Schull for background on the history of the commodification of attention and of the
"Any potentially addictive software/service should offer a way for users to lock themselves out of their own account for a predetermined set of hours."
It's really noticeable that Hacker News provides exactly this feature, and absolutely no other site that I've seen does.
For years I resorted to hacking /etc/hosts to block HN. Mentally, I just assumed that no website would have such settings so I never bothered to learn what noprocrast meant.
The idea of a hard block on tech is intuitive, but it doesn't do anything to undermine the habitual learning in tech addiction; it just frustrates it for a fixed amount of time.
Space focuses on slowly undermining the compulsions behind tech addiction. So that, over time, you don't have the impulse to open the app anymore.
I tried to work out how Space worked on iOS, before I downloaded it. Unfortunately, I don’t see anything in the app’s description or website that stops me from launching the Facebook app the over Space app.
This is a fairly similar argument to saying drug dealers are bad because they profit off your addiction. I don't know if I agree or disagree with that, but at what point do we actually acknowledge agency?
Drug dealers don't have intricate control over the effects of the drugs they're selling. I imagine if synthetic drugs were legal in general rather than specific compounds you might see similar problems pop up (e.g. 3-fluoroethamphetamine, a compound screened in rats to be maximally addictive) but it's reasonable to advocate for a drug market that avoids this.
I think Agency is always understood to be a part of the equation - and the people with the agency are not the issue.
By analogy - If the population of a city was exposed intentionally, to a cold virus - it is not the people with natural resistance, we are concerned about.
We are glad those people are not affected, and know they are not part of the % affected by the virus - and then we focus on the people affected by it.
---
If I were to engage with your point in the spirit it is intended, there are several issues -
1) Tech recruits the best of the best. IT pays for the smartest people in the world (and this is not hyperbole), to get past agency. This is not hard core drug dealer style sale of addiction. This is candy crush, happy smiley imagery, "just one more game", style behavioral addiction.
Most people don't even know they are being addicted/affected
2) This is all at internet scale. Just on a probabilistic level, the casualties/infection is going to vastly outperform personal agency.
3) The network effects of having more people doing this, at this scale, pushes it to crisis levels very quickly. It wouldn't matter what the few immune or capable can do.
The thin end of the wedge here is advertising. It is deliberately manipulating people using psychological insights, often targeting weaknesses to create a recurrent cash flow.
Yet the entertainment and content-creating side of the internet relies very explicitly on ad revenue.
Are creators that rely on revenue generated by manipulating human psychology morally bankrupt?
They invented a whole language to paper over the fact that their entire field is weaponized psychology employed against their customers. Do a noun for noun substitution to unpack a marketing meeting, and you'll want to puke.
Frankly, it's shameful for any company that says they care about their customers to employ marketing psychology tactics. Marketing and advertising are (in their modern, virulent form) simply an attempt to hack viewers to shift the locus of control from the customer to the business. (Other objectionable things about tech have this kind of behavior, too.)
Frankly, it's just disgusting and I can't blame customers for really viewing companies negatively based on their advertising practices.
(If all that wasn't bad enough, online advertising often comes with literal malware ontop.)
When you think about technology critics none seems more relevant today than Aldous Huxley.
The traditional Hollywood criticism on technology goes along the lines of Frankenstein, Metropolis, 2001 A Space Odyssey, Terminator and Matrix: technology will control us. I find it dumb, frankly.
Then there is the political criticism along the lines of George Orwell and some Luddites: technology is a tool for control and domination either by governments or evil capitalists. It is not totally wrong (e.g.: China, Turkey) but not the exact true sometimes.
But the Huxley criticism is, for me, the most interesting: technology will entertain us to stupidity, apathy and alienation. This seems to be very much the case in here.
We are in a society that has the distractions of Huxleys world with the surveliance of Orwell. The only thing different is the fact that no one expected our computers to get so small . That we would stare at screens that watch us every hour of every day. The anime psycho pass is much more relevant .
I think it's easy to get hung up on the means (that each authored). But as we know, the prediction business is a tough one.
For example, Orwell (in 1984) focused on the filtering, manipulation, constriction and reconstruction of information in order to continuouly redraw the history / truth. But today the truth is something that's often just as fuzzy because we're overwhelmed with too much information and too many distractions. Different means, same ends (read: confused and manipulated proles).
But it's the ends that matter. Both Huxley and Orwell have warned us. The irony is, at this point, too few are paying attention.
The truth has always been fuzzy/slippery. There never was a golden age of unbiased nothing-but-the-correct-truth journalism. The only change now is it's a little more obvious.
> The truth has always been fuzzy/slippery. There never was a golden age of unbiased nothing-but-the-correct-truth journalism. The only change now is it's a little more obvious.
And IMO that realization is a sort of "coming of age" that isn't in any way unique to modern generations.
When you become an adult and realize that the people running the show are just humans like you...
And that anyone selling you a simple "one right perspective" is only selling you part of the truth.
Or you realise that anyone selling you anything is merely wanting your money.
These days media organisations cause outrage for the sole purpose of getting you mad, only to read the article and realise the title is nonsense and you've just given them ad money.
I think you're wrong in a tiny but important point. It's not "All media organisations offer you senseless outrage and clickbait", it's "Some media organisations".
There are plenty of meaningful, in-depth, well-written articles which provide new viewpoints on topics that it's important to talk about. You just have to find mediums.
And I find it important to honor them instead of claiming "all media provides bullshit", which fuels a certain, toxic "truth is no where in media" atmosphere.
No disagreement here. The arc of my point is for those who look at 1984 and believe we're a long ways from there. On the contrary, we ARE there. The same ends, just different means.
I find lots of people miss Asimov with the Hari Seldon's Psychohistory. Google, Facebook, Amazon, Twitter, all manner of services now collect enough information to predict with relative accuracy what we like, want, and how we will behave in the imminent future or when presented with a certain situation. Google knows us better than we know ourselves.
Orwell wasn't criticizing technology. "Animal Farm" is not about animals or farms and "1984" is not about TVs with cameras. He took the lust for power and turned it to eleven (hence the dystopian setting) to write a book about oppression.
"If you want a picture of the future, imagine a boot stamping on a human face — forever."
Have you read Huxley's sequel to Brave New World? It's called "The Island." In it all of the technologies from BNW are used to set people free; to enhance the human condition.
I stand with Nir (and Huxley). These technologies are powerful, but that doesn't make them evil. They can be used for good.
Speaking of Hollywood, did someone notice that nearly every Hollywood movie made the 50s contained at least one scene in which the male star forces the female star to kiss him until she surrenders? At that time it was considered an obligatory part of the plot, now for such a thing you could go to jail.
Wasn't that psychology manipulation as well? Or is it now? Or both?
Just read that last week. Postman's critique of TV and media largly holds up. He would be glad about the iminent demise of TV, but apalled by what its successor technologies are doing to a still largely uncritical populace.
I find your comment rather ignorant. It's entirely possible that the scenarios can in some way coexist. I'd also argue that for 2001 and The Matrix, technological danger is not the main subject.
>Then there is the political criticism along the lines of George Orwell and some Luddites: technology is a tool for control and domination either by governments or evil capitalists.
I would not describe it like this. Proponents of technology but critics of the rationality it has created exist, most notably Marcuse. He criticises the development both in the capitaliat countries abd the USSR from a Marxist perspective in One-Dimensional Man, a widely acclaimed analysis of the rationality of creating false needs. Please read it, you may like it given your viewpoint. He very well integrates the last paragraph of your comment. Man is one-dimensional due to the rise of technological rationality in all expression, sexual, artistic and leisure. Nothing remains undominated, not even the great unknowns of nature and the human mind. Everything is integrated and rational through paradoxical irrationality of overproduction and false manufactured needs through advertisement.
Remember the startup mantra I've heard pg and others state: make stuff people want.
This is true locally, but globally and taken to the extreme it's a nightmare. People don't know what they want, and if you were to give them everything they wanted, you would destroy them. Take a look at anybody who's suddenly come into a lot of fame and fortune. The results are usually appalling.
Give children what they want - bright lights and shiny things, sugar & fat in equal measure, autonomy without responsibility, you'll end up with unhealthy sociopathic children.
We're all children and eschew what is good for us, just as adults we're cognisant of that failing.
Companies use our gluttony and greed to develop a desire in us, then fulfill that desire. The most financially profitable companies don't just give us what we want, they tell us first what we want, then sell it us.
Viz: You want to be lusted after by women, seen as powerful, make men jealous; our watch will do that for you; buying our watch will give you all you want.
Yes, agreed, globally is pathological, but it achieves the ends of Capitalism, making the rich richer off the backs of the poor man's labour.
The adaptations to a world of scarcity that made us successful as a species make us vulnerable in this world of abundance that we have created for ourselves.
We developed our own environment into a state in which we are not optimally adapted to it anymore.
You might also find the work of Marshall McLuhan to be interesting:
>Instead of tending towards a vast Alexandrian library the world has become a computer, an electronic brain, exactly as an infantile piece of science fiction. And as our senses have gone outside us, Big Brother goes inside. So, unless aware of this dynamic, we shall at once move into a phase of panic terrors, exactly befitting a small world of tribal drums, total interdependence, and superimposed co-existence.
IMO tech has simply become the dominant form of media. People who would earlier spend 3 hours/day watching TV might now spend 4 hours/day on Netflix/FB/Twitter etc.
The fundamental problem is not tech, it is the incentives of those who pay for the media we consume. Advertisers pay for media and it is in their interest to make the medium as addictive as possible. And when there is enough money to be made as an employee or shareholder of ad companies like FB/Google, people look the other way as long as they benefit.
The root problem is simple greed, and its a very old problem. Things are not going to change anytime soon as long as people are able to rationalize doing a small amount of evil for enough money/prestige. (not to say I wouldn't do the same)
Thats why these discussions seem pointless to me. Ultimately the people paying/investing in a company decide how a company operates, and they do not give a f* about the next article telling us how bad the product is. Even ordinary shareholders will happily buy FB stock if its projected to go up, nevermind why that is. When you operate within an economic system that is built solely to optimize revenue... it optimizes revenue, at all social costs.
I agree with you, but I think it's more than just greed. These companies consist of many employees. Depending on how you look at it, all those employees are contributing to the problem indirectly by working for the company. So one question that comes to mind is are these employees greedy too? What about the interns that get minimum wage? Are they not simply trying to make a living?
So I think the first part of your comment is more accurate in the sense that people rationalize things.
In that regard, I think we should look at schools emphasizing values in terms of work ethics. When I was in college I took a business ethics class. I realized that was the first time in my entire education that I was learning things about anything even remotely related to this subject, not to mention, the class was optional. That's a sad thing to happen in only the last years before entering the permanent work force.
The fact is if someone works for one of those companies, they are contributing to the problem. Maybe out of greed, maybe necessity, maybe ignorance. Moral judgements aside, it really doesn't matter why as long as they choose to work there - the company still benefits and the cycle still continues.
I'm sure teaching ethics would help.. but then again I don't think all the recent revelations about FB/Google/Twitter have done anything to make hiring more difficult for them. At most they pay 10-20% more and get the people they need. Parts of wall street are unethical and yet they get all the employees they need given the right salary.
I think there is a great truth to what you said, but the reduction of Tech to just media hides a set of critical and novel dangers in doing so.
Tech provides a new set of tools - both finely grained, invasive, and mass produce-able - which target areas of human existent earlier protected just because of how arcane and nebulous they were.
As human beings, we have many tendencies that are unguarded for buffer overflows, or malicious code injections.
With these tools, and corporate/shareholder incentives - firms and teams will run amok.
Its no longer just 1 way media, its now very close to mind invading media.
Sadly the only solution seems to be to ditch the smart phone whole.
You are saying tech is a more powerful tool than traditional media. I agree. But the purpose the tool serves in society is a question of how people wield it, not how powerful it is.
I wouldn’t call stackoverflow addictive - if anything I use it too little! - but I do have that problem with some of the linked network sites on stackexchange. “Why yes, who would win in a fight between Aquaman and The Flash is more interesting than why the third argument of the ExampleFooAPI constructor doesn’t behave itself!”
(I’m glad adblock lets me hide that block. Really helped me avoid those distractions).
There's a thing in essays where the author signifies cutting through the bull with some earthy and down-home talk, but I think this sentence "It’s too soon to tell, but it ain’t look so pretty from here." is a step too far.
It's not too far. Maybe You don't feel effects of antisocial media, but i do. Few years ago if friends wanted to go swimming. They were calling each other, there was human interaction. Now it's event on facebook. If You dont have favebook you're basically not a friend anymore, because noone is going to call You and invite. This is the effect of social media on our civilisation. It's like in '60 doctors were recommending smoking as stress reliever and everyone agreed until they knew better.
I was going to clap for this story, the first time ever (despite my misgivings regarding grammar and spelling - I'm sure I've made errors in this post, but none as egregious as the ones I've found in the essay) but anyway it turns out to clap on medium I need to create an account. I decided to forgo the opportunity out of respect for the author's evident wishes.
I agree. There is a contradiction here. And I am nearly as uncomfortable with centralized publishing platforms as I am with mass scale behavior design for profit.
And yet.
I originally published on my own site and it didn’t get much traction.
But when I edited it substantially and published as a direct response to Nir’s post, it reached 30,000 people, 6000 of whom finished the whole thing...in less than 24 hours.
Since the goal for me is spreading the message and fueling the converstion, I gotta go where distribution follows.
I’d be lying if I said that reaching an audience and seeing my post generate (mostly) positive, thoughtful discussion didn’t feel good.
But that feeling (for me, at least) is closer to gratitude and hopefulness than it is to ego gratification.
The claps and notifications of retweets are, however, quite addicting, in the truest sense of the word.
I find myself checking twitter more than is reasonable to see where it’s going.
And while there is some upside there (connecting with and engaging in conversation with like-minded peeps), the frequency of my checking is highly disproportionate to that Value.
When Rage Against the Machine signed to a major record label, they had a good quote, something along the lines of choosing to either preach to the choir or to address the masses. Them (and you) recognize the power of the medium and use it just like anyone else.
For me, the main problem is that I need my computer for what I consider "legit" activities such as working or communicating with my friends. Starting from there, it's easy to get trapped into other "toxic" activities such as facebook or news. I'd like to be able to just put the computer aside for a few days but it's almost impossible.
The thing is, it doesn't seem like much of this is even intentional. A/B tests (probably inspired by some sort of thought experiment re: 'getting more engagement') tell a story that says you have more users if you do this thing, when that thing just happens to be addictive.
Also, I would like to point out three things to folks who are asking "What about personal responsibility? What about agency?" here and on the article comments.
Facebook's shadow profiles take away any 'agency' and 'personal responsibility' out of the equation. While I don't yet know Facebook using shadow profiles to make someone's life worse off, their penchant for saying "Screw it" to any ethical considerations makes me feel its only a matter of time before even that crutch goes away.
Secondly, there is the complexity. You cannot take personal responsibility for something that you don't fully comprehend. At the same time, we are at a point where AI produces data insights that even the AI builders cannot reason about. For example, how does someone take responsibility for clicking the Like button on a post and then finding out that most of the people who liked the post were (say) closet anarchists which was then correlated with them being the best targets for some kind of subversive advertising from (say) the NRA?
Thirdly, there is the asymmetry involved in "moderate use". It is possible for Facebook to create "automated notifications" implemented at a very tiny cost to them in terms of their resources/profits while the effort needed on your part to make sure you don't turn into a Pavlovian "notifications" dog comes at an extremely high time cost to you.
108 comments
[ 3.2 ms ] story [ 185 ms ] threadMaybe we need to update the saying: “the keyboard and a content database can be mightier than the predator drone”
?
The killer feature may be that it does not exploit your human flaws.
People are always surprised what new needs we have for technology. Now we need software and services that respect us as humans.
* I've been manipulated to bring an umbrella because I don't like to get rained on.
* I've been manipulated into liking broadband because it provides the internet faster than dialup.
* I've been manipulated into owning a car because it's a fast convenient mode of transportation.
This is the basics of products and services that provide value, this isn't magical or new, and this book is probably being more honest than most industries are.
The point the book was making is that if you don't want a product to be flash in the pan garbage, follow some simple guidelines which ensure that your service is actually providing some sort of value to end users, or your end users will eventually leave and find better things to do with their time and resources (Unless you're selling heroin I guess).
I don't consider Facebook sending me an email when I get a personal message to be exploitative, but I find no value at all when they send me something every time one of my friends fart. I specifically avoid interacting with those "hey someone did something" emails because they are annoying and make me want to stop being on facebook altogether. That's the result of abusing the hook model, users go away because it's annoying and unproductive.
Same for surveillance. It was always there but now it can be done on an absolutely unprecedented level.
Your definition of "manipulate" is not what most people mean in these contexts. Your use of "manipulate" in such a way muddies the waters and distracts from the core issue.
Putting word choice aside, the core issue here is _not_ about people's preferences (e.g. many people don't like to get rained on). Rather, the core issue is about unknown and unwelcome influences, particularly influences that operate on parts of our minds that are hard to combat due to human nature.
Facebook's manipulation of our attention is conceptually related to how casinos trick people into staying longer than they would otherwise.
I can't afford a butler, but if I could, then I'd get one. He'd live in my house and know all about me and my personal life; knowing about my personal life would help him do his job better. I wouldn't worry that my butler knew I was in a relationship, or pregnant, and that he changed his behaviour accordingly.
On the other hand, I suppose I wouldn't want a butler who sold information about me to third parties, or used the information to advertise to me for kickbacks. "You have been down recently, sir... perhaps a new Bentley would cheer you up?"
And suppose you are of psychological steel. Should those who aren't still be exploited? There are entire industries and schools of design whose goal it is to extract as much money as possible from users, take slot machine design for example. (I'd definitely recommend looking up the work of Natasha Schull). Do you consider that practice immoral? Nir's existence is a testament to the fact that product designers use similar techniques
Also, there are many ways to fool people that still work even if the victim knows about them, and the pain of being told one has been fooled means every technique the victim knows can (and often will) be rattled off as a fully generic counter-argument to whoever tells them that they have been fooled.
So you're saying it's ok for the government to manipulate you, but not a company, because their goal isn't a vague idea of societal good, it's to make more money? I find it much easier to understand why a company does X, rather than why the age for drinking is set at 21. I take that understanding with me every time I sign up for any kind of service.
If anything should come out of any of this, it's that personal/private information should be fully controlled by individuals in the same way health information is handled. It shouldn't be so easy to generate huge databases of people and claim ownership over them, and it definitely shouldn't be so easy to use those social ties to create an addiction.
Counterexample: drink-driving blood alcohol limit.
> So you're saying it's ok for the government to manipulate you, but not a company, because their goal isn't a vague idea of societal good, it's to make more money?
In the right area, sort of (manipulation generally bad but a few things can override the badness) but I would say that governments are (in theory) democratically accountable, whereas business are legally and financially accountable. Yes, people do vote for dumb laws, but in principle that can be undone by another vote, businesses only get in trouble with their shareholders if the government doesn’t limit their behaviour.
While I agree with the point that it would be better if it wasn’t “so easy to generate huge databases of people and claim ownership over them, and it definitely shouldn't be so easy to use those social ties to create an addiction“, I don’t have any long term solutions. At some point the collective footprint of someone’s public statements on the internet is such a database, albeit in the informal sense of the word and spread over dozens of unconnected businesses, and demonstrating that the information has been used to cause addiction is more difficult than demonstrating tax avoidance.
To be manipulated, you'd have to exposed to a constant stream of ads telling you how wonderful umbrellas are and what an unattractive loser you are for not owning one; how desirable people with high status use exclusive umbrellas while most people get by under a wet newspaper; and how you should buy a new umbrella every year because they're constantly improving, and this year's umbrella already has more a billion users worldwide and offers 23% more anti-rain features thanks to the exclusive Umbrella Corporation[tm] lifestyle eco-system. (And now here's a video of attractive people smiling, and of a world-famous umbrella designer talking earnestly about engineering, design, and craftsmanship.)
You'd also see paid testimonials and likes on social media sites. They'd appear to be spontaneous and sincere praise, but in fact they'd be factory farmed and produced to order by unhappy people working in open plan offices paid so little they can barely afford the rent.
And you'd have an umbrella that only worked reliably some of the time, so you'd have to keep considering the purchase of an improved model - in the vain hope you could finally end your constant nagging frustration with umbrella technology.
Worst of all, you'd believe you were making a straightforward umbrella-buying decision entirely on your own initiative, using perfectly objective and rational criteria.
Then there is the games sector. I was only in that sector from 2007 to 2010, but when I left many of them were asking how to make their content “addictive”.
I’d say it’s immoral that Facebook et al have wasted subjective lifetimes of their users’ consciousness by addicting them to whichever drug the brain synthesises to reward the minor and unfulfilling social interactions that it provides (and the reverse, the pain from the fear of missing out).
You weren't manipulated. Manipulating you into buying broadband would entail creating the artificial perception that broadband is faster. If a company produces a product that performs as it is supposed to, and it solves a problem you have that was not artificially created, then you're not being manipulated. Same for umbrellas. Not the same for cars: the auto industry has pretty clearly lobbied for policies that ensure sustained demand for cars as opposed to public transit.
https://medium.com/the-mission/the-morality-of-manipulation-...
https://en.wikipedia.org/wiki/AmoralityNow that I've said the positive, let me talk about the negatives. I bet things will get worse before they get better. At present, the commercial pressures on social media platforms to manipulate "users" dwarf human pushback against these "dark art" techniques.
I don't see any easy solutions. As with many collective challenges, we need increased awareness to effect change. This is arguably a political problem; we need social organizing as a counterbalancing force.
At the moment we have the opposite. No one cares how you make your first billion or two, or how people suffer as a side effect. They only care that you have a billion or two. Toxic side-effects remain invisible and uncounted.
There's a certain irony in attacking EA for building a game economy that's closer to the effort/reward profile of the real economy - for both players and owners - than any other game economy to date.
Second, I advocate for increased awareness because awareness is the first step towards rational thought. Many psychological manipulation techniques work best on the irrational parts of our brains, so often a useful first step is to engage the rational mind. That said, I'm not claiming awareness guarantees systemic change.
Therein lies the problem. Who is to decide what is a social positive or negative among some of the more nuanced and innocuous bits of society?
This is not a road we should travel down any further than we have already.
On top of that we need awareness that such tools exit.
And on top of both of those we need a universal belief that this "stack" is an enabler and that enpowerment can have impact; that the little guy/ gal do matter.
Currently, best I can tell, we're lacking quite a bit at all three levels.
From many angles there are too many things working to passively shore this edifice up.
The best solution I found in years was after randomly catching up with a good friend. He just gave up on the smart phone.
Hes not a Techie, he used to love video games - but he has a unique quality to be embody Human-ness. Hes dumped social networks, and dropped smart phones.
In short the way out for those who can manage it is cold turkey.
I now suspect that the people who can handle this, are not on HN or social networks anymore
I believe the tech world has a huge problem on its hands here, and that problem doesn't get anywhere near the attention it deserves. Large swaths of the population are turning into zombies and tech is the facilitator. Try looking at the drivers turning through a busy intersection some time. It's obscene how many are staring down at their phones rather than at whatever might be in the road around the corner.
That said, I don't believe the problem is simply that certain products remind or encourage people to use them at times. Rather, it's that there is a class of truly addictive products that don't do much (if anything) to help people manage the resulting dependency. And that approach is celebrated far more than it's scrutinized.
Coincidentally, I wrote the following on Twitter yesterday:
"Any potentially addictive software/service should offer a way for users to lock themselves out of their own account for a predetermined set of hours." [0]
It was meant in my usual tongue-in-cheek style, but I do think that a solution for breaking tech product dependency is sorely needed and hope this discussion gains some steam.
[0] https://twitter.com/ncantelmo/status/930876372620374017
I'd also recommend checking out The Attention Merchants by Tim Wu and Addiction by Design by Natasha Schull for background on the history of the commodification of attention and of the
It's really noticeable that Hacker News provides exactly this feature, and absolutely no other site that I've seen does.
For years I resorted to hacking /etc/hosts to block HN. Mentally, I just assumed that no website would have such settings so I never bothered to learn what noprocrast meant.
The idea of a hard block on tech is intuitive, but it doesn't do anything to undermine the habitual learning in tech addiction; it just frustrates it for a fixed amount of time.
Space focuses on slowly undermining the compulsions behind tech addiction. So that, over time, you don't have the impulse to open the app anymore.
If you try it out, let me know what you think!
http://youjustneedspace.com/
yet
By analogy - If the population of a city was exposed intentionally, to a cold virus - it is not the people with natural resistance, we are concerned about.
We are glad those people are not affected, and know they are not part of the % affected by the virus - and then we focus on the people affected by it.
---
If I were to engage with your point in the spirit it is intended, there are several issues -
1) Tech recruits the best of the best. IT pays for the smartest people in the world (and this is not hyperbole), to get past agency. This is not hard core drug dealer style sale of addiction. This is candy crush, happy smiley imagery, "just one more game", style behavioral addiction.
Most people don't even know they are being addicted/affected
2) This is all at internet scale. Just on a probabilistic level, the casualties/infection is going to vastly outperform personal agency.
3) The network effects of having more people doing this, at this scale, pushes it to crisis levels very quickly. It wouldn't matter what the few immune or capable can do.
Yet the entertainment and content-creating side of the internet relies very explicitly on ad revenue.
Are creators that rely on revenue generated by manipulating human psychology morally bankrupt?
Frankly, it's shameful for any company that says they care about their customers to employ marketing psychology tactics. Marketing and advertising are (in their modern, virulent form) simply an attempt to hack viewers to shift the locus of control from the customer to the business. (Other objectionable things about tech have this kind of behavior, too.)
Frankly, it's just disgusting and I can't blame customers for really viewing companies negatively based on their advertising practices.
(If all that wasn't bad enough, online advertising often comes with literal malware ontop.)
The traditional Hollywood criticism on technology goes along the lines of Frankenstein, Metropolis, 2001 A Space Odyssey, Terminator and Matrix: technology will control us. I find it dumb, frankly.
Then there is the political criticism along the lines of George Orwell and some Luddites: technology is a tool for control and domination either by governments or evil capitalists. It is not totally wrong (e.g.: China, Turkey) but not the exact true sometimes.
But the Huxley criticism is, for me, the most interesting: technology will entertain us to stupidity, apathy and alienation. This seems to be very much the case in here.
For example, Orwell (in 1984) focused on the filtering, manipulation, constriction and reconstruction of information in order to continuouly redraw the history / truth. But today the truth is something that's often just as fuzzy because we're overwhelmed with too much information and too many distractions. Different means, same ends (read: confused and manipulated proles).
But it's the ends that matter. Both Huxley and Orwell have warned us. The irony is, at this point, too few are paying attention.
And IMO that realization is a sort of "coming of age" that isn't in any way unique to modern generations.
When you become an adult and realize that the people running the show are just humans like you...
And that anyone selling you a simple "one right perspective" is only selling you part of the truth.
These days media organisations cause outrage for the sole purpose of getting you mad, only to read the article and realise the title is nonsense and you've just given them ad money.
"If you want a picture of the future, imagine a boot stamping on a human face — forever."
I stand with Nir (and Huxley). These technologies are powerful, but that doesn't make them evil. They can be used for good.
I find your comment rather ignorant. It's entirely possible that the scenarios can in some way coexist. I'd also argue that for 2001 and The Matrix, technological danger is not the main subject.
I would not describe it like this. Proponents of technology but critics of the rationality it has created exist, most notably Marcuse. He criticises the development both in the capitaliat countries abd the USSR from a Marxist perspective in One-Dimensional Man, a widely acclaimed analysis of the rationality of creating false needs. Please read it, you may like it given your viewpoint. He very well integrates the last paragraph of your comment. Man is one-dimensional due to the rise of technological rationality in all expression, sexual, artistic and leisure. Nothing remains undominated, not even the great unknowns of nature and the human mind. Everything is integrated and rational through paradoxical irrationality of overproduction and false manufactured needs through advertisement.
This is true locally, but globally and taken to the extreme it's a nightmare. People don't know what they want, and if you were to give them everything they wanted, you would destroy them. Take a look at anybody who's suddenly come into a lot of fame and fortune. The results are usually appalling.
We're all children and eschew what is good for us, just as adults we're cognisant of that failing.
Companies use our gluttony and greed to develop a desire in us, then fulfill that desire. The most financially profitable companies don't just give us what we want, they tell us first what we want, then sell it us.
Viz: You want to be lusted after by women, seen as powerful, make men jealous; our watch will do that for you; buying our watch will give you all you want.
Yes, agreed, globally is pathological, but it achieves the ends of Capitalism, making the rich richer off the backs of the poor man's labour.
We developed our own environment into a state in which we are not optimally adapted to it anymore.
> [...] technology will entertain us to stupidity, apathy and alienation [...]
Perhaps a combination of both. The whole concept is as old as the Roman empire, [1].
[1] https://en.wikipedia.org/wiki/Bread_and_circuses
http://www.anorak.co.uk/wp-content/uploads/2013/06/huxley-or...
>Instead of tending towards a vast Alexandrian library the world has become a computer, an electronic brain, exactly as an infantile piece of science fiction. And as our senses have gone outside us, Big Brother goes inside. So, unless aware of this dynamic, we shall at once move into a phase of panic terrors, exactly befitting a small world of tribal drums, total interdependence, and superimposed co-existence.
-The Gutenberg Galaxy (1962)
Sounds like the current media panic about terrorism, etc.
The fundamental problem is not tech, it is the incentives of those who pay for the media we consume. Advertisers pay for media and it is in their interest to make the medium as addictive as possible. And when there is enough money to be made as an employee or shareholder of ad companies like FB/Google, people look the other way as long as they benefit.
The root problem is simple greed, and its a very old problem. Things are not going to change anytime soon as long as people are able to rationalize doing a small amount of evil for enough money/prestige. (not to say I wouldn't do the same)
Thats why these discussions seem pointless to me. Ultimately the people paying/investing in a company decide how a company operates, and they do not give a f* about the next article telling us how bad the product is. Even ordinary shareholders will happily buy FB stock if its projected to go up, nevermind why that is. When you operate within an economic system that is built solely to optimize revenue... it optimizes revenue, at all social costs.
So I think the first part of your comment is more accurate in the sense that people rationalize things.
In that regard, I think we should look at schools emphasizing values in terms of work ethics. When I was in college I took a business ethics class. I realized that was the first time in my entire education that I was learning things about anything even remotely related to this subject, not to mention, the class was optional. That's a sad thing to happen in only the last years before entering the permanent work force.
I'm sure teaching ethics would help.. but then again I don't think all the recent revelations about FB/Google/Twitter have done anything to make hiring more difficult for them. At most they pay 10-20% more and get the people they need. Parts of wall street are unethical and yet they get all the employees they need given the right salary.
Tech provides a new set of tools - both finely grained, invasive, and mass produce-able - which target areas of human existent earlier protected just because of how arcane and nebulous they were.
As human beings, we have many tendencies that are unguarded for buffer overflows, or malicious code injections.
With these tools, and corporate/shareholder incentives - firms and teams will run amok.
Its no longer just 1 way media, its now very close to mind invading media.
Sadly the only solution seems to be to ditch the smart phone whole.
https://blog.ycombinator.com/13-startup-ideas/
Please, no.
(I’m glad adblock lets me hide that block. Really helped me avoid those distractions).
http://fortune.com/2016/06/22/loneliness-is-a-modern-day-epi...
https://www.cbsnews.com/news/many-americans-are-lonely-surve...
And yet, the author decided to write his article on the very site that uses techniques Nir talks about to promote engagement.
And yet.
I originally published on my own site and it didn’t get much traction.
But when I edited it substantially and published as a direct response to Nir’s post, it reached 30,000 people, 6000 of whom finished the whole thing...in less than 24 hours.
Since the goal for me is spreading the message and fueling the converstion, I gotta go where distribution follows.
But that feeling (for me, at least) is closer to gratitude and hopefulness than it is to ego gratification.
The claps and notifications of retweets are, however, quite addicting, in the truest sense of the word.
I find myself checking twitter more than is reasonable to see where it’s going.
And while there is some upside there (connecting with and engaging in conversation with like-minded peeps), the frequency of my checking is highly disproportionate to that Value.
Ugh.
Facebook's shadow profiles take away any 'agency' and 'personal responsibility' out of the equation. While I don't yet know Facebook using shadow profiles to make someone's life worse off, their penchant for saying "Screw it" to any ethical considerations makes me feel its only a matter of time before even that crutch goes away.
Secondly, there is the complexity. You cannot take personal responsibility for something that you don't fully comprehend. At the same time, we are at a point where AI produces data insights that even the AI builders cannot reason about. For example, how does someone take responsibility for clicking the Like button on a post and then finding out that most of the people who liked the post were (say) closet anarchists which was then correlated with them being the best targets for some kind of subversive advertising from (say) the NRA?
Thirdly, there is the asymmetry involved in "moderate use". It is possible for Facebook to create "automated notifications" implemented at a very tiny cost to them in terms of their resources/profits while the effort needed on your part to make sure you don't turn into a Pavlovian "notifications" dog comes at an extremely high time cost to you.