We do this all the time already when we write various interface services to access a common data store. Each service provides whatever interface is appropriate and performs any desired transformations.
It's not clear to me what the real benefits of Object Lambda provides. Indeed it could move your transformation code closer to its data, but the cost of doing that is the management and complexity of the lambdas. I am curious what a best use case example would be for this.
> Resizing and watermarking images on the fly using caller-specific details, such as the user who requested the object.
Now we need to either do rescale directly after storing or need to use another service (on EC2) to do the resize. With Object Lamda, we could do it on-the-fly as the user calls the image. This is operational a lot easier.
At the moment this is currently handled with lambda @ edge with Cloudfront. The benefit with Cloudfront over S3 lambda is that the image is automatically cached after the resize is computed.
The benefit of S3 Object Lambda is that your function is adapted to the S3 API. If you already have an application integrated with S3, then you can use Object Lambda to provide some new behavior to the data without changing the application.
Your example is way better than what AWS has on their page (marketing, analytics) which made me think immediately: you shouldn't use a file store for that but a database.
S3 is basically a database. You can store everything there and interact with it via SQL with Athena, for example. I could see utilizing all of Amazon's "infinitely scalable" services to build very simple and powerful software really easily.
For small numbers of large queries, Athena has been an incredible technology stack .. I pay pennies a month to run my analytics jobs on it (aggregates across a few hundred million rows, joins with tens of millions of rows into hundreds of thousands of rows).
I start my data as .csv.gz but the first step is a CTAS to extract columns and convert to compressed parquet. This step basically costs the most but gives a 10x data size reduction to downstream steps.
Athena does not work at all if you perform large numbers of small indexed read queries, definitely use a traditional database for that.
Yes, it's great if you setup the data structures properly, we pay a few bucks a month to run a lot of infra on it to query telemetry data over a few hours to a few days at a time. I think an equivalent cost in just a database server instance to run the same load would be at least 25 to 50 a month. You're not really locked into athena, either, it just takes standard sql, so you could easily transition to a dedicated box later.
I copied an example from the AWS blog post and adapted to our use-case.
I did never work with it, but there is Athena that allows to query S3 in something like SQL. I think there _might_ be some cases were S3 is a useful database, as one can use pre-signed URLs to let any client write to the database (S3). That‘s something I‘m not sure how to set up with a classic database and without any API in between.
IMHO, I think it‘s all about the same functionality with a simpler architecture.
If S3 is database - Object Lambda is stored procedure?
I guess that for some systems that are heavily S3-based it will help adding new features without needing to rewrite the interface or duplicate the functionality.
It's nice to know that AWS achieved such lock-in with some customers.
At my last job we had a lot of data in S3 queried using the data tooling like Athena or EMR.
This was useful to a lot of teams in the org but the downside was trying to lockdown certain fields like IP address to people who needed access to that data for support/operational reasons.
Amazon did come up with some IAM/grouping policy things but these were a bit clunky from what I recall. I wonder if something like this would have made it a bit more flexible to allow us to redact the IP address on the fly
EDIT: although I wonder about the performance implication of this for the "big data" use cases
Despite being familiar with both of those individually, the combination of the two words side by side switched me into an entirely different context that took several tries to escape from...
If I understand things correctly, the modern way to to read-time masking in the AWS ecosystem is to introduce AWS Lake Formation ( https://aws.amazon.com/lake-formation/ ) as the abstraction layer between Athena and S3.
My use case would be to transform log files from something like TSV to JSON, but without being able to stream the output seems like the file size is going to be limited to the lambda memory size or the /tmp 512MB.
I wonder how well this could work as performance optimization.
When you do some data crunching you can offload some work to lambda. Theoretically, bringing execution closer to data brings more efficiency. Though AWS Lamba was very expensive for CPU intensive steady workload. Few times than using EC2 is some use cases.
They don't actually say that the lambda executes any closer to the data than any other instance. It would make sense for these to operate on the S3 nodes of course, but the description doesn't say one way or the other.
It looks like from within an S3 Object Lambda object handler you can do pretty much everything you can in a regular Lambda, including querying other S3 objects or performing any number of HTTP requests. This opens up some interesting use cases, as well as some extreme footgun scenarios.
True. But the idea is that these functions run in between your server/application requesting the object and AWS returning the object to your server. In other words, they happen synchronously with the object request.
So you don't want your server waiting around for 15 minutes on an open request, and most systems won't let you do that anyway (browsers definitely won't). 60 seconds is about as long as you'd want to wait around for your object, and arguably even that is too long. If you need to do some sort of lengthy process to your objects then it should probably be done in some sort of asynchronous queue using a normal Lambda.
How far ahead is AWS in comparison of other Cloud Providers? I only use the basic services (compute, storage, cloud functions, etc). But for people with advance use cases, are GCP and others even comparable?
GCP is behind. They're are behind in niche services to be fair but behind nonetheless. Nothing that prevents building similar services, but coming from AWS, I routinely feel frustrated by lack of finesse and comparable offerings. IMO, GCP's greatest benefits over AWS are:
* BigQuery
* Tighter k8s integration with GKE
* Single message service (PubSub)
Unless BigQuery is your first class citizen, I would avoid GCP.
If you sign a service contract with Google (which you will if you are spending any amount of serious money), then you will have a precisely defined guarantee that they will continue operating GCloud.
I doubt anyone’s service contract stipulates that Google will continue to invest heavily to keep feature parity with AWS and Azure for the next 20 years
AWS is definitely the leader. But Microsoft Azure is giving them a run for their money, and is stealing away a lot of the big clients. Azure is adding new products quickly and adding features to existing products to stay competitive with AWS. I've noticed that Azure seems to be much more "focused" than AWS. Meaning that AWS has a lot of random/niche services. Azure doesn't compete against these, but focuses on being ultra-competitive in the core services that enterprise customers really want. And I do think AWS execs are spending a lot of time talking about Azure and trying to keep the competitive edge. I don't think AWS execs are spending much energy worrying about GCP.
Google Cloud Platform (GCP) is a bit of a joke. I really don't think it is all that competitive. Its trying to stay caught up and they do offer a lot of similar services. But their documentation is all over the place. Their service administration is a disaster. You can tell that each product is built by separate teams that only meet together once a quarter over Zoom. Its very disjointed. GCP is definitely trying to play catch-up. That isn't to say that you couldn't build a powerful platform ontop of GCP, many respectable companies are doing it. Discord is one that comes to mind that is (or at least was...) built entirely on GCP. I don't want to say it is shit. But its not as cohesive as Azure and AWS.
Basically AWS and Azure are the two core competitors. I really think that in practical (non-niche, using core services) they are neck and neck. GCP is in the race, but definitely lagging behind the others.
Two questions I had was: 1. is there some way to cache (eg LRU upto a size limit) the results of transformation functions. 2. They mentioned custom header fields as a way for clients to pass along data what about query parameters do those get passed along?
Durable objects launched ages ago, and they’ve already built COVID vaccination waitlists on it. Different use cases too, Durable Object is for running JS code as a long lived object that resides at an edge close to your user.
I'm always amused at the vendor lock-in conversations.
It's never about technology. It's about cost. If AWS ever priced themselves wildly above Azure or GCP or ___, shit would hit the fan.
Outside of that, who cares if you're locked into one cloud or another? Features? Maybe if you're super picky about Power BI or SQL Server? It's never made any sense to me.
Parler would not have been saved by being more mobile between clouds, no provider would have them. Only self hosting would have saved them, which is rarely what people are proposing when they talk about vendor lock-in.
I would have agreed with you a year or two ago, but times are changing. Vendor lock-in isn't just about cost anymore. You're also gambling on ideology that the provider you are locking in to won't decide to terminate your account and destroy your business because they disagree with your philosophy or politics.
In Parler's case their platform was proudly used to organise illegal activity. Which is against ToS, not ideological.
If someone hosted an MP3 piracy site or torrent tracker or carding forum on AWS, it would likely terminated too, regardless of whether Amazon are ideologically against piracy.
I'm no fan of the content on Parler, but Twitter (who uses AWS) was used for far more organization of illegal activity than Parler was. Twitter wasn't targeted though. I don't think it was ideologically agreement with Twitter (although I'm sure there is much ideological agreement as the tech industry has become quite the echo chamber) as much as Twitter is just far too rich and powerful to be shut down.
If you're a big fish with powerfu rich users like Twitter, then vendor lock-in isn't a risk. If you're a small, inconsequential platform that can be smashed like a fly on the window, then you really need to care about such things.
>I'm no fan of the content on Parler, but Twitter (who uses AWS) was used for far more organization of illegal activity than Parler was.
Parler outright refused to delete posts calling for the mass killing of politicians, even when those posts were specifically pointed out to Parler by AWS. This is all detailed in court documents. AWS very patiently tried to prod Parler into following AWS's terms of service, but Parler refused. This is a very clear-cut matter and those suggesting otherwise are either trying to deceive or being misled.
What specific posts did Parler refuse to delete, and what is the evidence that they refused to delete them? IIRC, Parler was deleting posts that AWS complained about as they reviewed them, but had a large backlog, and AWS decided they were moving too slowly. I haven't seen any sourced claims that Parler refused to remove "posts calling for the mass killing of politicians", only unsupported allegations. If it really is so very clear-cut, do you have a very good source for those claims?
This matches my understanding as well. The CEO (at the time) was on a podcast and specifically said they were moderating/deleting the posts. AWS just wanted it to be done faster (in an automated fashion).
To reiterate, AWS provided this evidence in its court filings when it was sued by Parler. Pages and pages of email records where AWS customer service reps sent Parler specific posts that violated AWS's ToS, and Parler's responded refusing to take them down.
Twitter also uses GCP, and Azure I’m sure. Their serving layer is not hosted by AWS though, and I’m reasonably certain AWS could not shut down Twitter immediately simply terminating their AWS account.
Parler was set up to harbour the exact sort of people that got banned from platforms like Twitter for being too unhinged and in some cases dangerous. It's not really comparable.
If you’re not flaunting ToS, then making sure you have all your legal items taken care of (trademark, copyright, patents), you should be fine.
If you’re building something that a cloud provider might adopt or build their own, well, isn’t that risky regardless of the cloud provider element?
I know some companies avoid one provider or the other out of competitive concerns. That seems easily managed with contractual obligations.
Twitter has enabled some borderline illegal behavior, but the account holder has been the one at risk. I don’t think Twitter systematically allows illegal behavior like parler did.
> If you’re not flaunting ToS,... you should be fine.
https://aws.amazon.com/aup/: You may not use, or encourage, promote, facilitate or instruct others to use, the Services.. for any.. offensive use, or to transmit, store, display, distribute or otherwise make available content that is.. offensive.
> Twitter systematically allows illegal behavior like parler did.
Does this sound libelous to you, and do you have evidence supporting this? Even AWS's response didn't allege Parler "systematically allowed illegal behavior".
It is amusing that for years the common wisdom and fad was to write your database layer so that it could be independent of the database engine.
The idea was to avoid vendor lock and make it "easy" to change the database engine.
A project was on, perhaps 2 years ago, still followed this pattern, but it ended up also adopting somewhere between 10 - 15 AWS services and we never discussed an abstraction layer on top of it.
There are one or two efforts now to create an independent layer on top of cloud services. I have a hard time understanding how that will work outside of basic services.
Kubernetes is the independent layer on top of cloud services - unsurprisingly it is championed by Google who are #3 in the cloud business, so they have a vested interest in people not being vendor locked to AWS. K8s seems to be doing well.
I see the avoidance of vendor lock-in as a way to prevent Amazon from gaining too much power compared to their competition (they already do, but you see my point, hopefully)
I’m designing a system right now and I’m specifically not allowing it to become too ingrained into the AWS ecosystem so that I can easy move it to GCP, Azure, or elsewhere (would that Digital Ocean offered better support) if/when Amazon gets too big and starts to take advantage of their size. That will happen. It’s only a matter of time.
If you base your product on locked in services you are at the mercy of those services. Once you scale you will no longer afford to use AWS and have to colocate your own hardware, along with entirely reimplementing your stack to use open source software. Not every startup is a billion "forever" funded project like Imgur or Twitter that can throw money in a hole and not care. Lots of smaller companies are coming to terms with having to shun away cloud services and rely on "the old ways", because it's the only affordable and scalable option.
AWS is for privledged companies playing with silicon valley monopoly money based on made up "evaluations", most people don't have that benefit.
>Once you scale you will no longer afford to use AWS and have to colocate your own hardware, along with entirely reimplementing your stack to use open source software.
More companies are moving to the cloud from managed colo than the inverse. For example, companies like Uber and Adobe has been moving from their own datacenters to AWS. Even dinosaurs like Disney have moved to the cloud.
Anyone who saves a significant amount of money running their own DCs vs uses the cloud either has significant security requirements or is the in the business of selling infrastructure (ex. Dropbox)
And even for those "dinosaurs" that are persnickety about control, Outposts allows them to house the rack onsite while still being managed by the AWS console. On on-premise cloud, if you will.
I see it slightly differently. For most of the history of our industry, companies were penny-wise but pound-foolish. And I don't mean non technical leadership here, I include the developers themselves. (I'll spend 2 weeks to automate a tasks that takes 10 minutes to do manually once a week)
Everyone is waking up to the "total cost" of software development. Market forces are forcing people to realize that developer time is the most valuable commodity there is.
Companies don't go to the cloud because they have silicon valley money to burn. They go to the cloud because every minute of their developer time that is not spent on development operations and maintainance of code that a cloud provider offers instead, is a minute they can spend on the core UNIQUE business proposition of the company.
You can bemoan the skills lost in our industry, but that always happened at every layer of abstraction in every technical industry.
AWS is dirt cheap and many small and mid-range companies use their platform. If you architect things properly, you can get away with pennies on the dollar.
Not every platform needs EC2 instances. Most can build on Lambdas, API Gateway, and S3.
Not sure what evidence supports this notion of billions and privilege. If anything, AWS has democratized entrepreneurial endeavors.
AWS is not dirt cheap. It has a great value proposition in case people have a lot more money than time/expertise/etc. (So usually for one-man army startups, or startups with unlimited VC funds that want "hypergrowth", and for very high-margin high-growth/high-volume businesses where they can make a special deal with AWS, that is Netflix.)
> If anything, AWS has democratized entrepreneurial endeavors.
More likely AWS simply made a lot of money on the race to the bottom (or top) with regards to an ever faster, fatter, fancier and shinier Web. (Now everyone needs a CDN, everyone needs many PoPs, because the market forces demand this. Of course the total costs are enormous, but due to the increased scale and decades of technological changes the unit costs have gone down.)
> Most can build on Lambdas, API Gateway, and S3.
Sure, and it's great if people are aware of what kind of workload to put where. And if they build in rate limits. Otherwise they will be surprised when they run into the classic gotchas of "how we burnt all of our seed money on AWS bills".
Considering AWS dirt cheap is like considering a tesla to be dirt cheap. You're talking big money, I'm saying most people can't do that because they don't have that kind of money.
A simple colocated server nowadays can do almost a petabyte a month for less than $1000. Lookup how much that is on AWS.
Lambdas and s3 are cool until you get ddos'd and now owe way more money. Or, your software doesn't work with lambda, which is true of the overwhelming majority of services.
> Once you scale you will no longer afford to use AWS and have to colocate your own hardware, along with entirely reimplementing your stack to use open source software.
If you price right, then if you scale to the point you need it for ongoing costs, you'll have money to spend on the one-time cost of building an abstraction layer over any locked-in services and switching to that, enabling both a migration off and, potentially, its own resellable product.
We use the triggered lambda at work to do image resizing on upload and likely won’t be moving to this. This looks really cool for if you wanted to let clients request at whatever size they want instead of defining them in advance for example - you can resize on the fly. Their watermarking example is clever too IMO. I think it’s basically that this is s3 object + request details -> response, whereas the flow you talk of is just s3 object as input.
The caveat I would see here is that Lambdas are very limited (memory/disk) so if you're dealing with a big json/xml file you'd need specialized libraries to redact it.
This is really cool. Imagine we build non-relational simple read-only APIs powered by s3 with lambda authorizing, filtering, paginating requests? We have another lambda function poplulating s3 for writes.
96 comments
[ 3.1 ms ] story [ 126 ms ] threadIt's not clear to me what the real benefits of Object Lambda provides. Indeed it could move your transformation code closer to its data, but the cost of doing that is the management and complexity of the lambdas. I am curious what a best use case example would be for this.
> Resizing and watermarking images on the fly using caller-specific details, such as the user who requested the object.
Now we need to either do rescale directly after storing or need to use another service (on EC2) to do the resize. With Object Lamda, we could do it on-the-fly as the user calls the image. This is operational a lot easier.
Advantage here is that clients can just continue to use their existing S3 interface yet your lambda is still invoked.
Edit: I guess you meant that cloudfront caches it for you automatically.
I start my data as .csv.gz but the first step is a CTAS to extract columns and convert to compressed parquet. This step basically costs the most but gives a 10x data size reduction to downstream steps.
Athena does not work at all if you perform large numbers of small indexed read queries, definitely use a traditional database for that.
I did never work with it, but there is Athena that allows to query S3 in something like SQL. I think there _might_ be some cases were S3 is a useful database, as one can use pre-signed URLs to let any client write to the database (S3). That‘s something I‘m not sure how to set up with a classic database and without any API in between.
IMHO, I think it‘s all about the same functionality with a simpler architecture.
I guess that for some systems that are heavily S3-based it will help adding new features without needing to rewrite the interface or duplicate the functionality.
It's nice to know that AWS achieved such lock-in with some customers.
With this we might simplify the pipeline because we can keep the same S3 interface.
The lock-in mechanism. Imo this isn’t more functional than other approaches, it’s a way to ensure you don’t switch services.
It’s basically doing the same thing as an SQL view. It just hides and abstracts away some complexity.
The brilliant part is that it works even if you don't use AWS for compute: they can start making money on per-request compute for S3 "only" customers.
This was useful to a lot of teams in the org but the downside was trying to lockdown certain fields like IP address to people who needed access to that data for support/operational reasons.
Amazon did come up with some IAM/grouping policy things but these were a bit clunky from what I recall. I wonder if something like this would have made it a bit more flexible to allow us to redact the IP address on the fly
EDIT: although I wonder about the performance implication of this for the "big data" use cases
Despite being familiar with both of those individually, the combination of the two words side by side switched me into an entirely different context that took several tries to escape from...
https://en.wikipedia.org/wiki/Athenahealth
https://en.wikipedia.org/wiki/Electronic_medical_record
For larger objects, the WriteGetObjectResponse API supports chunked transfer encoding to implement a streaming data transfer.
When you do some data crunching you can offload some work to lambda. Theoretically, bringing execution closer to data brings more efficiency. Though AWS Lamba was very expensive for CPU intensive steady workload. Few times than using EC2 is some use cases.
So you don't want your server waiting around for 15 minutes on an open request, and most systems won't let you do that anyway (browsers definitely won't). 60 seconds is about as long as you'd want to wait around for your object, and arguably even that is too long. If you need to do some sort of lengthy process to your objects then it should probably be done in some sort of asynchronous queue using a normal Lambda.
* BigQuery
* Tighter k8s integration with GKE
* Single message service (PubSub)
Unless BigQuery is your first class citizen, I would avoid GCP.
I would only consider them if I need a massive scale of something they are cheaper on, that can quickly be migrated elsewhere.
If you sign a service contract with Google (which you will if you are spending any amount of serious money), then you will have a precisely defined guarantee that they will continue operating GCloud.
Google Cloud Platform (GCP) is a bit of a joke. I really don't think it is all that competitive. Its trying to stay caught up and they do offer a lot of similar services. But their documentation is all over the place. Their service administration is a disaster. You can tell that each product is built by separate teams that only meet together once a quarter over Zoom. Its very disjointed. GCP is definitely trying to play catch-up. That isn't to say that you couldn't build a powerful platform ontop of GCP, many respectable companies are doing it. Discord is one that comes to mind that is (or at least was...) built entirely on GCP. I don't want to say it is shit. But its not as cohesive as Azure and AWS.
Basically AWS and Azure are the two core competitors. I really think that in practical (non-niche, using core services) they are neck and neck. GCP is in the race, but definitely lagging behind the others.
It's never about technology. It's about cost. If AWS ever priced themselves wildly above Azure or GCP or ___, shit would hit the fan.
Outside of that, who cares if you're locked into one cloud or another? Features? Maybe if you're super picky about Power BI or SQL Server? It's never made any sense to me.
S3 Lambdas look like a solid addition to AWS.
If someone hosted an MP3 piracy site or torrent tracker or carding forum on AWS, it would likely terminated too, regardless of whether Amazon are ideologically against piracy.
If you're a big fish with powerfu rich users like Twitter, then vendor lock-in isn't a risk. If you're a small, inconsequential platform that can be smashed like a fly on the window, then you really need to care about such things.
Parler outright refused to delete posts calling for the mass killing of politicians, even when those posts were specifically pointed out to Parler by AWS. This is all detailed in court documents. AWS very patiently tried to prod Parler into following AWS's terms of service, but Parler refused. This is a very clear-cut matter and those suggesting otherwise are either trying to deceive or being misled.
Twitter also uses GCP, and Azure I’m sure. Their serving layer is not hosted by AWS though, and I’m reasonably certain AWS could not shut down Twitter immediately simply terminating their AWS account.
If you’re building something that a cloud provider might adopt or build their own, well, isn’t that risky regardless of the cloud provider element?
I know some companies avoid one provider or the other out of competitive concerns. That seems easily managed with contractual obligations.
Twitter has enabled some borderline illegal behavior, but the account holder has been the one at risk. I don’t think Twitter systematically allows illegal behavior like parler did.
https://aws.amazon.com/aup/: You may not use, or encourage, promote, facilitate or instruct others to use, the Services.. for any.. offensive use, or to transmit, store, display, distribute or otherwise make available content that is.. offensive.
What does offensive mean? Who defines it?
Does this sound libelous to you, and do you have evidence supporting this? Even AWS's response didn't allege Parler "systematically allowed illegal behavior".
All the slippery slope arguments really need at least one questionable cloud de-platforming before they will hold any significance...
The idea was to avoid vendor lock and make it "easy" to change the database engine.
A project was on, perhaps 2 years ago, still followed this pattern, but it ended up also adopting somewhere between 10 - 15 AWS services and we never discussed an abstraction layer on top of it.
There are one or two efforts now to create an independent layer on top of cloud services. I have a hard time understanding how that will work outside of basic services.
I have seen this fail in dozens of ways
I’m designing a system right now and I’m specifically not allowing it to become too ingrained into the AWS ecosystem so that I can easy move it to GCP, Azure, or elsewhere (would that Digital Ocean offered better support) if/when Amazon gets too big and starts to take advantage of their size. That will happen. It’s only a matter of time.
AWS is for privledged companies playing with silicon valley monopoly money based on made up "evaluations", most people don't have that benefit.
More companies are moving to the cloud from managed colo than the inverse. For example, companies like Uber and Adobe has been moving from their own datacenters to AWS. Even dinosaurs like Disney have moved to the cloud.
Anyone who saves a significant amount of money running their own DCs vs uses the cloud either has significant security requirements or is the in the business of selling infrastructure (ex. Dropbox)
Everyone is waking up to the "total cost" of software development. Market forces are forcing people to realize that developer time is the most valuable commodity there is.
Companies don't go to the cloud because they have silicon valley money to burn. They go to the cloud because every minute of their developer time that is not spent on development operations and maintainance of code that a cloud provider offers instead, is a minute they can spend on the core UNIQUE business proposition of the company.
You can bemoan the skills lost in our industry, but that always happened at every layer of abstraction in every technical industry.
Not every platform needs EC2 instances. Most can build on Lambdas, API Gateway, and S3.
Not sure what evidence supports this notion of billions and privilege. If anything, AWS has democratized entrepreneurial endeavors.
> If anything, AWS has democratized entrepreneurial endeavors.
More likely AWS simply made a lot of money on the race to the bottom (or top) with regards to an ever faster, fatter, fancier and shinier Web. (Now everyone needs a CDN, everyone needs many PoPs, because the market forces demand this. Of course the total costs are enormous, but due to the increased scale and decades of technological changes the unit costs have gone down.)
> Most can build on Lambdas, API Gateway, and S3.
Sure, and it's great if people are aware of what kind of workload to put where. And if they build in rate limits. Otherwise they will be surprised when they run into the classic gotchas of "how we burnt all of our seed money on AWS bills".
A simple colocated server nowadays can do almost a petabyte a month for less than $1000. Lookup how much that is on AWS.
Lambdas and s3 are cool until you get ddos'd and now owe way more money. Or, your software doesn't work with lambda, which is true of the overwhelming majority of services.
If you price right, then if you scale to the point you need it for ongoing costs, you'll have money to spend on the one-time cost of building an abstraction layer over any locked-in services and switching to that, enabling both a migration off and, potentially, its own resellable product.
Could also be a way to mess with people -- for example, you could have an image src that gives different results depending on the recipient.