It's worth noting that there is currently a gotcha with S3 Event Notifications such that they are not guaranteed. As a result, you may end up missing out on events.
I wouldn’t trust S3 events to lambda. Sure lambda supports retries and a dead letter queue but you can’t reprocess the data.
A much more resilient approach would be:
S3 event -> SNS Topic -> SQS Queue -> lambda.
and set up a dead letter queue for the SQS queue.
It doesn’t help with the reliability of S3 events (and I’ve never seen that happen), but it does help if their is an error running your lambda.
Move the S3 object after processing it. As long as you move it to a bucket in the same region, there aren’t any charges.
Then if you are really paranoid, you can have a timed lambda that checks the source S3 bucket periodically and manually sends SNS messages to the same topic to force processing.
S3 also doesn't provide a linearizable consistency model or even a vague approximation of one. You can't rely on the events you try to schedule happening in the order you try to schedule them in, or even happening at all.
This seems overcomplicated compared to using a regular timed event to trigger a lambda and having it decide what to execute conditionally.
If only; this is AWS. You pretty much need premium support just to tell them their products are broken. Before someone says "forums", the forums are a joke.
If I interviewed someone who explained that as a solution, it would definitely not help their case. The worse hire is someone who overcomplicates projects. It leads to “negative work”.
We're taking this serverless thing too far. We're coming up with elaborate overcomplicated schemes to accomplish simple tasks just so we can say "we're not running a server for this".
Instead of doing the simple thing and running a server process that periodically checks some queue for tasks to execute, this "scheme" involves triggering a lambda function every minute, to do some IO operations on a distributed file store where the execution date metadata is embedded in the filenames to see if there is some task to execute. If there is, some more distributed IO operations are required to move the file to another part of the distributed file storage, where some listener will notice a new file was added and trigger the actual lambda function with the task-specific code...
But hey, you're not running any servers yourself...
Don’t blame Serverless. Blame it on a poor implementation. AWS Cloudwatch supports cron rules that include the year that it should be scheduled and it can trigger lambdas directly. There was no reason he had to over complicate this.
As far as using a queue, lambda also supports triggering based on a queue.
Not saying it's right or wrong solution, but saying that a simpler solution would be to take a server and put on it a job scheduler process is misleading. People shouldn't forget that it never ends with a single server and a single process, you have to manage it to make it production ready and fault tolerant.
Sometimes doing the extra step to make it serverless compatible is worth the effort.
Like Amazon's new SQS Lambda trigger you can then schedule a serverless function that's triggered by new items in the queue to have arbitrary scheduling of tasks.
Using their Python API it's pretty nice:
from datetime import datetime, timedelta
from azure.servicebus import ServiceBusService, Message
sbs = ServiceBusService('task-queue', shared_access_key_name=key_name, shared_access_key_value=key_value)
d = datetime.utcnow() + timedelta(minutes=1)
task = {"some": "object"}
sbs.send_queue_message(
"task-queue",
Message(
task,
broker_properties={'ScheduledEnqueueTimeUtc': d}
)
)
Dynamodb can also be used as a one-time scheduler and I will say it will look also a lot simpler than this example.
In dynamodb you can set TTL for a record. When the record expires it will fire event to your designated lambda. That's it. You simply write a record, wait for the record to expire and you get notified on your lambda.
Correct me if I'm wrong, but s3 unlike dynamodb does not require a specific region, therfore it's more fault tolerance or it requires more work on dynamodb to achieve same level of stability
CloudWatch Events already supports triggering Lambda functions on a cron schedule. You don't even need S3. This seems like a ton of work to basically invoke a function every n minutes.
If you want a reoccurring task then you are right, but if you want to run a one time task in a specific time in the future depending on internal logic, then it won't work
This isn’t using S3 as a scheduler. Cloudwatch already supports cron and rate expressions, as the post alluded to. This is a hack to schedule something once.
All Cloudwatch would have to do is implement a recurrence = 1 feature but I’m guessing it's not a common enough use case.
You would have to delete the rule. In theory, it’s just like deleting an item from a queue once it has been processed. Another advantage is that you could look in the console to see which overdue rules haven’t been deleted.
And if you really need more than 50 rules, it’s just a soft limit. You send a request to support and they will raise it.
And as with everything overly flexible, people will abuse it and build over-engineered solutions on top of it, when simpler solution is already present, but possibly not obvious or poorly documented.
43 comments
[ 2.9 ms ] story [ 94.9 ms ] threadYou can measure the impact and potentially automate recovery of missed events by:
1) Keep a track of events published.
2) Generate an S3 inventory daily.
3) Compare events received to objects listed in inventory.
You rarely end up with fewer events than objects, but it does occur.
I've personally only observed this with SNS target, but due to it being a problem with S3, I believe Lambda can fall afoul of this too.
A much more resilient approach would be:
S3 event -> SNS Topic -> SQS Queue -> lambda.
and set up a dead letter queue for the SQS queue.
It doesn’t help with the reliability of S3 events (and I’ve never seen that happen), but it does help if their is an error running your lambda.
Move the S3 object after processing it. As long as you move it to a bucket in the same region, there aren’t any charges.
Then if you are really paranoid, you can have a timed lambda that checks the source S3 bucket periodically and manually sends SNS messages to the same topic to force processing.
This seems overcomplicated compared to using a regular timed event to trigger a lambda and having it decide what to execute conditionally.
Mainly just calling out a gotcha where you might quietly miss out on scheduled events with no warning.
For example:
1. Object written to successfully to bucket.
2. `s3:ObjectCreated:Put` is _never_ delivered.
The possibility of duplicate events are warned about a lot in the AWS ecosystem, and this sets up an expectation of "at-least-once" delivery.
If only; this is AWS. You pretty much need premium support just to tell them their products are broken. Before someone says "forums", the forums are a joke.
The title should be "Using s3 as storage and some side thingy that runs every minute as a scheduler".
Instead of doing the simple thing and running a server process that periodically checks some queue for tasks to execute, this "scheme" involves triggering a lambda function every minute, to do some IO operations on a distributed file store where the execution date metadata is embedded in the filenames to see if there is some task to execute. If there is, some more distributed IO operations are required to move the file to another part of the distributed file storage, where some listener will notice a new file was added and trigger the actual lambda function with the task-specific code...
But hey, you're not running any servers yourself...
As far as using a queue, lambda also supports triggering based on a queue.
Waking on items in queue would be better, though, and AWS has it built-in; the authors just did not use it for some reason.
Or you create a rule for a specific date / time. Any reason why that would not work?
Like Amazon's new SQS Lambda trigger you can then schedule a serverless function that's triggered by new items in the queue to have arbitrary scheduling of tasks.
Using their Python API it's pretty nice:
- set a CloudWatch scheduled rule to schedule a lambda.
- or set a rule to send an sns, subscribe a queue to the sns topic and subscribe a lambda to the queue.
Agree regarding sns, but in our case sns == s3
Why use S3? S3 events aren’t guaranteed. SNS was specifically designed for this use case.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/S...
In dynamodb you can set TTL for a record. When the record expires it will fire event to your designated lambda. That's it. You simply write a record, wait for the record to expire and you get notified on your lambda.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/S...
Now I kinda want to try that myself.
All Cloudwatch would have to do is implement a recurrence = 1 feature but I’m guessing it's not a common enough use case.
So when doing one-time events (by specifying a year) - the triggered lambda need to delete it.
Or have a hourly scheduled lambda to delete already fired one-time events.
Or maybe CloudWatch is so smart it can delete them by itself?
And if you really need more than 50 rules, it’s just a soft limit. You send a request to support and they will raise it.
And as with everything overly flexible, people will abuse it and build over-engineered solutions on top of it, when simpler solution is already present, but possibly not obvious or poorly documented.
https://docs.aws.amazon.com/cli/latest/reference/events/put-...
The data you attach will be included in the SNS message or the data that gets sent to the lambda handler.
You can put static JSON as an event from either the AWS Console, the CLI, or CloudFormation.