Hey HN! Just wanted to share a side project I've been hacking on. It's a news ingestion tool that's ridiculously cheap to run - we're talking less than a buck for a million articles.
Why I built it
I got fed up with the insane costs of existing news ingestion services. Plus, I wanted something I could tweak and scale easily. So I rolled up my sleeves and built this bad boy.
How it works
The secret sauce is a Lambda architecture combined with SQS queues. Here's the gist:
Lambda functions scrape news sources and dump article metadata into SQS queues.
Another set of Lambdas processes these queues, fetching full article content.
Processed articles get stored in S3, with metadata in DynamoDB.
Why it's so damn cheap
Lambdas only run when there's work to do. No idle servers burning cash.
SQS queues act as a buffer, smoothing out traffic spikes without over-provisioning.
We're leveraging AWS's generous free tier limits across multiple services.
The result? Pennies on the dollar compared to traditional setups or SaaS solutions.
Tech stack
AWS (Lambda, SQS, S3, DynamoDB)
Python (because life's too short for verbose code)
BeautifulSoup & Newspaper3k for content extraction
Open invitation
The repo is public, and I'd love for you all to check it out, star it if you like it, and maybe even contribute. Whether it's adding new features, optimizing performance, or just fixing my inevitably messy code, all PRs are welcome!
https://github.com/Charles-Gormley/IngestRSS
Let me know what you think! Has anyone else tackled this problem differently? I'm always down to learn new approaches or optimizations.
P.S. If this picks up steam, I might write a more detailed blog post about the architecture and cost breakdown. Let me know if that's something you'd be interested in!
Also this is my first post on hacker news so hopefully I'm not breaking any rules.
2 comments
[ 3.4 ms ] story [ 14.2 ms ] threadLambda functions scrape news sources and dump article metadata into SQS queues. Another set of Lambdas processes these queues, fetching full article content. Processed articles get stored in S3, with metadata in DynamoDB.
Why it's so damn cheap
Lambdas only run when there's work to do. No idle servers burning cash. SQS queues act as a buffer, smoothing out traffic spikes without over-provisioning. We're leveraging AWS's generous free tier limits across multiple services.
The result? Pennies on the dollar compared to traditional setups or SaaS solutions. Tech stack
AWS (Lambda, SQS, S3, DynamoDB) Python (because life's too short for verbose code) BeautifulSoup & Newspaper3k for content extraction
Open invitation The repo is public, and I'd love for you all to check it out, star it if you like it, and maybe even contribute. Whether it's adding new features, optimizing performance, or just fixing my inevitably messy code, all PRs are welcome! https://github.com/Charles-Gormley/IngestRSS
Let me know what you think! Has anyone else tackled this problem differently? I'm always down to learn new approaches or optimizations. P.S. If this picks up steam, I might write a more detailed blog post about the architecture and cost breakdown. Let me know if that's something you'd be interested in!
Also this is my first post on hacker news so hopefully I'm not breaking any rules.