We see a lot of demand for onboard transformations, and JS is our first trial, since most devs are familiar with it. We plan to look into Rust and all of the Wasm supported languages, based on user demand
Yup, as of now we are leveraging Nodejs for the technical preview. We also have another preview using embedded v8 for inline transforms, that is your transform runs in-process and apply() boils down to a function call. We are still evaluating other runtimes like wasmer as well, v8 was initially chosen for its maturity and community support.
can materialized topics have custom properties? what if I want a longer retention in a materialized topic, perhaps I'm just counting clicks and I don't want the whole header data from that topic.
Currently they cannot, as they share all of the same properties as their respective source topics, but this idea is good! I'm adding it to our list of design enhancements.
Question for you -- what are the performance implications of "re-keying" my records during the transform? Should I try to keep the same key for my transformed record as from my original record so that they align with the same partitions, or are they likely going to get sent to different partitions across different brokers anyway?
Good question! You can't re-key your records but this done is by design. We want the materialized partitions to reside on the same brokers as their sources so there is no network overhead for performing the transform, and it occurs on all nodes. So by choosing this design we traded off more CPU for lower latency and less network bandwidth.
This is a good post but besides the non-organic upvotes, it looks like your friends/fans/teammates have seeded the thread with comments and questions. This is a super bad idea. HN readers can smell this kind of thing in the parts per million and will flame you for it. It's also against the rules—this is in both the guidelines and the FAQ:
Hiya! Excited to try this,
The devil is of-course in the details :)
What are the throughput implications for the rest of the cluster? IS there any way to profile the transforms usage as a percentage of the whole?
How much of the API is exposed? can it create topics?
Hi, of course there are :) So the transformations are deployed in a separate NodeJs process per node. You can use familiar tools to take a look at its cpu + memory usage.
As for how it affects redpanda, currently we are working on exposing metrics via Prometheus. All of this work occurs on background threads, so the only way it would affect throughput is if enough coprocessors are deployed with enough actual work that would eventually start to fill up our seastar reactors.
We have been running hundreds of services that effectively consume from various topics, perform mostly simple transformations and produce back to other topics. Providing this kind of functionality makes a lot of sense.
We are using TANK, and that functionality is not available there, but given the benefits such a feature provides, we will likely use RP as well in order to save on development time and number of distinct services we use for various events streams transformations.
22 comments
[ 3.2 ms ] story [ 61.6 ms ] threadDo you already have companies using this in production or maybe evaluating it? What language do you see most demand for?
Question for you -- what are the performance implications of "re-keying" my records during the transform? Should I try to keep the same key for my transformed record as from my original record so that they align with the same partitions, or are they likely going to get sent to different partitions across different brokers anyway?
https://github.com/scylladb/scylla/pull/9108
Folks can hear a talk on the implementation at Scylla Summit next week [free to register and yes, will be available free on demand after].
We'll also have a talk by Redpanda's Denis Rystsov.
https://www.scylladb.com/summit/
https://news.ycombinator.com/newsguidelines.html
https://news.ycombinator.com/newsfaq.html
As for how it affects redpanda, currently we are working on exposing metrics via Prometheus. All of this work occurs on background threads, so the only way it would affect throughput is if enough coprocessors are deployed with enough actual work that would eventually start to fill up our seastar reactors.
We are using TANK, and that functionality is not available there, but given the benefits such a feature provides, we will likely use RP as well in order to save on development time and number of distinct services we use for various events streams transformations.