Author of the blog here, curious what a better alternative would be in this context. The channel has to be passed around for the producer and consumer to interface with each other. Are there better patterns for this?
To get higher throughput we would need one goroutine to pull from the replication slot while the other is pushing to the target. The idea is to keep the Postgres connection useful and reading the slot while also pushing to the target asynchronously.
Because then you are consuming, or producing, you can’t do both at the same time. You are either reading from a stream of data, or you are writing it. Using goroutines to separate these allows you to do both at the same time, as soon as data is available on the channel or you receive the signal to stop.
Not the parent, but I personally dislike it when Go libraries use channels in their public APIs, as it forces a specific concurrency model on the consumer; in particular, channels are quite slow, being protected by an internal mutex, so you're always paying for the overhead no matter if you need it or not.
You also have to be very careful about managing the channel lifecycle. If you're not pulling (selecting from) the channel, the library will be permanently stuck. So you must now have a way to tell the library to stop sending, and it must cancel any in-flight send operations if you call producer.Stop() or whatever. In my experience libraries often have bugs in their channel code. It's far too easy to get deadlocks with channels that have interdependencies, and you have to be very careful about buffered versus unbuffered channels, as they behave differently.
A better API, in my opinion, is to offer a callback or single-method interface. Then the implementer of that callback or interface can choose to use channels internally if they desire, or they can use something else. You get the same backpressure support since you can treat it as synchronous.
After all, a channel's send interface is essentially just:
type Channel[T any] interface {
Send(T)
}
But a "chan T" doesn't offer this flexibility.
My rule of thumb for channels is that they're goroutine glue, not an API primitive. Build APIs out of interfaces, not channels. The only thing that uses channels should be the one that's controlling the goroutines, because it's the thing that orchestrates them.
That said, it's not a hard rule. There are places where channels may have their place in a public API, though I'm not sure I can think of any examples off-hand.
I think it is a matter of preference. For me personally I use raw channels and goroutines all day every day and I really like using them. Channels are a core primitive in golang so I think it is worth getting familiar with them.
As you say being able to select is really nice too.
> as it forces a specific concurrency model on the consumer
I found your excuse above is really nonsense. when your program is in Golang, you've already picked side, the concerned concurrency model has already been chosen by the user.
we are not talking about one of random concurrency models, we are talking about channel based sychnronization and communication in golang, if you don't want that and consider it as an issue, you shouldn't be using golang in the first place.
Looks like the channel field is private in CDCRecordStream, but exposed by GetRecords. The callers mostly loop over Record objects. [1]
If I wanted to encapsulate iterating over a channel of Records, maybe it would be something like Go's io.Pipe function [2], which returns a PipeReader and PipeWriter? Except that it would work on Records rather than byte streams.
I don't have enough context to know if the extra encapsulation is a good idea in this case, though.
Except it itself is exposed via a public API. Context interface also exposes a chan
Done() <- chan struct{}
What are you trying to say? That using a chan as a function argument can shoot yourself in the foot? The whole idea behind channels is to not shoot yourself in the foot with mutex's, deadlocks, etc. What are you trying to say? Wrapping a chan in a type handle is safer than passing a chan of a type? Nope.
I think the rule is: don't use a channel unless at some point you need a select with 2+ branches. The context.Done is a case in point. You're not just waiting around for a context to finish. You do two things: try some network request and wait to see if the context cancels before it finishes. Because it's made to work in a select block like that, there wasn't another, better thing to use than a channel for context.Done. But for just iterating through some stream, you take a context in and return an iterator object that will do all the channel stuff internally.
That's not at all what I was referring to. It's not merely having channels as function arguments. Channels broadly have footguns around how closed and nil channels are treated and you can run into work contention/starvation issues accidentally, since they do have their own throughput limits. And wrapping them in your own handle types that's specific to the library's use case allows you to deal with those issues and swap out the implementation of the cross-goroutine bits without breaking the downstream API.
Channels broadly have footguns around how closed and nil channels are treated
I appreciate your concerns but one could say similar things about nil pointers or accessing slices out of range. Anything you use should be handled with care.
No. Being the first class element in the go language, channel is nothing different from a []byte or a regular struct.
The whole purposes of having such built-in channel as a part of the core language is to encourage everyone to use it whenever possible, the reason is dead simple - it is much less likely to shoot yourself in the foot.
Well, I think read-only channels in public APIs are fine. Writable or write-only channels are dangerous and should be avoided given that the "close ownership" -- to coin a phrase -- is ambiguous. For the write end, I agree that a handle type is appropriate.
I wonder if they have a setup without Docker. Not a fan, wonder why they even bother with Docker if they're using GO. Thats one of the great things about GO. Easy distributable that works on all platforms.
Not all of our code is in Go. PeerDB has multiple components: the workers, the UI and the query layer. Some of which is in Rust, Go and Typescript.
While it would certainly be possible to package them into individual binaries, I found it significantly easier to define the stack in a docker compose file with the requisite environment setup.
Java developers tend to produce more value since they can write the code in a more concise way using Lambdas, functional interfaces, record classes, etc. Overengineering like in your example above can be done in any language, but shouldn't.
I strongly disagree with this view for applications, and mildly disagree for infrastructure projects.
There is one thing I believe Go is by far and away the leader at: keeping codebases maintainable over time.
Go is designed very clearly with this goal and it's excellent at it. The verbosity you complain about makes it much, much clearer what something is doing.
That coupled with some of the baked in opinions means it is significantly more straightforward to pick up code that you or someone else wrote a few years ago and modify it.
See, this is a problem I see many junior engineers struggle with: Excessive desire to make one's own life easier at the cost of customers.
Yes, your codebase is nice and elegant and maintainable. But you haven't shipped as many features as the Python or Java developer, haven't delivered as much value to customers. Ultimately, the users suffer. It's a form of selfishness.
Agreed. We should make life easier for our users. That's why a language like Python or Java, with a faster development time, where we can fix bugs faster and ship features faster, is vastly preferable.
How do you deliver your python package ? How does the end user fetch dependencies ? A proxy ? Pip ? Jfrog ? System packages ? Is he root ? Or behind a corporate proxy / firewall ?
Between Go and Rust for infra, let's say that Go is the #1, Rust if far far behind. Most recent infra tools are built in Go, Kubernetes, Docker, Prometheus, Grafana, Terraform etc ...
Yes, SIMD is possible. But what about the other more obscure hardware features there's no nice library for? It's much easier to use those in Rust than Go.
Author of the blog here: I'm a huge proponent of Rust, but in this project while interfacing with BigQuery and Snowflake go proved to be the right choice. There weren't official drivers for these in Rust and also generally onboarding new devs in Go was easier. I also personally think async in rust has a few more releases to go before I would consider it stable.
I have extensive experience with Go, Python, Java, and Scala. I write Scala and Java at my day job, and find I am far more productive in Go than either of these.
Your framing of the issue in terms of infrastructure projects vs. applications seems bizarre and simplistic. For example, one of the areas where Go shines is in highly concurrent, io-bound, networked services. Many examples of good use cases for Go in the wild: Kubernetes, CockroachDB, TiDB, etc.
I don't think choosing channel itself is an issue, if it works for the said platform, that is great.
What I don't understand is why there should be such a blog article telling people that you decided to use channel in your go based program? This is like writing an article telling people that you decided to use the MMU when building an OS.
Thanks for posting this question. As explained in the article we started off with pull/push model (with configurable batching) and it worked well for streaming to data warehouses, where 30s+ was acceptable latency.
We added Queues as a supported target, where one of our users wanted single digit second latency. This is when we introduced channels. We agree that it is a small change. But the latency improvements were significant and wanted to share it with broader programming/Go community of Go Channels and the objective impact they can have.
Pull/Push itself is not the problem unless you have stats to back such implied claims. It is far more likely that your highly inefficient Pull/Push implementation caused the problem. Without any real insight on why the pull/push based approach is slower, there is nothing the "Go community" can get.
All valuable details were omitted, e.g. 1) how much data is being moved? a few megabytes? what is the state of art latency? maybe 0.001 second for that? when latency is still seconds huge, what can be further improved, in the go runtime or in the apps? etc
A highly efficient pull/push model without using go channel can easily beat your 1-5 seconds latency. A crappy implementation using channel can get worse.
Delay of several seconds is when you pass on information mouth to mouth, in 2023, that kind of delay doesn't justify a blog article on such fancy implementation. I'd suggest you to delete the implementation to rebuild from ground zero.
Thanks for the comment. We will aim to write a more deeper blog on this in the future.
We were observing 60<>40 ratio for Pull/Push, as specified in the blog and majority of the Push was target data-store (network) bound. So there was less room of optimization on the Go side.
The test was done with ~10-15K TPS on Postgres and the target was Azure Event Hubs, which has a limit of 1MB write batch size.
52 comments
[ 2.9 ms ] story [ 116 ms ] threadYou also have to be very careful about managing the channel lifecycle. If you're not pulling (selecting from) the channel, the library will be permanently stuck. So you must now have a way to tell the library to stop sending, and it must cancel any in-flight send operations if you call producer.Stop() or whatever. In my experience libraries often have bugs in their channel code. It's far too easy to get deadlocks with channels that have interdependencies, and you have to be very careful about buffered versus unbuffered channels, as they behave differently.
A better API, in my opinion, is to offer a callback or single-method interface. Then the implementer of that callback or interface can choose to use channels internally if they desire, or they can use something else. You get the same backpressure support since you can treat it as synchronous.
After all, a channel's send interface is essentially just:
But a "chan T" doesn't offer this flexibility.My rule of thumb for channels is that they're goroutine glue, not an API primitive. Build APIs out of interfaces, not channels. The only thing that uses channels should be the one that's controlling the goroutines, because it's the thing that orchestrates them.
That said, it's not a hard rule. There are places where channels may have their place in a public API, though I'm not sure I can think of any examples off-hand.
you can always wrap channels to make them worse and less capable, but your API should expose the more capable option.
As you say being able to select is really nice too.
I found your excuse above is really nonsense. when your program is in Golang, you've already picked side, the concerned concurrency model has already been chosen by the user.
we are not talking about one of random concurrency models, we are talking about channel based sychnronization and communication in golang, if you don't want that and consider it as an issue, you shouldn't be using golang in the first place.
If I wanted to encapsulate iterating over a channel of Records, maybe it would be something like Go's io.Pipe function [2], which returns a PipeReader and PipeWriter? Except that it would work on Records rather than byte streams.
I don't have enough context to know if the extra encapsulation is a good idea in this case, though.
[1] https://github.com/search?q=repo%3APeerDB-io%2Fpeerdb%20GetR... [2] https://pkg.go.dev/io#Pipe
you can't abstract over channels easily unless you're missing the point of channels.
I appreciate your concerns but one could say similar things about nil pointers or accessing slices out of range. Anything you use should be handled with care.
The whole purposes of having such built-in channel as a part of the core language is to encourage everyone to use it whenever possible, the reason is dead simple - it is much less likely to shoot yourself in the foot.
While it would certainly be possible to package them into individual binaries, I found it significantly easier to define the stack in a docker compose file with the requisite environment setup.
https://github.com/Hello-World-EE/Java-Hello-World-Enterpris... is an excellent demonstration of this.
There is one thing I believe Go is by far and away the leader at: keeping codebases maintainable over time.
Go is designed very clearly with this goal and it's excellent at it. The verbosity you complain about makes it much, much clearer what something is doing. That coupled with some of the baked in opinions means it is significantly more straightforward to pick up code that you or someone else wrote a few years ago and modify it.
Yes, your codebase is nice and elegant and maintainable. But you haven't shipped as many features as the Python or Java developer, haven't delivered as much value to customers. Ultimately, the users suffer. It's a form of selfishness.
If I write code more slowly that others find easier to maintain, that is selfish?
What’s the deployment story like ? Here’s a hot take of mine: we should treat our users as master and make their lives easier, not ours.
That’s a hot mess to manage.
Between Go and Rust for infra, let's say that Go is the #1, Rust if far far behind. Most recent infra tools are built in Go, Kubernetes, Docker, Prometheus, Grafana, Terraform etc ...
Your framing of the issue in terms of infrastructure projects vs. applications seems bizarre and simplistic. For example, one of the areas where Go shines is in highly concurrent, io-bound, networked services. Many examples of good use cases for Go in the wild: Kubernetes, CockroachDB, TiDB, etc.
https://www.cockroachlabs.com/blog/why-go-was-the-right-choi...
What I don't understand is why there should be such a blog article telling people that you decided to use channel in your go based program? This is like writing an article telling people that you decided to use the MMU when building an OS.
We added Queues as a supported target, where one of our users wanted single digit second latency. This is when we introduced channels. We agree that it is a small change. But the latency improvements were significant and wanted to share it with broader programming/Go community of Go Channels and the objective impact they can have.
All valuable details were omitted, e.g. 1) how much data is being moved? a few megabytes? what is the state of art latency? maybe 0.001 second for that? when latency is still seconds huge, what can be further improved, in the go runtime or in the apps? etc
A highly efficient pull/push model without using go channel can easily beat your 1-5 seconds latency. A crappy implementation using channel can get worse.
Delay of several seconds is when you pass on information mouth to mouth, in 2023, that kind of delay doesn't justify a blog article on such fancy implementation. I'd suggest you to delete the implementation to rebuild from ground zero.
We were observing 60<>40 ratio for Pull/Push, as specified in the blog and majority of the Push was target data-store (network) bound. So there was less room of optimization on the Go side.
The test was done with ~10-15K TPS on Postgres and the target was Azure Event Hubs, which has a limit of 1MB write batch size.