Hopefully the upvotes and being on the front page is enough, but if not just want to say props, this is amazing. Already had it saved in my bookmarks glad to see it pop up again :)
Honestly the title doesn't do it justice. For a plotting tool it's not an impossible task, but making a webpage this responsive is incredibly hard, so well done there.
Hi, I'm very interested in this. I've been using dask/plotly for a side project but it's just too slow. If you know dask, do you think you could discuss which features of dask your project has and doesn't have?
I believe so. An optimization of Datashader is that it _doesn't_ plot all the points, but rather optimizes which points actually influence pixel values.
>The range field tells you the approximate area within which the cell could be, in metres. This is effectively an estimate of how accurate the location is.
There's one marked in the middle of a corn field near me with a range of 48,000m. ?
Kudos for directing folks to other projects if uPlot isn't fast enough for them. Every developer should take the time to write out appropriate and inappropriate use-cases on the readme.
This is awesome. I wrote some software to pull readings off my weather station and stream them to the browser. The plotting library that I use is pretty awful. Besides being slow, it's constantly hosing up the numbers on the y-axis. I want to give uPlot a shot.
Question: in the demo you load papaparse but the data is delivered as JSON?
Would you mind sharing what your backend looks like for the project where you're using this, and how do you extract all of this data at once from storage?
> Question: in the demo you load papaparse but the data is delivered as JSON?
good catch, leftovers from original PoC i made locally. the exported data was still in CSV back then. i should clean that up!
> Would you mind sharing what your backend looks like for the project where you're using this, and how do you extract all of this data at once from storage?
i just did a manual export from Analytics. how you get the data out is up to you, but a time-series or columnar database would probably be a good start :)
For a dataset like this, Highcharts would be slower for sure since it will create DOM SVG components for each of the points. <canvas> (which this is built with) is much quicker at presenting tons and tons of data points like this.
Just want to ask, because I love talking about render performance: have you tried doing this using offfscreen canvas? That should allow you to move a lot of things to a worker, so you avoid blocking the rendering with js? It probably won’t speed up the total time to finished render, but I assume it will lock the page for a shorter period of time?
If the rendering is this controlled (and sparse), blocking the thread with it isn't really a concern. And with this data size, serialization/deserialization between the main and worker thread would probably become nontrivial.
If you wanted to render these on a regular basis, WebGL is fairly straightforward, and works really well for this simple sort of rendering. You could do this with one polygon and a small fragment shader (treat the data as a texture, and use SDF to draw the line and fills), or use the actual geometry (render as a triangle strip; and separate the line if you want to do more interesting stuff in your fragment shader).
I probably wouldn't even generate an SDF, instead read the data directly from a 1D texture and fill the anti-aliased line directly in the fragment shader.
Yeah, I meant more in the abstract sense, rather than "create a two-dimensional texture of the distance field", which as you point out would be unnecessary.
You could probably sample all the 1D textures in one pass, and draw all the lines and fills there. One additional nice side effect of this is that you can easily have sequences at different resolutions.
Super cool! It would be great if you could provide some insight into how you built this and the kind of tricks you had to use to make this possible. Looking forward to a blog post in the future :)
- there is no per-datapoint memory allocation beyond whatever is necessary for the browser to construct the canvas Path2D object. this keeps the memory pressure low and the GC nearly silent.
- the amount of draw commands is reduced by accumulating the min/max data values per pixel
- uPlot does not generate axis ticks by walking the data. it only uses min/max of the x and y ranges and finds the divisions from that.
- there is no mass-creation of Date objects, data is kept in timestamp format except when hovered.
- the date/time formatting is done by pre-compiling templates and not re-parsing them all the time.
- the cursor interaction uses a binary search over the x array
it's highly dependent on what's actually drawn. i regressed it by accident and didnt notice a huge difference until i randomly opened the stress test (which is also densely packed so probably a worst case for AA)[1]. it went up by a factor of 2-4. cant remember exactly.
I find it curious that the volume of users are similar for day of the month regardless of year. For instance the first of the month is not always the same day of the week.
Eyeballing the data, there doesn't seem to be a strong weekday effect. Not sure if this is real data (human activity data usually show weekday effects).
If there was a very pronounced weekday effect, a quick hack is to use a 364-day instead of full-year offset for comparison.
I didn't even read the title before clicking and was blown away at the speed. Honestly thought something acted up since you're not used to seeing things render this fast.
interestingly, focusing on performance in web dev (and actually delivering on it) really presents new UI challenges, because users are not used to near instantaneous feedback in web-apps. you have to start adding complexity like css transitions, and UI delays just to reduce user confusion.
Would it be possible to allow for the cursor to still update when doing a horizontal scroll? When I move the mouse left and right it does that, but it doesn't move when I traverse the page by horizontally scrolling
Either way, just a bit of feedback on a beautiful page
That looks amazing, nice job! I was trying to create a seb dashboard with several graphs each with a few hundred data points recently. None of the libraries felt "snappy" on a mobile device in terms of load time or responsivity. I ended up abandoning the proejct because of that, but if only I knew about this library at the time...
The only thing I wish µPlot had is support for tooltips rather than numbers in the legend, but that's a small price to pay for this level of performance.
I was slightly disappointed when I zoomed in as I expected 3 years of data in 150ms increments (600 million measurements) but that's probably not possible
Assuming there is some culling, I’ve found myself somewhat obsessed about the methods of dataset-culling. If there’s a single outlier, for example, some methods would skip it, where it should really be highlighted.
https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z000... is a good algorithm to use to keep peaks. At a maximum you need 4x the chart width data points. In practice, you can easily reduce that. I've implemented the algorithm before and it can be done in linear time.
The 150ms benchmark in the title is really selling this short, the performance is very impressive. The 150ms seem to refer to the time it takes to initialize the graph, and with a hot cache that is more like 50ms for me here. Redrawing seems much, much faster.
I have done some visualization with WebGL because I couldn't get it fast enough with just drawing lines. That was a while ago so I'm not sure about the details, but even a simple prototype just drawing a few tens of thousands to a hundred thousand lines using canvas was slower than this for me (subjectively).
I'll have to look at the code later, but I'm curious about where this library is getting the performance from. Before I saw this I would have said you can't do this at this speed without using previously downsampled version of the data points, I'm not entirely sure now.
I'm still looking at the performance tab in the browser dev tools and how impressively empty the main Javascript usage plot is.
It looks like the mouse lines and the selection highlight are just partially-transparent divs stacked on top of the canvas that get moved around, so nothing actually gets redrawn unless the date range changes (which is a pretty clever approach!).
but really tricky to align correctly at different screen pixel densities and rounding errors. it's still not pixel-perfect, but i decided it was good enough.
That's something I expected, you really don't want to trigger redraws on mouse over. What surprised me was that I couldn't tell the the times where I zoomed in or out from the JS flame chart. Usually that is really obvious, but in this case zooming was so fast that you could hardly see it. And this graph has probably around ~194k data points (388k in the source data, I assume that's x and y). I'm not entirely sure about the number of points here, I'm taking that from the json delivered to the site.
The selection highlight is unusual, I admit. That's something I'd just skip if I were going for high performance.
Slick! I too felt that many libraries were not slick/fast enough, but I eventually ended up using react-stockcharts (in canvas mode) for my use case and was quite satisfied with its performance.
Could you compare your performance against this one if possible?
Minor suggestion: not really a bug, but inconvenient UX. When I move start the selection and I move my mouse outside the chart it stops selecting it. If you added the mouse movement listeners to the document/body then when I move my mouse outside the chart it would keep working.
117 comments
[ 3.3 ms ] story [ 70.4 ms ] threadA small part of the problem is drawing the trend charts. So I decided to make uPlot [1] to see what was really possible.
[1] https://github.com/leeoniya/uPlot
a lot of time is spent on just DOM layout and JITing the JS, which get amortized with huge datasets. the 166k bench doesnt take any longer.
The source code, which can be used as a Dash + Dask boilerplate, is here: https://github.com/plotly/dash-world-cell-towers
Feel free to ask questions at community.plot.ly
https://en.m.wikipedia.org/wiki/Stingray_phone_tracker
Credibility?
>The range field tells you the approximate area within which the cell could be, in metres. This is effectively an estimate of how accurate the location is.
There's one marked in the middle of a corn field near me with a range of 48,000m. ?
Station: https://carlisleweather.com
Software: https://github.com/chrissnell/gopherwx
Question: in the demo you load papaparse but the data is delivered as JSON?
Would you mind sharing what your backend looks like for the project where you're using this, and how do you extract all of this data at once from storage?
good catch, leftovers from original PoC i made locally. the exported data was still in CSV back then. i should clean that up!
> Would you mind sharing what your backend looks like for the project where you're using this, and how do you extract all of this data at once from storage?
i just did a manual export from Analytics. how you get the data out is up to you, but a time-series or columnar database would probably be a good start :)
Here's a sample chart with 1,000,000 points: https://jsfiddle.net/5bvLgs5w/1/
https://github.com/leeoniya/uPlot/blob/master/bench/Highchar...
Just want to ask, because I love talking about render performance: have you tried doing this using offfscreen canvas? That should allow you to move a lot of things to a worker, so you avoid blocking the rendering with js? It probably won’t speed up the total time to finished render, but I assume it will lock the page for a shorter period of time?
You could probably sample all the 1D textures in one pass, and draw all the lines and fills there. One additional nice side effect of this is that you can easily have sequences at different resolutions.
- the data is flat arrays of numbers
- there is no per-datapoint memory allocation beyond whatever is necessary for the browser to construct the canvas Path2D object. this keeps the memory pressure low and the GC nearly silent.
- the amount of draw commands is reduced by accumulating the min/max data values per pixel
- uPlot does not generate axis ticks by walking the data. it only uses min/max of the x and y ranges and finds the divisions from that.
- there is no mass-creation of Date objects, data is kept in timestamp format except when hovered.
- the date/time formatting is done by pre-compiling templates and not re-parsing them all the time.
- the cursor interaction uses a binary search over the x array
- drawing at exact aligned pixel boundaries to avoid or minimize antialiasing
this makes uPlot charts a bit rougher looking, but the perf impact is quite large.
I prefer the look of things snapped to pixels; I didn't realize it also speed things up!
it's highly dependent on what's actually drawn. i regressed it by accident and didnt notice a huge difference until i randomly opened the stress test (which is also densely packed so probably a worst case for AA)[1]. it went up by a factor of 2-4. cant remember exactly.
[1] https://github.com/leeoniya/uPlot/blob/master/bench/uPlot-60...
If there was a very pronounced weekday effect, a quick hack is to use a 364-day instead of full-year offset for comparison.
lol :(
Either way, just a bit of feedback on a beautiful page
The only thing I wish µPlot had is support for tooltips rather than numbers in the legend, but that's a small price to pay for this level of performance.
https://leeoniya.github.io/uPlot/demos/tooltips.html
https://leeoniya.github.io/uPlot/demos/cursor-tooltip.html
[1] https://leeoniya.github.io/uPlot/demos/candlestick-ohlc.html
I was slightly disappointed when I zoomed in as I expected 3 years of data in 150ms increments (600 million measurements) but that's probably not possible
I have done some visualization with WebGL because I couldn't get it fast enough with just drawing lines. That was a while ago so I'm not sure about the details, but even a simple prototype just drawing a few tens of thousands to a hundred thousand lines using canvas was slower than this for me (subjectively).
I'll have to look at the code later, but I'm curious about where this library is getting the performance from. Before I saw this I would have said you can't do this at this speed without using previously downsampled version of the data points, I'm not entirely sure now.
I'm still looking at the performance tab in the browser dev tools and how impressively empty the main Javascript usage plot is.
The selection highlight is unusual, I admit. That's something I'd just skip if I were going for high performance.
no, the top graph is pretty much what the title says: 3 * 365 * 24.
but 194k is also no problem :)
https://news.ycombinator.com/item?id=23047156
Could you compare your performance against this one if possible?
https://github.com/rrag/react-stockcharts
Great job btw, really instant.