Show HN: Microsoft releases Flint, a visualization language for AI agents (microsoft.github.io)
But building AI agents that can generate visualizations reliably can be very tricky:
- simple chart specs can be reliable, but generated charts are often of low quality due to reliance on system defaults; - complex chart specs with explicit details can produce good-looking charts, but they are verbose and agents can struggle with reliability
We figured out it is a limitation on the language issue (not just AI capability thing) -- current visualization languages are a bit too low-level for AI agents, requiring them to explicitly make visual decisions that are supposed to be handled by a good compiler. Flint is a visualization intermediate language to address this issue, allow AI agents to solve this last-mile human-agent interaction problem. It provides a simple semantic-type based specification, and contains a layout optimization engine that can produce good-looking charts (filled with derived low-level details) from simple high-level specs. The result is also very human understandable and adaptable. Flint powers data formulator for generating visualizations (another open source project from microsoft https://data-formulator.ai/).
Flint is available open source, and we built a MCP server that you can directly plug flint in your favorite agent app to play with data.
126 comments
[ 2.9 ms ] story [ 49.8 ms ] threadMCP setup: https://microsoft.github.io/flint-chart/#/mcp
but enterprise
Isnt graphviz there for the same reason?
Edit: I see it is using JSON as the declaration language, I am OK with llms being "good at json" but a syntax also consumable by humans it is not!
Btw, Flint is intentionally designed to allow agent skip low-level params like scale, axe, zero, step size etc (which are extremely crucial for "GOOD-looking") and they are dynamically optimized by the compiler. So AI agents can have a easier time.
Plant, Mermaid, Graphviz are all declarative textual representations designed for human authoring, JSON is made for tools. Its not a criticism just a statement that if interop across agent and human was intended this is not the simplest option.
In some composite chart examples, the good-looking echart spec is like 5x longer than the simple Flint one!
I’ve been building https://smalldocs.org for this exact reason. It’s an office suite for AI agents - but my main use case is giving a cli based LLM the canvas to express itself - charts, mermaid diagrams, etc. I’ve extended it a bit further to be a format for all types of work so the agent can embed slides and spreadsheets in a document.
Sample document: https://smalldocs.org/blogs/what-is-a-smalldoc
Source: https://github.com/espressoplease/smalldocs
It sounds like cool work, but you shouldn't be using HN in this way. Linking to your project occasionally in a relevant context is fine, but this should be a (small) minority of what you post to HN.
From https://news.ycombinator.com/newsguidelines.html: "Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity."
and then that spec would be rendered either to a Bubble TUI via NTCharts or to HTML/SVG via ECharts. That Echarts HTML could be naturally served by a Golang http service.
But Flint goes much deeper with semantic layers and settings optimizations. Perhaps a NTChart, or whatever terminal chart, could be a rendering target? I'll add it to the list to explore...
https://github.com/NimbleMarkets/ntcharts/blob/spec/spec/REA...
Also, I find NTChart very fun, maybe we should add NT chart to the list of compilation backend for Flint so it works in the library. Putting a reminder here: https://github.com/microsoft/flint-chart/issues/45
My understanding is that Vega was already an expressive DSL for visualizations and its probably already well spread through LLM training data.
Flint is a higher-level abstraction, with simpler much shorter spec, and the compiler derives low-level decisions so that charts are looking good.
So: flint lets agent write short program that achieving good looking charts that had to be done with lengthy program in the past.
I don't quite get what the goal of this is other than abstracting away a little bit of the complexity at the expense of flexibility. To me, the promise of LLMs is the opposite, I can get flexibility and customisation without the cost of complexity.
The intention here is that Flint is a simpler abstraction to get basic setups right and any followup edits can be done on top of the first compiled outputs (thus not limiting expressiveness). It also makes it easier for user to manipulate (like swapping axes, click to change something, which can be very hard if LLM generates a complex chart spec upfront).
But for many basic stuff your intuition is completely right.
Generally speaking, I suggest anyone interested in learning to make charts get familiar with grammar of graphics [0] libraries like Vega-Lite, Observable Plot, ggplot2, Altair. There is a bit of a learning curve if you're used to selecting chart types like in Excel, but once it clicks, it gives you virtually unlimited choices in the kinds of charts you can make. And with ggplot or Observable Plot [1], it's about the same number of lines as something like Flint.
0: https://data.europa.eu/apps/data-visualisation-guide/why-you...
1: https://observablehq.com/@observablehq/plot-gallery
A challenge with GoG is that it assumes configurations as second-class stuff, which makes it quite difficult for users to deal with things like changing formatter, scale, annotations. Flint kinda want to hide this aways (so Flint sets them on behalf of the agent or the user). But yeah, GoG is still the foundation for expressivenss.
The paper's line of reasoning seems to continue the endless subjective loop of assuming your viz framework has the right abstractions & defaults , which the next person will rightfully disagree with for their slightly different eval set
We found in practice:
- LLM's generate charts fine
- LLM's tweak charts fine
- LLM's take user feedback to tweak them fine
In that sense, going higher-level for abstractions, as is being argued for here, is strictly worse: it's better to give controls so the LLM can go deep and customize
In practice, we found the choice of json config language X vs json config language Y to be pretty equivalent across different charting systems (vega, plotly, perspective, etc), LLM's do them all fine
The harder parts were deciding what a good chart is (model, reasoning, context), and opposite of this approach, giving lower-level facility for doing user change requests on tweaks, interactivity, and tricky in practice, when they have a lot of data on it.
This is exactly why this is an intermediate language designed to get 95% stuff right easily (for expressiveness and reliability purpose), while 5% of more advanced case where the agents need to revise chart for other purpose can be done easily on top of the compiled low-level spec (low in terms of Vega-Lite etc, not SVG). We are not really designing a higher abstraction to replace existing ones.
In the past, the split is like 50% good at first run for some common stuff, all other stuff requires agent-loop or user involvement.
Our goal is to make it easy for most case, not everything needs a full multi-round trip agentic workflow to solve. :)
We are kinda all advanced users in fact, for a lot of users, they are easily get confused with the first time result if that is not as good, and the interactivity cost / multi-round isn't an option.
Functions extremely well and the result is a very clear (and consitent) human-readable "output layer." Cool idea, fun to see people converging on similar concepts in the space.
I find that besides training better models, designing new language for agents is also a super viable paths to improve their performance!
N of only a few of us working on an analytics agent, I don't think we've been finding this to be the case. We've been impressed with just how good LLMs (even smaller open weight models) are at using Python and R for visualization. Often any shortcomings go away if we iterate a bit to about ambiguity. Are there any threads of research that could better support this claim or highlight where issues might be?
Or at least, maybe that's the idea?
IME, Claude and ChatGPT do just fine generating ggplot models, but extensive customization can get a bit hairy.
A deterministic layer like a compiler or generator of code with some kind of IR that the LLM generates and feeds it with.
I feel we will be seeing this more and more in the near future.
I don't want to use an agent at all, but i wouldn't mind generating some charts with an easy-to-generate markup language...
The only reason to use this instead of existing, mature ones designed for humans is if you are an AI agent.
It is only better than nothing for the purpose of showing it to people whether produced by an LLM, by a human, or by both.
Your employees may just accept the internal slop, but at some point, you have to show your charts to your customer.
anything more than their own handpicked examples and you're better off using d3 or yfiles. layering, clustering, boundaries, rearranging are all basic needs for text to diagrams. None support it.
Both suck at being any good for rendering diagrams from readable structured text. There is a gap to be addressed.
either way the number of people willing or compelled to learn it will be tiny. and it hence becomes a niche, perfect for a long-term side project but with no real return.
my conclusion was to stop looking and use D3 or custom code.
Good news is that AI do make new language a bit more accessible than before! If your agents can use it well and you can steer easily, it will naturally be good adoption.
Presumably Microsoft thinks this one is better. Why? And why would that answer be any different for a human vs an LLM?
> Instead of requiring verbose low-level parameters such as scales, axes, spacing, and layout, the Flint compiler derives optimized chart settings from the data, semantic types, chart type, and encodings.
Interesting to see a Chinese University collab for once.
https://www.chartjs.org/
https://echarts.apache.org/en/index.html
https://vega.github.io/vega/
.....
.....
https://github.com/microsoft/flint-chart/blob/main/packages/...
Ok, Microsoft is conflating two different things here: LLMs don't really care about code being low level and verbose, they can read things like Assembly and SPIR-V just fine: visualization is the real issue in that LLMs have no natural understanding of spatial composition through visual comparison because they literally "see" things differently than humans, so the way to get around that is provide them with "visualization" in code form that they can easily reason about and understand, so basically anything that's not deeply nested and has hidden states that they have to reason about.
Also, Flint being stringly typed in JSON is a decision that I don't think I agree with. Looking at the actual spec, this could have just been a normal, human usable TypeScript library, and it would have been 100x better. Using their own example (excuse the formatting):
type SemanticType = "Category" | "YearMonth" | "Profit";
type ChartType = "Heatmap" | "BarChart" | "LineChart" | "ScatterPlot"; // extend as needed
interface ChartEncodings { x: string; y: string; color?: string; size?: string; tooltip?: string; }
interface ChartProperties { colorScheme: string; [key: string]: unknown; // allow other optional properties }
interface ChartSpec { chartType: ChartType; encodings: ChartEncodings; chartProperties: ChartProperties; }
type SemanticTypes = Record<string, SemanticType>;
interface ChartConfig<TData = Record<string, unknown>> { data: TData; semantic_types: SemanticTypes; chart_spec: ChartSpec; }
// The actual typed object literal: const chartConfig: ChartConfig = { data: {}, // replace with your actual data shape/type semantic_types: { game: "Category", period: "YearMonth", newUsers: "Profit", }, chart_spec: { chartType: "Heatmap", encodings: { x: "period", y: "game", color: "newUsers", }, chartProperties: { colorScheme: "redblue", }, }, };
Related: https://www.openui.com/blog/stop-making-ai-write-json
Make something people want.
Agents, npm, typescript, MCP. All buzzwords are there. Will anyone look at the slop charts? Of course not, the tokens are the goal.
MSFT stock is at 2024 levels. Maybe someone should produce a flint chart and present the agentic work to Nadella. No one buys this AI slop any more.
I'm not sure if Flint is the right tool for me. I'd like to have a tool that expresses code in visual form for me. For example, right now I need to reverse engineer some code for debugging purposes.
I already found out there are three tasks:
Visually it's easily expressed: 3 bubbles lined up with 2 connections between the neighboring ones.Which ML tools suited best for that?
> @mermaid-chart
The functionality of the vscode-mermAId extension was merged into vscode FWIU? From https://code.visualstudio.com/updates/v1_109#_mermaid-diagra... :
> Mermaid diagrams in chat responses
> Chat responses can now render interactive Mermaid diagrams with the renderMermaidDiagram tool. This lets models use flowcharts, sequence diagrams, and other visualizations to visually break down complex concepts. The diagrams are interactive, so you can pan and zoom to explore them in detail, or open them in a full-sized editor for easier viewing.