> Every developer, more or less, uses grep, awk, and sed to quickly build a temporary analysis platform. Although they can achieve the effect of a powerful data analysis pipeline in the hands of masters, they are still too difficult for most developers.
Something is not right with this thesis. Rather than learning the basic tools of their trade, this article encourages developers to reach for the nearest black-box LLM. This approach will certainly not scale, nor be replicable across any system.
The reason grep, awk, and sed are important to learn is because they are universal. Each of those tools, being decades old, is baked into the core fabric of most operating systems today. If you have access to a shell, you can get extremely far with just those 3 tools.
It's worthwhile to spend a couple hours a night for a week or two and become proficient in each. The dividends it will pay over one's programming career will be astonishing.
In principle I agree with you, but I just wang to explore how to cluster and extract logs automated. In current we have text embedding model and large language model, it is really good at text clustering and summarization. Maybe it brings potential better tools.
And I think Text Embedding + LLM is one of the front-end of this tool. The kernel of the idea is post-extract log analysis + in-process OLAP DB + Python playground, I think add grep + awk as its another frontend is also a good idea.
How do you think about? Hope for your reply, thanks.
My understanding is that LLMs are more useful for making sense of unstructured information than making sense of structured information. Based on your explanation, I now understand why you chose to do this experiment.
In the conclusion, the next steps you described were:
- Automatically detect column types
- Parse datetime
This is pretty easy to do with existing tools for a structured data set. What value is there to having an LLM do these tasks? I'm curious about how you might use an LLM for some more interesting use cases.
Datetimes can come in thousands of formats, I think rule based tools would have a hard time detecting unknown formats perfectly. AI could be a potentially better way to parse datetimes, maybe not LLM though, what are your thoughts on this?
I haven't experienced these complex datetime situations personally. Yeah it can be a pain to figure out the formatting sometimes, but once it's known then you just roll with that format. I try to stick with unambiguous formats that are easily internationalized, such as 10-Jan-2023 instead of 10-1-2023 (could be Jan 10th or Oct 1st)
If you have a situation where there are a variety of datetime formats in a single field, then I can see where an LLM approach could be interesting. But even then, I think the priority is to migrate all of those values to a unified format. After that, add sanitation and validation on the input side and database to make sure all newly entered datetimes are the same format.
Yes, I agree with you, governance is always more efficient than post-processing. But if humans are always diligent in data governance, we do not need the idea about "data lake"...
6 comments
[ 2.9 ms ] story [ 29.2 ms ] threadSomething is not right with this thesis. Rather than learning the basic tools of their trade, this article encourages developers to reach for the nearest black-box LLM. This approach will certainly not scale, nor be replicable across any system.
The reason grep, awk, and sed are important to learn is because they are universal. Each of those tools, being decades old, is baked into the core fabric of most operating systems today. If you have access to a shell, you can get extremely far with just those 3 tools.
It's worthwhile to spend a couple hours a night for a week or two and become proficient in each. The dividends it will pay over one's programming career will be astonishing.
And I think Text Embedding + LLM is one of the front-end of this tool. The kernel of the idea is post-extract log analysis + in-process OLAP DB + Python playground, I think add grep + awk as its another frontend is also a good idea.
How do you think about? Hope for your reply, thanks.
In the conclusion, the next steps you described were:
- Automatically detect column types
- Parse datetime
This is pretty easy to do with existing tools for a structured data set. What value is there to having an LLM do these tasks? I'm curious about how you might use an LLM for some more interesting use cases.
If you have a situation where there are a variety of datetime formats in a single field, then I can see where an LLM approach could be interesting. But even then, I think the priority is to migrate all of those values to a unified format. After that, add sanitation and validation on the input side and database to make sure all newly entered datetimes are the same format.