I created this tool for myself because I often work with xlsx and csv files. This is usually done through Python Pandas. But if you just need to read these files and work with the values in the lines, you don't need to import the whole Pandas library. It is not even necessary to install Pandas for easier deployment. This can save you up to 2GB space if you do docker images with Pandas.
One of the things I still don't understand are services like snyk.io, which are supposed to do security analysis. But they penalize a tool like this for not having CoC, Contributing in the GitHub repository, and what is most shocking to me is that they measure Popularity. I understand that if more people are involved in the SW, it is probably safer. But penalizing someone for having few stars on GitHub seems weird to me. Especially when the tool is used by several people / companies and it has over 5,000 downloads.
Not OP nor well-versed with any of this, but the standard csv module doesn't parse xlsx files. Since the majority of people who work with spreadsheets use excel [citation needed], this probably greatly increases the application potential of the library.
> Thanks for removing some boilerplate from that process
Do you mean "boilerplate" or something else? Because a shell script (e.g. bash with calls to sed, awk, grep, and Perl) is not "boilerplate". It's an implementation. And much more efficient than some of the unstable, high-complexity solutions that claim to be "simple".
TBH, the choice of "solution" doesn't matter much for a small dataset. It is of course overkill to run a glorified REPL just to do some math on a small dataset.
As datasets grow, the introduction of unstable complexity can cause problems.
The tooling that you say is being removed may well be the the fastest and most reliable tools, proven over decades of use.
I know what you mean. Now it returns a field with dictionaries. Repeating keys as such is not possible and this is the desired state. But yes, with namedtuple you could manoeuvre more and add line numbers for example. Good idea, I'll think about it in the future.
I usually import CSVs into a Python Pandas Dataframe and then iterate over the dataframe in a loop or manual line by line interventions and then beam the data out somewhere else...
Here it is the same. But with the difference that you don't have to install a relatively "large" Pandas library. It always depends on what you want to achieve. If you just read the lines and values, you can always use something small to help. If you want to Dockerize similar solution, you have the difference that Pandas needs specific system libraries and it will increase your Docker Image to almost 2GB compared to sheet2dict where it is a couple of KB.
It is true. I usually use it in tandem with flask / fast-api to parse both: CSV and XLSX. In the near future I will add XLS (yes, someone still uses it) and also the open document format.
openpyxl has good xlsx support and is easy to use. This seems to be an abstraction layer on top of both. Unfortunately this will use a lot of RAM working with large files.
You won't be able to use this if your file doesn't fit in RAM. This unnecessarily clones the file into a list instead of returning a generator and leaving the list conversion up to the user.
Nice, I did something like this and made it a gist ages ago for XLS, it's good you're putting up something that's more maintained and working with multiple formats.
Speaking of excel files, Does anyone know of a good way to port sheets with equations/functions to python? Sometimes I need a calculations from a sheet and I have to manually copy them over.
26 comments
[ 2.2 ms ] story [ 74.3 ms ] threadOne of the things I still don't understand are services like snyk.io, which are supposed to do security analysis. But they penalize a tool like this for not having CoC, Contributing in the GitHub repository, and what is most shocking to me is that they measure Popularity. I understand that if more people are involved in the SW, it is probably safer. But penalizing someone for having few stars on GitHub seems weird to me. Especially when the tool is used by several people / companies and it has over 5,000 downloads.
[0] https://docs.python.org/3/library/csv.html
Thanks for removing some boilerplate from that process for people!
Do you mean "boilerplate" or something else? Because a shell script (e.g. bash with calls to sed, awk, grep, and Perl) is not "boilerplate". It's an implementation. And much more efficient than some of the unstable, high-complexity solutions that claim to be "simple".
TBH, the choice of "solution" doesn't matter much for a small dataset. It is of course overkill to run a glorified REPL just to do some math on a small dataset.
As datasets grow, the introduction of unstable complexity can cause problems.
The tooling that you say is being removed may well be the the fastest and most reliable tools, proven over decades of use.
Is this a better approach?
Isn’t a big point of dataframes providing tools that are more efficient so you don’t have to use Python loops for operations across a body of data?
Xlsx I know nothing about.
https://github.com/Pytlicek/sheet2dict/blob/main/sheet2dict/...
https://github.com/jmcnamara/XlsxWriter
https://github.com/capitalone/DataProfiler
My question, how do you do header detection? That's a _very_ difficult problem.
https://github.com/h2oai/datatable/blob/385a9b370db32de90135...