Dump a CSV, import to SQLite. All your columns will be type TEXT and your nulls will be “” (empty string) but hey at least you didn’t do any manual work
Nice, I needed to do this task recently for large CSVs (15gb) and didn’t come across this tool.
Under the hood it manipulates the CSV using pandas data frames, then finally dumps the data frame into sqlite using `dataframe.to_sql(sqlite_conn)`. I wonder how fast this is, if it has to take each row from the data frame through Python it can’t be that quick but maybe there’s something smart happening under the hood that can punch a native-to-native pathway through the Python abstractions.
I ended up writing my own in Golang since the ones I found in C and other languages didn’t use threads or did too much memory copying.
Ha, yes, Go is probably going to be at least an order of magnitude speedup from Simon's Python based tool. I'd be curious to give your tool a test run if the code is up anywhere.
However, the project doesn't follow best practices in the JavaScript ecosystem. The included WebAssembly (wasm) file should be fetched from npm on build time, instead of being uploaded to the project.
14 comments
[ 4.4 ms ] story [ 46.3 ms ] threadIt did work a few times, but it did not or was quite slow at other times.
Good idea though.
0 - https://sql-workbench.com/
Under the hood it manipulates the CSV using pandas data frames, then finally dumps the data frame into sqlite using `dataframe.to_sql(sqlite_conn)`. I wonder how fast this is, if it has to take each row from the data frame through Python it can’t be that quick but maybe there’s something smart happening under the hood that can punch a native-to-native pathway through the Python abstractions.
I ended up writing my own in Golang since the ones I found in C and other languages didn’t use threads or did too much memory copying.
However, the project doesn't follow best practices in the JavaScript ecosystem. The included WebAssembly (wasm) file should be fetched from npm on build time, instead of being uploaded to the project.
https://github.com/nalgeon/sqlime/blob/0bb684c/js/sqlite/sql...