Yes. I've worked a couple of jobs that had a lakehouse with transforms written in notebooks. Usually sitting on a data lake, with processing handled by a Spark cluster. I've much preferred it to the ETL tools of yore, even if Spark can throw some weird curveballs.
That's on the warehousing side, however. On the reporting side, I've only seen notebooks catch on with small research teams, a la Jupyter Lab or something similar. Most of the hype today is around self-service BI tools, not tools that require hosting and a working knowledge of Python, Scala, or whatever.
Depends bit on audience and if you are managing other parts outside of notebooks. Like pipelines dags etc. notebooks can be tricky if you stuff too many responsibilities into them.
8 comments
[ 461 ms ] story [ 553 ms ] threadThat's on the warehousing side, however. On the reporting side, I've only seen notebooks catch on with small research teams, a la Jupyter Lab or something similar. Most of the hype today is around self-service BI tools, not tools that require hosting and a working knowledge of Python, Scala, or whatever.