1) Not limited to pd.Series -- any type works. So if you have a dataset or a way of expressing types it should just work. That said, pd.Series/DataFrames are convenient, and we have some special tooling around them. We're looking at tooling to add more expressiveness to datasets. Something like koalas type-hinting (there are a few approaches) might be helpful in this: https://koalas.readthedocs.io/en/latest/user_guide/typehints.... With the type-equivalency check we added this should be fairly extensible.
2) Yes! Although not too extensively. I think OpenLineage would be a pretty simple thing to add -- just have to get the abstractions right.
3 comments
[ 3.9 ms ] story [ 15.1 ms ] threadTwo questions:
1) What’s the story on type hints? I see pd.Series in the examples but have you done anything to offer more specific typing?
2) Have you considered pluggable extensions for data lineage platforms or supporting OpenLineage?
1) Not limited to pd.Series -- any type works. So if you have a dataset or a way of expressing types it should just work. That said, pd.Series/DataFrames are convenient, and we have some special tooling around them. We're looking at tooling to add more expressiveness to datasets. Something like koalas type-hinting (there are a few approaches) might be helpful in this: https://koalas.readthedocs.io/en/latest/user_guide/typehints.... With the type-equivalency check we added this should be fairly extensible.
2) Yes! Although not too extensively. I think OpenLineage would be a pretty simple thing to add -- just have to get the abstractions right.