Sounds like dlt as soon as we merge this data contract Pull request. Are you shilling for us without knowing :) ?
- auto deploy - we have CLI 1 line deploy to airflow or git actions. Embeddable in any orchestrator.
- standardised ingestion with auto schema inference, typing, normalisation and evolution
- Validate and generate data contracts for input and output data assets - PR in review.
- shift left: there is no more left than before ingestion, this is it.
- data catalogs and lineages: We can currently register tables in catalogs. Ingestion row and column level lineage is included. Schema is exportable and can be stored somewhere like github for reuse as contract.
That's how I found the article, really grateful for it! But i mean we are going for those principles and I am glad to see the alignment in thinking!
this is the PR https://github.com/dlt-hub/dlt/pull/594, it contains confiburable evolution between hard contract (row must match), soft contract (data row must contain contract row) or configurable partial or complete evolution
Sounds like dlt as soon as we merge this data contract Pull request. Are you shilling for us without knowing :) ?
- auto deploy - we have CLI 1 line deploy to airflow or git actions. Embeddable in any orchestrator.
- standardised ingestion with auto schema inference, typing, normalisation and evolution
- Validate and generate data contracts for input and output data assets - PR in review.
- shift left: there is no more left than before ingestion, this is it.
- data catalogs and lineages: We can currently register tables in catalogs. Ingestion row and column level lineage is included. Schema is exportable and can be stored somewhere like github for reuse as contract.
https://dlthub.com/docs/intro
thank you!
I mentioned you in the data lineage section 😉
I didn't know about the latest feature, and it looks really good. I'll take a look and maybe prepare an article in the future for my Substack.
Thanks for the summary! 🙌🏼
That's how I found the article, really grateful for it! But i mean we are going for those principles and I am glad to see the alignment in thinking!
this is the PR https://github.com/dlt-hub/dlt/pull/594, it contains confiburable evolution between hard contract (row must match), soft contract (data row must contain contract row) or configurable partial or complete evolution
- Adrian