There is a world outside data engineering, in Category Theoretical Mathematics, that's more efficient & powerful. Welcome to a world without ETL with always-on 100% fidelity to source data, metadata.
The moment ETL comes into play, the resulting data set is forever disconnected from source data and meta data. Of course one can add a hundred stitches, tools, AI to connect it back. But is this the only way? Is there a more effective and elegant to way achieve the same outcome without ETL? Especially, with disconnected metadata, anything relying the ETL’d data is stale in context. This is where Category Theoretical approach, designed for structure modelling and preservation, comes in handy. This is an attempt to relook at data modelling at a fundamental level - creating a foundation for ETL-free, always-on data integration with 100% fidelity to source data, data model, metadata and any related changes.
Read further here https://surecan.substack.com/p/integration-off-abstract-data-model
Comments