Great read. Although we (@ http://helloinsurance.substack.com) can see where you are coming from, in our view point we see Medallion and Data products as complementary, not competing, parts of a modern data landscape.
This diagram (https://imgur.com/a/sgnJ8HP) shows how they both coexist in an insurance context. Medallion ensures consistency and governance, while domain teams build data products with the flexibility and latency they need.
Would love to hear your take: where do you see this hybrid model working well, and where might it run into trouble?
I don't see medallion architecture as the culprit in this. This is all about governance, and with tools that track lineage, you can become very efficient with what you do or do not need. Data mesh is a framework built for people and processes; it is not a tool. I see the medallion as the foundation on which you apply data mesh. There are no set rules on where your users have to pull their data. ML/DS pull from silver and even bronze. Data Marts can and do pull from silver. You can also sub-divide your layers, there are no major rules. This is all predicated on how mature an organization is, and anything is better than siloed Excel sheets unless we want to try to start the "edge data architecture.".....
exactly, especially if you need to time travel and run audits in regulated environments. Or backfill any data or push it into quarantine. This gives tremendous benefit having raw data accessible, especially for a self service systems. Although, some hybrid variations are possible..like normalisation if data uploaded into bronze storage, at least we pull there data we are going to use.
Great illustrations and practices to achieve better data solutions. I don’t see the big difference in structure. Both are in some way a multi-hop / layered / medallion architecture. The difference is as you explain it, in the approach to data modeling were business centric is the key.
How someone could write such a lengthy article and not use the word 'mart' anywhere in the convo is beyond me. Let me give you a clue .. there's not one silver there's many. Not one gold, many. That's why people are so confused by medallion - they always miss that one important point. There is no reason for sales to be looking at HR data, is there!!??
Great read. Although we (@ http://helloinsurance.substack.com) can see where you are coming from, in our view point we see Medallion and Data products as complementary, not competing, parts of a modern data landscape.
This diagram (https://imgur.com/a/sgnJ8HP) shows how they both coexist in an insurance context. Medallion ensures consistency and governance, while domain teams build data products with the flexibility and latency they need.
Would love to hear your take: where do you see this hybrid model working well, and where might it run into trouble?
I don't see medallion architecture as the culprit in this. This is all about governance, and with tools that track lineage, you can become very efficient with what you do or do not need. Data mesh is a framework built for people and processes; it is not a tool. I see the medallion as the foundation on which you apply data mesh. There are no set rules on where your users have to pull their data. ML/DS pull from silver and even bronze. Data Marts can and do pull from silver. You can also sub-divide your layers, there are no major rules. This is all predicated on how mature an organization is, and anything is better than siloed Excel sheets unless we want to try to start the "edge data architecture.".....
exactly, especially if you need to time travel and run audits in regulated environments. Or backfill any data or push it into quarantine. This gives tremendous benefit having raw data accessible, especially for a self service systems. Although, some hybrid variations are possible..like normalisation if data uploaded into bronze storage, at least we pull there data we are going to use.
Great illustrations and practices to achieve better data solutions. I don’t see the big difference in structure. Both are in some way a multi-hop / layered / medallion architecture. The difference is as you explain it, in the approach to data modeling were business centric is the key.
Too many follow the data centric approach.
Seems to fit with the Kimball Group approach which has a Staging layer and a Presentation layer. Only bring in data as needed to model business processes. https://decisionworks.com/2004/01/data-warehouse-dining-experience/
How someone could write such a lengthy article and not use the word 'mart' anywhere in the convo is beyond me. Let me give you a clue .. there's not one silver there's many. Not one gold, many. That's why people are so confused by medallion - they always miss that one important point. There is no reason for sales to be looking at HR data, is there!!??
I prefer "business process model" instead of "data mart" as espoused by Inmon. https://decisionworks.com/2000/01/design-tip-3-focus-on-business-process-not-business-departments/
You’re not wrong - pulling clean and lean requirements down the medallion architecture sounds like a great idea.
In practice is it ever going to work out that way? Where we always have the luxury to shift left on quality? Maybe some day.
Very elaborate and well explained.