Data at the Centre vs. Pipeline-First Data Management, What Constitutes Resiliency in Data Systems Today, and Defensive Designs that are Built to Mistrust.
The situation described in your article may be a bit simplistic. Data coordination in a pipeline is a necessary consideration for architects.
Re:
• “P2 says: ‘Was there fresh weather data this past hour?’
• If yes → enrich and proceed.
• If no → skip, wait, recheck later.
No panic. No failure. Just… pause. A calm system that has not burnt itself out.
P2 didn’t fail. It just didn’t find the input it needed yet. It’ll try again. It’s decoupled from P1’s timing or retry logic. It behaves more like a thinking agent than a dumb task on a schedule.”
If I’m adding a part to an assembly (a jet to a carburetor, for example), my logic might look like this:
Process One –
Identify and retrieve Part 123 from Parts Inventory
Was Part Retrieved
Yes -Update Parts Inventory
Pass Part 123 to Process Two
No – Wait and try to retrieve Part 123 later
Process Two
Add Part 123 to Assembly ABC and update Assembly Inventory
Processes One and Two are separate because the Part may be used in a variety of Assemblies. However, Processes One and Two aren’t truly decoupled. If my Parts Inventory is out of date because an inventory update failed, the failure can lead to problems with the Assembly Inventory in Process Two.
The process of maintaining the Data Pipeline must be compatible with the Process Outline(s) that accompany it.
The situation described in your article may be a bit simplistic. Data coordination in a pipeline is a necessary consideration for architects.
Re:
• “P2 says: ‘Was there fresh weather data this past hour?’
• If yes → enrich and proceed.
• If no → skip, wait, recheck later.
No panic. No failure. Just… pause. A calm system that has not burnt itself out.
P2 didn’t fail. It just didn’t find the input it needed yet. It’ll try again. It’s decoupled from P1’s timing or retry logic. It behaves more like a thinking agent than a dumb task on a schedule.”
If I’m adding a part to an assembly (a jet to a carburetor, for example), my logic might look like this:
Process One –
Identify and retrieve Part 123 from Parts Inventory
Was Part Retrieved
Yes -Update Parts Inventory
Pass Part 123 to Process Two
No – Wait and try to retrieve Part 123 later
Process Two
Add Part 123 to Assembly ABC and update Assembly Inventory
Processes One and Two are separate because the Part may be used in a variety of Assemblies. However, Processes One and Two aren’t truly decoupled. If my Parts Inventory is out of date because an inventory update failed, the failure can lead to problems with the Assembly Inventory in Process Two.
The process of maintaining the Data Pipeline must be compatible with the Process Outline(s) that accompany it.
John Boddie