Why you need a Data Mesh Architecture

Asim Razvi

While we have discussed Fabric architecture to bring your data into the modern world where time to value is key, there is a new architecture leveraging mesh that is worth considering. While there are similarities there are key differences that might warrant your time to review what can now be done.

The concept behind a data fabric is to bring data together in a solid non silo’d architecture. The idea is to move data into analytic platforms quickly while keeping the data available for everyone. This lets you mix and match data sets as needed. The approach is to become data powered rather than data driven, in essence rely on the data for business growth, asset valuation, and the ability to move to new markets. While being data driven works fine for most companies the ones experiencing rapid growth even during a pandemic are those that can leverage their data as a strategic asset. 

To gain traction here the data fabric or data supply chain concept has worked really well for companies that are starting up and bringing new products and services to the market. The issue has been that well established companies that have extensive legacy systems and complex regulated sets of data that need to be kept separate cannot adapt quickly to the new paradigm without almost a complete redo of the architecture which in some cases is too expensive to make sense. Enter the advantages of the Mesh Network. 

To grasp how the Mesh architecture works think of it as a giant fishing net that you throw out and capture all that data that is currently swimming in that data lake. You can leave the data lake as is, you are bringing everything into a net for review. The advantage is that this can become a parallel effort as you erect the other components. The data integration can be done through a virtualized model and data sets can be managed on separate nodes. The centralized hub will be responsible for managing the governance policy and the nodes can be managed by the domains themselves. 

The reasons and advantages to this model are as follows. Data itself needs to be curated and processed by those that understand it and if you have centralized area for that then likely they will not understand all of it.  In a decentralized model you move the capability ownership into the domains that understand the data and thus can manage it, regulate it and leverage it correctly. The upside is that through virtualization you can remove the silos and make that data available but keep ownership in the domains. The advantages are that you move the data from being a source into a managed product. A product then becomes an asset, and this brings you into a valuation model for your company that is based on the value of your data.  

So to reiterate, a mesh network can bring enterprise level agility to a large company that is having difficulty adapting to a competitive set of vendors using AI/ML and predictive analytics to take advantage of a changing market. If you are interested in how this can work drop me a note and we can flesh out the details, or you can wait for my next more detailed article on the architecture and a working example.  

Latest Posts

-->