How Data Fabric fits in Big Data world

How Data Fabric fits in Big Data world

Nowadays, a new concept of building “data fabric” is getting a lot of attention. With the enormous rise of data in recent times, we have already seen an influx of data management solutions. Hadoop, as a Big data framework, is about ten years old. But it is called considered as “legacy” and “monolithic.” So what is data fabric and how well does it live up to the hype? Let’s try to find out.

Data fabric is a unified platform for storing, integrating, and managing all your data within a single environment. Isn’t other solutions like Hadoop, data lake, data warehouse also provide the same?
Well, Hadoop and data lake are distributed file systems that bring disparate data together. The integration of complex and large datasets, which are petabytes, is challenging. In comparison, Data warehouses are suited for structured data. They are relatively slow and inflexible and cannot handle today’s complex demands.

Data Fabric provides seamless data integrations by combining various datasets and data structures. Regardless of where the data resides and whether it needs data virtualization, ETL, cleansing of data to improve data quality, data fabric can integrate data for any use.

As the name suggests, data fabric is a weave that is stretched over an ample space that connects multiple locations, types, and sources of data to access that data. The data can be processed, managed, and stored as it moves within the data fabric. The data can also be obtained by or shared with internal and external applications for a wide variety of analytical and operation use cases for all organizations.

Unlike Hadoop, which keeps the storage with compute, data fabric keep the storage in a separate pool and bring compute to the data. Data Fabric does that by running distributed computing in containers. Multiple containers can be spun off as the demand require. The compute containers are managed and orchestrated by the Kubernetes service.

The Data Fabric Stack

A data fabric has four layers:
Storage layer: The data is stored at the lowest level in the storage pool. It can support any variety of data in large sizes.
Data Services layer: Through a set of API, security, accessibility, data movement, and governance are handled
Control Panel: Tools to control, manage, and optimize the workloads are managed here. Kubernetes service also lives in control plain.
Data Analytics layer: Service data analytics, AI/ML needs.

Main features of data fabric:

Data fabric provides the following features:
★ Pre-packaged connector to connect to any data without the need for manual programming
★ Data integration and ingestion capabilities to store the data
★ Able to bring multiple environment s like on-premises data, public cloud, private, and hybrid cloud under one platform
★ Support structured and unstructured, batch and real-time data
★ Data governance capabilities to govern accessibility and quality to build trust
★ Extensive API support to service any data request

By leveraging Kubernetes and APIs to service any data request, irrespective of whether it generated internal, external, on-premise, or cloud, data fabric architecture becomes the latest choice for the distributed demands of the modern data ecosystem. Whether it will live up to the hype is yet to be seen.

This blog is originally published on Medium.