DataSwitch’s DataMaps: A Unified Approach to Data Traceability and Transformation

DataSwitch has unveiled a new product called DS DataMaps, which aims to build traceability of data and transformation flow from data pipelines, helping to re-engineer the data platform and providing explainable data flow for data producers and consumers to enhance adoption and trust in the data platform.

However, challenges arise when transitioning to modern data platforms like data lakehouses, which necessitate significant data restructuring and technological changes. 

DataSwitch CEO Karthikeyan Viswanathan described that there are three types of data migration and transformation—rehosting, replatforming, and reengineering. According to him, reengineering is the most extensive transformation which involves changes to infrastructure, technology, and the data structure itself.

“When changing infrastructure, I’m moving from on-premise to cloud. I’m changing from one technology to another technology. However, during reengineering, I’m changing the infrastructure, I’m changing the technology, and also changing the data structure,” he added.

Reengineering, the most intricate of these approaches, involves a comprehensive overhaul of infrastructure, technology, and data structure. He explained that this allows customers to feed more data to the AI systems for analysis and other applications. 

The Role of DataMaps in Data Restructuring

DataMaps provides essential data traceability and lineage, which are indispensable for successful reengineering.

“I need to find the traceability of data. Then only I can restructure my data in the new world,” Viswanathan said. The tool operates through a series of key functionalities, including metadata extraction, which consolidates metadata from various data pipelines into a common format. It also provides end-to-end traceability, identifying links between data elements to offer comprehensive job level and end-to-end lineage.  

Furthermore, DataMaps utilises predictive mapping, using pre-configured data models and automatic mapping to generate code for the new platform. Its polyglot code generation capability allows the creation of code for diverse platforms, such as Snowflake or Databricks, based on specific customer needs. 

Citing an example, he said that users can modify how data is structured. For instance, if a field called ‘Party’ in the old system needs to be renamed ‘Client’ in the new system, the tool makes this transformation easy.

To facilitate testing and validation, the tool features synthetic data generation and live unit testing. Finally, it provides visual pipeline outlining, offering a clear representation of the data transformation process.  

“Think about the simple analogy of Google Maps,” Viswanathan said. “We apply the same concept to data.” He explained that just as Google Maps builds a knowledge graph to determine the best routes, DS DataMaps builds a knowledge graph to track data movement across systems

Moreover, DataMaps integrates with DataSwitch’s existing product suite, including DS Migrate and DS Citizen. DS Migrate manages infrastructure and technology migration, while DS Citizen handles data modeling and code generation. DataMaps serves as the central tool for data restructuring during re-engineering.

The comprehensive tool suite from DataSwitch maintains data quality and reduces mistakes by 70%, enabling more reliable strategic choices. Its plug-and-play solutions simplify the transformation process and accelerate project delivery by 60%. 

The launch of DataMaps highlights Dataswitch’s broader strategy of addressing challenges in data engineering and operations. As companies adopt cloud-native architectures, the demand for intelligent lineage and migration tools continues to grow.