Data Lineage in Context of Interactive Analysis

• October 17, 2018
• apache-spark - read more about Content about Apache Spark, an open-source distributed computing system that provides fast and flexible data processing capabilities.data-lineage - read more about Data lineage refers to the ability to trace the origin, movement, and transformation of data as it flows through an organization's systems and processes.data-extraction - read more about Data extraction refers to the process of retrieving data from various sources, including web pages, and APIs, and transforming it for analysis and reporting.data-mesh - read more about Building Distributed Data Meshes since 2016 as a full-time job.cybersecurity - read more about Cybersecurity is the practice of protecting computer systems, networks, and sensitive information from unauthorized access, theft, and damage.java - read more about Java is a widely used, object-oriented programming language known for its portability, scalability, and security.adyen - read more about

This presentation focuses on tracking data lineage for interactive data exploration through notebooks. A set of techniques is shown to demonstrate how to audit data journey from code entered in notebook down to levels of execution planning, DataFrames, RDDs and Hadoop’s file formats back to visualizations displayed to data analyst. Custom tool has been created using Java Instrumentation API, that allowed us to add extra security to certain parts of Spark driver’s JVM runtime environment.

See Also

• Reverse-Engineering a Search Language (talk)
• Building Our Data Science Platform With Spark and Jupyter (external)
• Reflecting on the Year 2023
• Unlocking the Power of Databricks SDKs (talk)
• Strategies for Data Quality With Apache Spark