Building Robust Python Applications on Top of Databricks

• June 12, 2024
• databricks - read more about I'm working at Databricks since 2019 and here I share my experience with integrating and building the ecosystem around it.product launch - read more about As a driven and accomplished professional, I have had the pleasure of leading and driving numerous product launches throughout my career.open source - read more about My personal weighted opinions about Open Source Software - from building it as a full-time job to using it.python - read more about Python is a popular high-level, open-source programming language used for a wide range of applications, including data science, machine learning, and more.code linting - read more about

Discover how to build robust Python applications on Databricks by leveraging lessons from Terraform, UCX, and the Python SDK. Learn to utilize Databricks Labs UCX to identify incompatible code with Unity Catalog , automate fixes, and migrate extensive Hive Metastore Tables, maintaining permissions and updating cluster settings.

This session covers essential topics like failure recovery, rate limits, logging, multithreading, and unified client authentication, illustrated with real-world examples. Gain practical insights into handling API call errors, background thread issues, and Databricks Workflow failures. Explore PyTest for creating reproducible sandbox scenarios in shared Workspaces, enabling simultaneous verification by multiple developers. Additionally, discover other Databricks Labs tools to manage complex challenges effectively.

See Also

• Unlocking the Power of Databricks SDKs (talk)
• Building Databricks Integrations With Go (talk)
• Introducing Databricks Labs Sandbox (external)
• Reflecting on the Year 2023
• Upgrading Between Databricks Runtime Versions Made Easier (external)