Effective patterns for putting your data to work on Azure
The Spline open-source project can be used to automatically capture data lineage information from Spark jobs, and provide an interactive GUI to search and visualize data lineage information. We provide an Azure DevOps template project that automates the deployment of an end-to-end demo project in your environment, using Azure Databricks, Cosmos DB and Azure App […]
Various interactive options are available to inspect messages in Azure Event Hubs or IoT Hubs, such as Azure Portal, Service Bus Explorer (Windows only) or Azure IoT Explorer. However, for developer convenience or scripting automation, you may prefer using a CLI tool to stream the messages directly into a console or file. The Kafka Console […]
A simple way to unit test notebooks is to write the logic in a notebook that accepts parameterized inputs, and a separate test notebook that contains assertions. The sample project https://github.com/algattik/databricks-unit-tests/ contains two demonstration notebooks: The normalize_orders notebook processes a list of Orders and a list of OrderDetails into a joined list, taking into account […]
