Tech We Like: Managing Big Data with Azure Databricks and Unravel

The term “Big Data” has a couple of meanings. 

One is the sheer amount of unstructured data, the product of an ever-growing number of connected devices. The other reflects the fact that managing all that unstructured data in various workloads is a big challenge.

For example, keeping track of everything happening within a single cluster, especially when several teams are working at once. You need visibility into your workloads, along with the ability to manage and effectively use all the data in your pools.

In this edition of Tech We Like, we’re looking at two of the popular tools for managing workloads: Azure Databricks and Unravel.

Azure Databricks

An automated cluster management tool for artificial engineering (AI), data engineering, and data science, Azure Databricks utilizes “notebooks” to ease the process of constant and iterative Big Data processing.

For example, a data scientist or data engineer at your company can quickly fire up a cluster using Azure Databricks, work on that cluster, and then simply destroy it and move on—all through a single interface.

This not only increases efficiency, it helps keep your data from becoming cluttered and disorganized, which is the recipe for analytics failure. 

Additionally, Azure Databricks plays extremely well with GitHub and Bitbucket, making for easy collaboration. Combine all this with Azure Databricks’ integration directly into Azure Active Directory—which means security requires no real configuration—and it’s a solid choice for most organizations.

You can learn more about Azure Databricks on their website.

Unravel

In the Big Data ecosystem, there’s nothing quite like Unravel.

Where the tool really shines is in helping companies migrate to the cloud, whether it’s partially or in full. Not only does it provide visibility into clusters and the workloads happening within them, it goes a step further and actually identifies the best workloads to migrate.

Even for work with clusters on premises, Unravel provides a big assist by making it easy to sort through competing workloads and compute needs. It also helps resolve conflicts between other tools, such as TensorFlow and Keras.

Perhaps most impressive is Unravel’s ability to do all this while still providing a single pane of glass of visibility across all your workloads, so you always know which ones should be where and what’s being run on them.

You can learn more about Unravel at their website. And for more on Big Data and the many tools available within its ecosystem, check out our free resource Big Data Made Simple.

Note: This post was originally published in 2019 and has recently been updated.