As Platform Engineers, we’re often tasked with enabling autonomy for data teams without letting costs spiral out of control. In the world of Databricks, cluster policies are one of the most powerful (and underrated) tools to achieve that balance.

In this post, I want to focus on a key capability of cluster policies: enforcing the separation between interactive (all-purpose) clusters and automated job clusters.

💭 Why Should You Care?

All-purpose clusters are intended for interactive, exploratory work like notebooks. But it’s tempting—and dangerous—to re-use them for scheduled jobs.

Why?

  • ❌ They stay alive longer, often leading to idle resource costs.
  • ❌ They can lead to unexpected behaviors when shared with multiple users and tasks.
  • ❌ They bypass the monitoring and tagging you may have set up for job clusters.

As a result, your costs can quietly creep up, and your architecture becomes harder to reason about.

🔧 Enforcing It in Cluster Policies

Let’s say we want to ensure that only notebooks can run on a given cluster policy—and prevent users from misusing it for scheduled jobs.

Here’s the relevant part of the Databricks cluster policy:

{
  "workload_type": {
    "type": "fixed",
    "value": "all-purpose",
    "hidden": true
  },
  "workload_type.clients.jobs": {
    "type": "fixed",
    "value": false
  },
  "workload_type.clients.notebooks": {
    "type": "fixed",
    "value": true
  }
}

🚫 What Does This Do?

  • "workload_type.clients.jobs": false — Disables running scheduled jobs on this cluster.
  • "workload_type.clients.notebooks": true — Ensures this is only for interactive work.
  • "workload_type": "all-purpose" — Locks the cluster into the interactive compute mode.

Combined, this enforces strict usage boundaries and aligns well with the principle of using job clusters for automation and all-purpose clusters for ad hoc work.

💡 Bonus: Better Tagging and Cost Tracking

You can take it further by enforcing tags, such as team, owner, or cost_center, using custom_tags. This makes it easier to attribute cost to the right business unit or team:

"custom_tags": {
  "type": "fixed",
  "value": {
    "team": "data-engineering",
    "cost_center": "data-platform"
  }
}

🧩 When to Use This Policy

Use this policy when:

  • You want to give notebook users a safe and cost-controlled environment.
  • You want to prevent production jobs from piggybacking on shared clusters.
  • You want better visibility and accountability on interactive workloads.

🚀 Wrap-Up

Cluster policies aren’t just for compliance—they’re enablers for scale. By separating workloads, you encourage good behavior, save on compute costs, and make your platform more maintainable.

Have you tried locking down workloads like this? Drop a comment and let’s share war stories. 👇