Skip to content

Databricks Apps (data/AI app runtime)

Category: App-Runtime · Areas: ui, api, infra

Description

Category

app-runtime

Areas

ui, api, infra

Slot

deploy-target

Platform

Platform-specific (Databricks). Databricks Apps is Databricks’ framework for building and hosting interactive data/AI applications natively on the Data Intelligence Platform. HELIX already treats Databricks as a known runtime ecosystem (the databricks-genie install target, docs/install/databricks-genie.md); this concern is the app-hosting member of that family.

Boundary

This concern owns how an interactive data/AI app is hosted, identified, and wired to data on Databricks — the managed serverless runtime, the app service-principal identity, OAuth auth, resource bindings, and the rule that data access flows through Unity Catalog. It must not duplicate its neighbors:

  • frontend-framework (e.g. react-nextjs, or a supported Python UI framework) owns what the UI is built with — components, routing, styling, the client experience. Databricks Apps owns where it runs and how it is identified and granted. The UI framework runs inside this runtime; this concern does not specify component patterns.
  • Generic deploy-target owns deploy mechanics for self-hosted infra (containers you operate, a cloud you provision). Databricks Apps is the Databricks-specific deploy target: a managed serverless runtime that eliminates that infra. When this concern fills deploy-target, do not also stand up parallel self-hosted hosting.
  • unity-catalog owns the data governance the app reads through. This concern owns that the app reads through it (and how the app’s identity is granted); it does not restate the catalog/grant model.
  • security-owasp / auth own application-layer auth semantics. Databricks Apps supplies the platform identity layer (app service principal + OAuth, on-behalf-of-user authorization); compose, do not duplicate the app’s own RBAC.

Components

  • Managed serverless runtime — the app runs as a containerized web service on Databricks serverless compute. Each app has its own configuration, identity, and isolated runtime; Databricks supplies the hosting (no separate infra to provision). Billed per hour of running compute.
  • Supported frameworks — Python: Streamlit, Dash, Gradio (and other Python web frameworks such as Flask/FastAPI); JavaScript/Node: React, Angular, Svelte, Express. The chosen framework fills frontend-framework and runs inside the runtime.
  • app.yaml manifest — declares the startup command, environment variables, and the resources the app binds to (SQL warehouses, model serving endpoints, jobs, secrets, Unity Catalog volumes, Genie spaces). Dependencies: requirements.txt/pyproject.toml (Python) or package.json (Node).
  • App service principal — Databricks creates a service principal per app; it is the app’s own identity (shared across all users) for resource access.
  • OAuth 2.0 auth + two identity models:
    • App authorization — actions run as the app’s service principal (shared permissions for all users).
    • User authorization (on-behalf-of) — the end user (who must belong to the Databricks account and signs in via SSO) authorizes, so per-user permissions — including Unity Catalog grants — are enforced.
  • Resource bindings — an app binds to existing resources (declared in app.yaml); it cannot create them. Workspace admins review the requested permissions at deploy; Databricks enforces least-privilege.
  • Data persistence options — in-memory cache (session-only, lost on restart), local filesystem (ephemeral), Databricks/Unity Catalog tables (persistent), Unity Catalog volumes, workspace files, or Lakebase (managed Postgres) for transactional app state.
  • Networking & isolation — inherits the workspace’s networking protections; isolated per-app runtime; encryption in transit and at rest.

Constraints

Databricks hosts the app — no parallel self-hosted infra

  • The app runs on the managed serverless runtime; this concern fills deploy-target. Do not provision separate hosting (your own containers, cluster, or cloud) for the same app — that defeats the runtime and splits the identity/governance model.
  • In-memory and local-filesystem state is ephemeral (lost on restart); durable state goes to Unity Catalog tables/volumes or Lakebase, never to the app’s local disk.

Identity is the app service principal or the on-behalf-of user — chosen deliberately

  • Resource access runs as either the app service principal (shared) or the signed-in user (on-behalf-of). Pick deliberately: use user authorization when per-user Unity Catalog grants must be enforced; use the service principal only for shared, app-owned actions. Do not silently run all data access as the service principal when the product needs per-user governance.
  • The app binds to existing resources via app.yaml; it does not create resources or self-grant. Requested permissions are admin-reviewed and least-privilege.

Data access flows through Unity Catalog

  • The app reads and writes lakehouse data through Unity Catalog governance (via SQL warehouses / governed tables / volumes), under the appropriate identity’s grants — not around it via hardcoded cloud paths or embedded credentials (see unity-catalog).

The UI framework is a separate slot

  • The framework (Streamlit/Dash/Gradio/React/…) fills frontend-framework and composes inside this runtime. This concern owns hosting/identity/data wiring, not component patterns.

Drift Signals (anti-patterns to reject in review)

  • A Databricks-targeted app given its own self-hosted infra (separate cluster/cloud/containers) in parallel → host it on the Databricks Apps serverless runtime (this concern fills deploy-target)
  • Durable app state written to the app’s local filesystem or in-memory cache (lost on restart) → persist to Unity Catalog tables/volumes or Lakebase
  • All data access run as the app service principal when the product needs per-user governance → use user (on-behalf-of) authorization so Unity Catalog grants are enforced per user
  • The app reaching data around Unity Catalog (hardcoded cloud path, embedded credential) → read through governed tables/volumes/SQL warehouse under grants
  • Resources expected to be created by the app → bind to existing resources in app.yaml; provision them out of band
  • Treating Databricks Apps as the UI framework and skipping a real frontend framework → fill frontend-framework (Streamlit/Dash/Gradio/React/…) inside the runtime

When to use

A product that is an interactive data or AI application hosted natively on Databricks — a dashboard, data/AI tool, or agent UI that lives next to the lakehouse and serves Databricks-account users. Selection signal: the product targets the Databricks lakehouse / is a data+AI app on Databricks. It fills the deploy-target slot (Databricks hosts it); compose with frontend-framework (the actual UI framework runs inside the runtime), unity-catalog (the data it reads is governed there), and databricks-declarative-pipelines (when the data it reads is produced by declarative ETL). areas: ui, api, infra scopes its practices to the UI, service, and hosting work items.

Do not select it for an app hosted off Databricks, or one with no Databricks account/lakehouse — use the generic deploy-target and frontend concerns there.

Artifact Impact

Selecting this concern requires these artifacts to change (a selected concern absent from them is drift):

  • ADR: auth model (app service-principal vs on-behalf-of-user) + durable-state store (UC tables/volumes vs Lakebase)
  • TD: managed serverless runtime, app.yaml manifest + resource bindings, identity model, data access via Unity Catalog
  • IMPLEMENTATION_PLAN: app.yaml command/env/resource-bindings; bind to existing resources (cannot create)

ADR References

Record an ADR for the auth model choice (app service principal vs on-behalf-of-user) — it is design-defining for per-user governance — and for the app’s durable-state store (Unity Catalog tables/volumes vs Lakebase). A material uncertainty (workspace networking, account membership, resource-binding permissions) is a tech-spike, not a silent assumption (see workflows/references/concern-resolution.md).

Practices by activity

Agents working in any of these activities inherit the practices below through runtime work context, such as a DDx bead context digest.

These practices govern hosting, identity, and data wiring for an interactive data/AI app on the Databricks managed serverless runtime. They do not govern UI component patterns (that is the frontend-framework filler) or the catalog/grant model (unity-catalog) — see the boundary in concern.md.

Requirements (Frame activity)

  • Confirm the product is hosted natively on Databricks and its users belong to the Databricks account (SSO).
  • Decide the auth model: app service principal (shared) vs on-behalf-of-user (per-user Unity Catalog grants). This is design-defining — record it.
  • Identify the resources the app needs (SQL warehouse, model serving endpoint, jobs, secrets, Unity Catalog volumes/tables, Genie space) and which groups own them.

Design

  • Choose the UI framework (Streamlit/Dash/Gradio/Flask/FastAPI or React/Svelte/Express) — it fills frontend-framework and runs inside the runtime.
  • Design the app.yaml: startup command, environment variables, and the resource bindings (bind to existing resources; the app cannot create them).
  • Design the durable-state store as Unity Catalog tables/volumes or Lakebase (managed Postgres) — never the app’s local disk or in-memory cache.
  • Design data access to flow through Unity Catalog (SQL warehouse / governed tables / volumes) under the chosen identity’s grants.

Implementation

  • Deploy to the managed serverless runtime — no separate self-hosted infra.
  • Declare dependencies in requirements.txt/pyproject.toml (Python) or package.json (Node); declare command, env, and resources in app.yaml.
  • Use on-behalf-of-user authorization where per-user governance matters; use the app service principal only for shared, app-owned actions.
  • Persist durable state to Unity Catalog tables/volumes or Lakebase; treat in-memory/local-filesystem state as ephemeral.
  • Read/write lakehouse data through Unity Catalog — no hardcoded cloud paths or embedded credentials.
  • Request least-privilege resource permissions (admin-reviewed at deploy).

Testing / Verification

  • Verify the app runs on the Databricks serverless runtime (deployed app URL reachable), not a self-hosted stand-in — observed, not assumed.
  • Verify the chosen auth model behaves correctly: under on-behalf-of-user, a user without a Unity Catalog grant is denied the data (negative control); a granted user succeeds.
  • Verify durable state survives an app restart (it is in Unity Catalog/Lakebase, not in-memory).
  • Verify the app binds to existing resources (no create-on-deploy) and permissions are least-privilege.

Boundary with neighbors

See concern.md for the canonical Boundary (vs frontend-framework, generic deploy-target, unity-catalog, security-owasp / auth). Composition in the Databricks family: this concern hosts; databricks-declarative-pipelines produces the data; unity-catalog governs it — each owns its piece, none restates the others.

Quality Gates

  • The app is hosted on the Databricks Apps managed serverless runtime (no parallel self-hosted infra); the deployed app URL is reachable.
  • A real UI framework fills frontend-framework and runs inside the runtime (the app is not “Databricks Apps as the UI”).
  • Data access flows through Unity Catalog under a deliberate identity (app service principal vs on-behalf-of-user); for per-user governance, a user without the grant is denied (negative control).
  • Durable state lives in Unity Catalog tables/volumes or Lakebase and survives a restart — not in-memory/local-filesystem.
  • The app binds to existing resources via app.yaml with least-privilege permissions (admin-reviewed); it creates no resources and embeds no credentials.