Now it's time to cover some elements of Azure Databricks Security. You'll take a look at data protection, identity access management and authentication, network, and compliance. Let's focus first on Data Protection, which is comprised of: Encryption at-rest, service manage keys, and user manage keys. Encryption in transit, Transport Layer Security or TLS. File and folder level access control lists for Azure Active Director users, groups and service principles. ACLs for clusters, folders, notebooks, tables, and jobs, and secrets with Azure Key Vault. Azure Databricks has separation of compute and storage. Azure Databricks is a compute platform. It does not store data except for notebooks. Clusters are transient in nature. They process the data and are then terminated. All data is stored in the customer subscription. Because the Azure storage services use server-side encryption, communication between these services and the Databricks clusters is seamless. Storage services such as Azure Storage Blobs and Azure Data Lake Storage, Gen1 and 2 provide: Encryption of data, automatic server-side encryption, in addition to the encryption on storage attached to the VMs, customer-managed keys, bring your own keys with Key Vault integration, and file and folder level ACLs in Azure Data Lake Storage Gen1 and Gen2. All the traffic from the control plane to the clusters in the customer subscription or data plane is always encrypted with TLS.3. When clusters access data from various Azure services, TLS is always used to ensure encryption in transit. When customers access notebooks via their web browsers, the connection is also secured with TLS. When enabled, authentication automatically takes place in Azure Data Lake Storage from Azure Databricks clusters using the same Azure Active Directory Identity that one uses to log into Azure Databricks. Commands running on a configured cluster will be able to read and write data in ADLS without needing to configure service principle credentials. Any ACLs applied at the folder or file-level in ADLS are enforced based on the user's identity. ADLS passthrough is configured when you create a cluster in the Azure Databricks workspace. ADLS Gen1 requires Databricks run time 5.1 or above. ADLS Gen2 requires 5.3 or above. On a standard cluster, when you enable ADLS passthrough, you must set single user access to one of the Azure Active Directory users in the Azure Databricks workspace. Only one user is allowed to run commands on this cluster when Credential Passthrough is enabled. High concurrency clusters can be shared by multiple users. When you enable ADLS passthrough on this type of cluster, it does not require you to select a single user. Access control is available only in the Azure Databricks premium plan. By default, all users can create and modify workspace objects unless an administrator enables workspace access control. With workspace access control, individual permissions determine a user's abilities. Let's explore the individual permissions and how to enable and configure workspace access control. You can assign five permission levels to notebooks and folders, no permissions, read, run, edit, and manage. A table in this module's further resources reading shows the abilities for each notebook and folder permission. All notebooks in a folder inherit all permission settings of that folder. For example, a user that has run permission on a folder has run permission on all notebooks in that folder. To enable workspace access control, you would follow these four steps. Go to the admin console, select the Access Control tab, then click the "Enable" button next to Workspace Access Control. Finally, click "Confirm" to confirm the change. All users can view libraries. To control who can attach libraries to clusters, manage access control on clusters. By default, all users can create and modify clusters unless an administrator enables cluster access control. With cluster access control, permissions determine a user's abilities. There are for permission levels for cluster; no permissions, can attach to, can restart, and can manage. A table in this module's further resources reading shows the permission combinations for each cluster ability. To control who can run jobs and see the results of job runs, manage access control on jobs. There are five permission levels for jobs; no permissions, can view, can manage run, is owner, and can manage. The can manage permission is reserved for administrators. A table in this module's further resources reading shows the permission combinations for each job ability. Table access control lets you programmatically grant and revoke access to your data from SQL, Python, and PySpark. By default, all users have access to all data stored in a cluster's managed tables, unless an administrator enables table access control for that cluster. Once table access control is enabled for a cluster, permissions can be set for data objects on that cluster. Before you can grant or revoke privileges on data objects, an administrator must enable table access control for that cluster. The Azure Databricks view-based access control model defines several privileges. Select, gives read access to an object. Create, gives ability to create an object. For example, a table in a database. Modify, gives ability to add, or delete, or modify data to or from an object. Read_metadata, gives ability to view an object and it's metadata. Create_named_function, gives the ability to create a named user-defined function in an existing catalog or database. All privileges, gives all privileges, gets translator into all the previously mentioned privileges. The privileges mentioned apply to these classes of objects. Catalog, controls access to the entire data catalog. Database, controls access to a database. Table, controls access to a managed or external table. View, controls access to SQL views. Function, controls access to a named function. Anonymous function, controls access to anonymous or temporary functions. Any file, controls access to the underlying file system. Using the secrets APIs, secrets can be securely stored, including in an Azure Key Vault or Databricks backend. Authorized users can consume the secrets to access services. Azure Databricks has two types of secret scopes; Key Vault-backed and Databricks-backed. These secret scopes allow you to store secrets such as database connection strings securely. As a best practice, instead of directly entering your credentials into a notebook, use Azure Databricks secrets to store your credentials and reference them in notebooks and jobs. To set up secrets, you would create a secret scope. Secret scope names are case insensitive. Add secrets to the scope. Secret names are case insensitive. If you have the Azure Databricks premium plan, assign access control to the secret scope.
