Data Lakehouse Experimentation, Presentation / Serving Layer The best example is to think about DataFrames APIs or SQL APIs in Apache Spark.ĭepending on the APIs and type of workload, you might want to assign different types of computing power (CPUs/GPUs).

These libraries and APIs are highly optimized for consuming your data assets in your Data Lake layer. The APIs layer creates a level of abstraction that allows developers and consumers to take advantage of different libraries and languages. In the Data Lakehouse metadata APIs layer, a set of APIs is used to access information. You still need to look at following data warehousing best practices, applying complex transformations and performing data cleansing. This layer does not replace critical thinking for data warehouse workloads. Durability: guarantees that once a transaction has been committed, it will remain committed even if the system fails.Isolation: multiple transactions can be executed at the same time without affecting each other.Consistency: ensures that when a transaction starts and when it finishes, it will be in a consistent state.Atomicity: guarantees that each transaction is treated as a single “unit”, which either succeeds or fails completely.ACID transactions to guarantee data consistency.Partitioning and bucketing – allow you to distribute data using different predicates (like filters) to optimize reads.The metadata layer includes features like: Examples are creating tables, slowly changing dimensions, using upserts, defining access and features that help match and improve the performance of RDBMS. This layer allows you to have data warehouse capabilities that are available in relational database management systems (RDBMS. The Data Lakehouse Metadata layer allows you to govern and optimise your data assets. This layer is decoupled from computing, allowing the compute power to scale independently from storage. With the rise of cloud technologies like Azure Data Lakes and AWS S3, storage has not only become fast but also cheap, accessible and limitless. The Storage or Data Lake layer in Data Lakehouses is used to store structured, semi-structured and unstructured datasets using open-source file formats (example: ORC or Parquet files). To begin, data lakehouses combine the best of data warehousing, Data Lakes and advanced data analytics to create a single platform. Users can build analytical solutions while minimising costs and overcoming existing challenges.ĭata lakehouse architecture is defined in multiple layers: Data Lake and Data Warehouse Challenges.Data Lakehouse Experimentation, Presentation / Serving Layer.Data Lakehouse Data Lake / Storage Layer.For further details on this, read more here. Data observability, just like its DevOps counterpart, uses automated monitoring, alerting, and triaging to identify and evaluate data quality issues. Inspired by the proven best practices of application observability in DevOps, data observability is an organization’s ability to fully understand the health of the data in their system. To further extend this part, Databricks has joined up with Monte Carlo to improve overall Data observability. Additional features such as a fully automated lineage of all your developed workloads (SQL, R, Python, Scala) are included with this, which you can read more about here. This means you can build a catalogue of your files, tables, dashboards, but also your ML Models.

Databricks has now implemented Unity Catalog, a centralized unified governance solution for all data & AI assets. Knowing which data you have and understanding its quality is key for a successful data platform implementation. When building a data platform, governance is of utmost importance, especially when your dataset is growing exponentially.

Lakehouse databricks paper