{"id":16423,"date":"2021-06-06T19:48:32","date_gmt":"2021-06-06T14:18:32","guid":{"rendered":"http:\/\/onlineappsdba.com\/?p=16423"},"modified":"2021-06-21T10:51:57","modified_gmt":"2021-06-21T05:21:57","slug":"delta-lake","status":"publish","type":"post","link":"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/","title":{"rendered":"Delta Lake architecture!"},"content":{"rendered":"<p>\ud83d\udccd Delta Lake is an open-source storage layer that brings ACID (atomicity, consistency, isolation, and durability) transactions to Apache Spark and big data workloads.<\/p>\n<p>\ud83d\udccd Using delta lake in a well-organized data lake can aid the integrity and performance of both batches and streaming data solutions.<\/p>\n<p>\ud83d\udccd The common misconception made with Delta lake is that people think it is a data platform service. It\u2019s not. It is simply a format that is defined when you create a DataFrame against the storage layer, and this is what brings the ACID capabilities to the data held in the DataFrames.<\/p>\n<p>\ud83d\udccd We organize our data into layers or folders as defined as bronze, silver, and gold as follows:<\/p>\n<p>\u27a0 Bronze &#8211; Tables contain raw data ingested from various sources (JSON files, RDBMS data, IoT data, etc.).<\/p>\n<p>\u27a0 Silver &#8211; Tables will provide a more refined view of our data.<\/p>\n<p>\u27a0 Gold &#8211; Tables provide business-level aggregates often used for reporting and dashboarding.<\/p>\n<p>More comprehensive knowledge on this topic in this blog: <a class=\"waffle-rich-text-link\" href=\"https:\/\/k21academy.com\/microsoft-azure\/data-engineer\/delta-lake\/?utm_source=onlineappsdba&amp;utm_medium=referral&amp;utm_campaign=dp20317_june21\" target=\"_blank\" rel=\"noopener noreferrer\">k21academy.com\/dp20317<\/a><\/p>\n<p>As a bonus to read this message, register for FREE LIVE CLASS at <a href=\"https:\/\/k21academy.com\/free-class-azure-data-engineer-certification\/?utm_source=onlineappsdba&amp;utm_medium=referral&amp;utm_campaign=dp20302_june21\" target=\"_blank\" rel=\"noopener noreferrer\">k21academy.com\/dp20302 \u200b<\/a>to see why everyone working on Data (Data Architects, Data Analysts, Data Admin, Database Administrator or even Data Scientists) should go for this certification.<\/p>\n<p><a href=\"https:\/\/k21academy.com\/free-class-azure-data-engineer-certification\/?utm_source=onlineappsdba&amp;utm_medium=referral&amp;utm_campaign=dp20302_june21\" target=\"_blank\" rel=\"noopener noreferrer\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-51077 size-full\" src=\"https:\/\/k21academy.com\/wp-content\/uploads\/2020\/10\/DP.200.201_CU.gif\" alt=\" [DP.200.201]_CU\" width=\"1338\" height=\"201\" \/><\/a><\/p>\n<p>FREE Telegram group &gt;&gt; <a href=\"https:\/\/t.me\/k21microsoftazure\">https:\/\/t.me\/k21microsoftazure<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>\ud83d\udccd Delta Lake is an open-source storage layer that brings ACID (atomicity, consistency, isolation, and durability) transactions to Apache Spark [&hellip;]<\/p>\n","protected":false},"author":115,"featured_media":16424,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[570,540],"tags":[],"class_list":["post-16423","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-engineer","category-microsoft-azure"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.8 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Delta Lake Architecture | Bronze, Silver, And Gold Architecture<\/title>\n<meta name=\"description\" content=\"In this blog post, you&#039;ll learn how to create Azure Databricks workspace and perform Structured Streaming with batch jobs by using Delta Lake.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Delta Lake Architecture | Bronze, Silver, And Gold Architecture\" \/>\n<meta property=\"og:description\" content=\"In this blog post, you&#039;ll learn how to create Azure Databricks workspace and perform Structured Streaming with batch jobs by using Delta Lake.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/\" \/>\n<meta property=\"article:published_time\" content=\"2021-06-06T14:18:32+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-06-21T05:21:57+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/onlineappsdba.com\/wp-content\/uploads\/2021\/06\/DataLake_BlogImage_ed2.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"1080\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Masroof Ahmad\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Masroof Ahmad\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/\",\"url\":\"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/\",\"name\":\"Delta Lake Architecture | Bronze, Silver, And Gold Architecture\",\"isPartOf\":{\"@id\":\"https:\/\/onlineappsdba.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/onlineappsdba.com\/wp-content\/uploads\/2021\/06\/DataLake_BlogImage_ed2.png\",\"datePublished\":\"2021-06-06T14:18:32+00:00\",\"dateModified\":\"2021-06-21T05:21:57+00:00\",\"author\":{\"@id\":\"https:\/\/onlineappsdba.com\/#\/schema\/person\/909a876ed58d400faf82caf81d61bfdb\"},\"description\":\"In this blog post, you'll learn how to create Azure Databricks workspace and perform Structured Streaming with batch jobs by using Delta Lake.\",\"breadcrumb\":{\"@id\":\"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/#primaryimage\",\"url\":\"https:\/\/onlineappsdba.com\/wp-content\/uploads\/2021\/06\/DataLake_BlogImage_ed2.png\",\"contentUrl\":\"https:\/\/onlineappsdba.com\/wp-content\/uploads\/2021\/06\/DataLake_BlogImage_ed2.png\",\"width\":1920,\"height\":1080,\"caption\":\"DataLake_BlogImage_ed2\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/onlineappsdba.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Delta Lake architecture!\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/onlineappsdba.com\/#website\",\"url\":\"https:\/\/onlineappsdba.com\/\",\"name\":\"\",\"description\":\"Oracle Implementation &amp; Training Experts\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/onlineappsdba.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/onlineappsdba.com\/#\/schema\/person\/909a876ed58d400faf82caf81d61bfdb\",\"name\":\"Masroof Ahmad\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/onlineappsdba.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/10f9db7bdbbd7f9ccfbe9b2d208e5978fc28315e9c704383e639a926ea0fce5f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/10f9db7bdbbd7f9ccfbe9b2d208e5978fc28315e9c704383e639a926ea0fce5f?s=96&d=mm&r=g\",\"caption\":\"Masroof Ahmad\"},\"url\":\"https:\/\/onlineappsdba.com\/index.php\/author\/masroof\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Delta Lake Architecture | Bronze, Silver, And Gold Architecture","description":"In this blog post, you'll learn how to create Azure Databricks workspace and perform Structured Streaming with batch jobs by using Delta Lake.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/","og_locale":"en_US","og_type":"article","og_title":"Delta Lake Architecture | Bronze, Silver, And Gold Architecture","og_description":"In this blog post, you'll learn how to create Azure Databricks workspace and perform Structured Streaming with batch jobs by using Delta Lake.","og_url":"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/","article_published_time":"2021-06-06T14:18:32+00:00","article_modified_time":"2021-06-21T05:21:57+00:00","og_image":[{"width":1920,"height":1080,"url":"https:\/\/onlineappsdba.com\/wp-content\/uploads\/2021\/06\/DataLake_BlogImage_ed2.png","type":"image\/png"}],"author":"Masroof Ahmad","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Masroof Ahmad","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/","url":"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/","name":"Delta Lake Architecture | Bronze, Silver, And Gold Architecture","isPartOf":{"@id":"https:\/\/onlineappsdba.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/#primaryimage"},"image":{"@id":"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/#primaryimage"},"thumbnailUrl":"https:\/\/onlineappsdba.com\/wp-content\/uploads\/2021\/06\/DataLake_BlogImage_ed2.png","datePublished":"2021-06-06T14:18:32+00:00","dateModified":"2021-06-21T05:21:57+00:00","author":{"@id":"https:\/\/onlineappsdba.com\/#\/schema\/person\/909a876ed58d400faf82caf81d61bfdb"},"description":"In this blog post, you'll learn how to create Azure Databricks workspace and perform Structured Streaming with batch jobs by using Delta Lake.","breadcrumb":{"@id":"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/#primaryimage","url":"https:\/\/onlineappsdba.com\/wp-content\/uploads\/2021\/06\/DataLake_BlogImage_ed2.png","contentUrl":"https:\/\/onlineappsdba.com\/wp-content\/uploads\/2021\/06\/DataLake_BlogImage_ed2.png","width":1920,"height":1080,"caption":"DataLake_BlogImage_ed2"},{"@type":"BreadcrumbList","@id":"https:\/\/onlineappsdba.com\/index.php\/2021\/06\/06\/delta-lake\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/onlineappsdba.com\/"},{"@type":"ListItem","position":2,"name":"Delta Lake architecture!"}]},{"@type":"WebSite","@id":"https:\/\/onlineappsdba.com\/#website","url":"https:\/\/onlineappsdba.com\/","name":"","description":"Oracle Implementation &amp; Training Experts","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/onlineappsdba.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/onlineappsdba.com\/#\/schema\/person\/909a876ed58d400faf82caf81d61bfdb","name":"Masroof Ahmad","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/onlineappsdba.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/10f9db7bdbbd7f9ccfbe9b2d208e5978fc28315e9c704383e639a926ea0fce5f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/10f9db7bdbbd7f9ccfbe9b2d208e5978fc28315e9c704383e639a926ea0fce5f?s=96&d=mm&r=g","caption":"Masroof Ahmad"},"url":"https:\/\/onlineappsdba.com\/index.php\/author\/masroof\/"}]}},"_links":{"self":[{"href":"https:\/\/onlineappsdba.com\/index.php\/wp-json\/wp\/v2\/posts\/16423","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/onlineappsdba.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/onlineappsdba.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/onlineappsdba.com\/index.php\/wp-json\/wp\/v2\/users\/115"}],"replies":[{"embeddable":true,"href":"https:\/\/onlineappsdba.com\/index.php\/wp-json\/wp\/v2\/comments?post=16423"}],"version-history":[{"count":0,"href":"https:\/\/onlineappsdba.com\/index.php\/wp-json\/wp\/v2\/posts\/16423\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/onlineappsdba.com\/index.php\/wp-json\/wp\/v2\/media\/16424"}],"wp:attachment":[{"href":"https:\/\/onlineappsdba.com\/index.php\/wp-json\/wp\/v2\/media?parent=16423"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/onlineappsdba.com\/index.php\/wp-json\/wp\/v2\/categories?post=16423"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/onlineappsdba.com\/index.php\/wp-json\/wp\/v2\/tags?post=16423"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}