Najlepsze podcasty z kategorii: Data Engineering (2024)

1
Stitching Together Enterprise Analytics With Microsoft Fabric 53:22

3d ago53:22

53:22

Summary Data lakehouse architectures have been gaining significant adoption. To accelerate adoption in the enterprise Microsoft has created the Fabric platform, based on their OneLake architecture. In this episode Dipti Borkar shares her experiences working on the product team at Fabric and explains the various use cases for the Fabric service. Ann…

1
Vector Databases Won’t Replace SQL - Andy Pavlo 42:59

22d ago42:59

42:59

SQL’s slow. SQL’s stupid. We hear these claims every time a new shiny tool enters the market, only to realize five years later when the hype dies down that SQL is actually a good idea. In this super techie episode of the Data Engineering Show, Andy Pavlo, Associate Professor at Carnegie Mellon University, joins the bros to delve into database inter…

1
The Future of AI in Data Engineering With Astronomer’s Julian LaNeve and David Xue 23:36

27d ago23:36

23:36

The world of data orchestration and machine learning is rapidly evolving, and tools like Apache Airflow are at the forefront of these changes. Understanding how to effectively utilize these tools can significantly enhance data processing and AI model deployment.This episode features Julian LaNeve, CTO at Astronomer, and David Xue, Machine Learning …

1
Being Data Driven At Stripe With Trino And Iceberg 53:19

10d ago53:19

53:19

Summary Stripe is a company that relies on data to power their products and business. To support that functionality they have invested in Trino and Iceberg for their analytical workloads. In this episode Kevin Liu shares some of the interesting features that they have built by combining those technologies, as well as the challenges that they face i…

1
X-Ray Vision For Your Flink Stream Processing With Datorios 42:22

16d ago42:22

42:22

Summary Streaming data processing enables new categories of data products and analytics. Unfortunately, reasoning about stream processing engines is complex and lacks sufficient tooling. To address this shortcoming Datorios created an observability platform for Flink that brings visibility to the internals of this popular stream processing system. …

1
Practical First Steps In Data Governance For Long Term Success 1:00:40

23d ago1:00:40

1:00:40

Summary Modern businesses aspire to be data driven, and technologists enjoy working through the challenge of building data systems to support that goal. Data governance is the binding force between these two parts of the organization. Nicola Askham found her way into data governance by accident, and stayed because of the benefit that she was able t…

1
The Power of Airflow in Modern Data Environments at Wynn Las Vegas with Siva Krishna Yetukuri 24:31

28d ago24:31

24:31

Understanding the critical role of data integration and management is essential for driving business success, particularly in a dynamic environment like a luxury casino resort.In this episode, we sit down with Siva Krishna Yetukuri, Cloud Data Architect at Wynn Las Vegas, to explore how Airflow and other tools are transforming data workflows and cu…

1
Powering the Texas Rangers World Series Win With AI on Airflow with Alexander Booth 23:38

28d ago23:38

23:38

The integration of data and AI in sports is transforming how teams strategize and perform. Understanding how to harness this technology is key to staying competitive in the rapidly evolving landscape of baseball.In this episode, we sit down with Alexander Booth, Assistant Director of Research and Development at Texas Rangers Baseball Club, to explo…

1
Data Migration Strategies For Large Scale Systems 1:00:00

1M ago1:00:00

1:00:00

Summary Any software system that survives long enough will require some form of migration or evolution. When that system is responsible for the data layer the process becomes more challenging. Sriram Panyam has been involved in several projects that required migration of large volumes of data in high traffic environments. In this episode he shares …

1
Zenlytic Is Building You A Better Coworker With AI Agents 54:19

1M ago54:19

54:19

Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. Unfortunately this often turns into an exercise in frustration for everyone involved due to complex workflows and hard-to-understand dashboards. The team at Zenlytic have leaned on the promise of la…

1
Release Management For Data Platform Services And Logic 20:08

1M ago20:08

20:08

Summary Building a data platform is a substrantial engineering endeavor. Once it is running, the next challenge is figuring out how to address release management for all of the different component parts. The services and systems need to be kept up to date, but so does the code that controls their behavior. In this episode your host Tobias Macey ref…

1
Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach 54:16

2M ago54:16

54:16

Summary Artificial intelligence has dominated the headlines for several months due to the successes of large language models. This has prompted numerous debates about the possibility of, and timeline for, artificial general intelligence (AGI). Peter Voss has dedicated decades of his life to the pursuit of truly intelligent software through the appr…

1
Build Your Second Brain One Piece At A Time 50:10

2M ago50:10

50:10

Summary Generative AI promises to accelerate the productivity of human collaborators. Currently the primary way of working with these tools is through a conversational prompt, which is often cumbersome and unwieldy. In order to simplify the integration of AI capabilities into developer workflows Tsavo Knott helped create Pieces, a powerful collecti…

1
Making Email Better With AI At Shortwave 53:43

2M ago53:43

53:43

Summary Generative AI has rapidly transformed everything in the technology sector. When Andrew Lee started work on Shortwave he was focused on making email more productive. When AI started gaining adoption he realized that he had even more potential for a transformative experience. In this episode he shares the technical challenges that he and his …

1
How ZoomInfo transitioned from data graveyards to ROI-driven data projects 39:46

2M ago39:46

39:46

Too often expensive resources and manhours are spent on dashboards no one uses, resulting in zero ROI. Philip Philip Zelitchenko, VP of Data & Analytics at ZoomInfo met the bros to talk about adopting product management principles to ensure data projects have value, and provide an unfiltered peak into ZoomInfo’s data stack and unique tech culture.…

1
Designing A Non-Relational Database Engine 1:16:01

2M ago1:16:01

1:16:01

Summary Databases come in a variety of formats for different use cases. The default association with the term "database" is relational engines, but non-relational engines are also used quite widely. In this episode Oren Eini, CEO and creator of RavenDB, explores the nuances of relational vs. non-relational engines, and the strategies for designing …

1
Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer 56:23

3M ago56:23

56:23

Summary Maintaining a single source of truth for your data is the biggest challenge in data engineering. Different roles and tasks in the business need their own ways to access and analyze the data in the organization. In order to enable this use case, while maintaining a single point of access, the semantic layer has evolved as a technological sol…

1
Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary 50:44

3M ago50:44

50:44

Summary Working with data is a complicated process, with numerous chances for something to go wrong. Identifying and accounting for those errors is a critical piece of building trust in the organization that your data is accurate and up to date. While there are numerous products available to provide that visibility, they all have different technolo…

1
Matthew Weingarten from Disney Streaming about Data Quality Best Practices 27:21

3M ago27:21

27:21

Matthew Weingarten, Lead Data Engineer at Disney Streaming, talks about principles essential for data quality, cost optimization, debugging, and data modeling, as adopted by the world's leading companies.Autor: The Firebolt Data Bros

1
Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+ 55:39

3M ago55:39

55:39

Summary A core differentiator of Dagster in the ecosystem of data orchestration is their focus on software defined assets as a means of building declarative workflows. With their launch of Dagster+ as the redesigned commercial companion to the open source project they are investing in that capability with a suite of new features. In this episode Pe…

1
Reconciling The Data In Your Databases With Datafold 58:14

3M ago58:14

58:14

Summary A significant portion of data workflows involve storing and processing information in database engines. Validating that the information is stored and processed correctly can be complex and time-consuming, especially when the source and destination speak different dialects of SQL. In this episode Gleb Mezhanskiy, founder and CEO of Datafold,…

1
Version Your Data Lakehouse Like Your Software With Nessie 40:55

4M ago40:55

40:55

Summary Data lakehouse architectures are gaining popularity due to the flexibility and cost effectiveness that they offer. The link that bridges the gap between data lake and warehouse capabilities is the catalog. The primary purpose of the catalog is to inform the query engine of what data exists and where, but the Nessie project aims to go beyond…

1
Iceberg at Netflix and Beyond with Ryan Blue 47:37

4M ago47:37

47:37

Apache Iceberg is an open source high-performance format for huge data tables. Iceberg enables the use of SQL tables for big data, while making it possible for engines like Spark and Hive to safely work with the same tables, at the same time. Iceberg was started at Netflix by Ryan Blue and Dan Weeks, and was open-sourced and donated to the Apache S…

1
When And How To Conduct An AI Program 46:25

4M ago46:25

46:25

Summary Artificial intelligence technologies promise to revolutionize business and produce new sources of value. In order to make those promises a reality there is a substantial amount of strategy and investment required. Colleen Tartow has worked across all stages of the data lifecycle, and in this episode she shares her hard-earned wisdom about h…

1
Joseph Machado, Senior Data Engineer @ LinkedIn talks best practices 25:59

4M ago25:59

25:59

Data engineering should be less about the stack and more about best practices. While tools may change, foundational principles will remain constant. Joseph Mercado, Senior Data Engineer at LinkedIn, is on The Data Engineering Show to talk about principles that are key to success, leveraging AI for automation, and adopting software engineering metho…

1
Find Out About The Technology Behind The Latest PFAD In Analytical Database Development 56:00

4M ago56:00

56:00

Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. When Paul Dix decided to re-write the InfluxDB engine he found the Apache Arrow …

1
Using Trino And Iceberg As The Foundation Of Your Data Lakehouse 58:46

4M ago58:46

58:46

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Multiple open source projects and vendors have been working together to make this vision a reality. In this episode Dain Sundstrom, CTO of Starburst, explains how the combinatio…

1
Data Sharing Across Business And Platform Boundaries 59:55

5M ago59:55

59:55

Summary Sharing data is a simple concept, but complicated to implement well. There are numerous business rules and regulatory concerns that need to be applied. There are also numerous technical considerations to be made, particularly if the producer and consumer of the data aren't using the same platforms. In this episode Andrew Jefferson explains …

1
Building a Data Lake with Adam Ferrari 46:19

5M ago46:19

46:19

Starburst is a data lake analytics platform. It’s designed to help users work with structured data at scale, and is built on the open source platform, Trino. Adam Ferrari is the SVP of Engineering at Starburst. He joins the show to talk about Starburst, data engineering, and what it takes to build a data lake. Full Disclosure: Starburst is a sponso…

1
Tackling Real Time Streaming Data With SQL Using RisingWave 56:55

5M ago56:55

56:55

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. In this episode Yingjun Wu explains how it is architected to power analytical workflows on continu…

1
Build A Data Lake For Your Security Logs With Scanner 1:02:38

5M ago1:02:38

1:02:38

Summary Monitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. The majority of products that are available either require too much effort to structure the logs, or aren't fast enough for interactive use cases. Cliff Crosland co-founded Scanner to provide fast querying…

1
Professors Joe Hellerstein and Joseph Gonzalez on LLMs 46:07

5M ago46:07

46:07

Joe Hellerstein is the Jim Gray Professor of Computer Science at Berkeley and Joseph Gonzalez is an Associate Professor in the Electrical Engineering and Computer Science department. They’ve inspired generations of database enthusiasts (including Benji and Eldad) and have come on the show to talk about all things LLM and RunLLM which they co-founde…

1
Modern Customer Data Platform Principles 1:01:33

5M ago1:01:33

1:01:33

Summary Databases and analytics architectures have gone through several generational shifts. A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. In this episode Tasso Argyros, CEO of ActionIQ, gives a summary of the major epochs in database technologies and how…

1
Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel 50:26

6M ago50:26

50:26

Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. Unfortunately, the volumes of data that are being generated continue to double, requiring further advancements in the platform capabilities to keep up. As the sophistication increases, so does the complexity, leading to challenges for user ex…

1
Megan Lieu on powerful notebooks that enable collaboration 31:31

6M ago31:31

31:31

There are two types of data influencers on LinkedIn: 1. Those who talk directly about the products and companies they work for 2. Those that provide more general guidance, tips and opinions Can influencers actually be passionate about the products they’re developing and straightforwardly talk about them without sounding salesly? We’re kicking off 2…

1
Designing Data Platforms For Fintech Companies 47:56

6M ago47:56

47:56

Summary Working with financial data requires a high degree of rigor due to the numerous regulations and the risks involved in security breaches. In this episode Andrey Korchack, CTO of fintech startup Monite, discusses the complexities of designing and implementing a data platform in that sector. Announcements Hello and welcome to the Data Engineer…

1
Rama with Nathan Marz 45:06

6M ago45:06

45:06

Building scalable software applications can be complex and typically requires dozens of different tools. The engineering often involves handling many arcane tasks that are distant from actual application logic. In addition, a lack of a cohesive model for building applications can lead to substantial engineering costs. Nathan Marz is the creator of …

1
Bonus Episode: SurrealDB with Tobie Morgan Hitchcock 57:17

6M ago57:17

57:17

SurrealDB is the result of a long-time collaboration between brothers Tobie and Jaime Morgan Hitchcock. The project has modest origins and started merely to support other projects the brothers were working on. However, over time the project grew and in 2021 they started working on it full-time. Since then the project has gained serious adoption. Wh…

1
Troubleshooting Kafka In Production 1:14:43

6M ago1:14:43

1:14:43

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Operating it at scale, however, is notoriously challenging. Elad Eldor has experienced these challenges first-hand, leading to his work writing the book "Kafka: : Troubleshooting in Production". In this episode he hi…

1
Adding An Easy Mode For The Modern Data Stack With 5X 56:12

6M ago56:12

56:12

Summary The "modern data stack" promised a scalable, composable data platform that gave everyone the flexibility to use the best tools for every job. The reality was that it left data teams in the position of spending all of their engineering effort on integrating systems that weren't designed with compatible user experiences. The team at 5X unders…

1
Run Your Own Anomaly Detection For Your Critical Business Metrics With Anomstack 51:17

7M ago51:17

51:17

Summary If your business metrics looked weird tomorrow, would you know about it first? Anomaly detection is focused on identifying those outliers for you, so that you are the first to know when a business critical dashboard isn't right. Unfortunately, it can often be complex or expensive to incorporate anomaly detection into your data platform. And…

1
2023 re:Invent Data Recap with Alex DeBrie 39:29

7M ago39:29

39:29

Alex Debrie and Bob Haffner recap their favorite announcements from 2023 re:Invent #data #dataengineering #aws Connect with AlexTwitter: @alexbdebrieBlog: alexdebrie.comBook: dynamodbbook.comPodcast: youtube.com/@SoftwareHuddle Connect with BobTwitter - @bobhaffnerLinkedIn - linkedin.com/in/bobhaffner Alex’s talkhttps://www.youtube.com/watch?v=PVUo…

1
Tracking Drug Smugglers and Migrating Databases with Benny Keinan and Lior Resisi 50:40

7M ago50:40

50:40

Maritime logistics is the process organizing the movement of goods across the ocean. Historically, this has been a challenging problem because of the multinational nature of shipping, as well as piracy, smuggling, and legacy technology. It’s also profoundly important for security reasons, and because 90% of what we buy travels over the oceans. Ocea…

1
Designing Data Transfer Systems That Scale 1:03:57

7M ago1:03:57

1:03:57

Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. Andrei Tserakhau has dedicated his careeer to this problem, and in …

1
The Right to Be Forgotten with Gal Ringel 47:45

7M ago47:45

47:45

Data breaches at major companies are so now common that they hardly make the news. The Wikipedia page on data breaches lists over 350 between 2004 and 2023. The Equifax breach in 2017 was especially notable because over 160 million records were leaked, and much of the data was acquired by Equifax without individuals’ knowledge or consent. Data brea…

1
Sofascore with Josip Stuhli 49:48

7M ago49:48

49:48

If you’re a sports fan and like to track sports statistics and results, you’ve probably heard of Sofascore. The website started in 2010 and ran on a modest single server. It now has 25 million monthly active users, covers 20 different sports, 11,000 leagues and tournaments, and is available in over 30 languages. Josip Stuhli has been with Sofascore…

1
Addressing The Challenges Of Component Integration In Data Platform Architectures 29:42

7M ago29:42

29:42

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. One of the core complexities that needs to be addressed is the fractal set of integrations that need to be managed across the individual components. In this episode Tobias Macey shares his thoughts on the challenges that he is faci…

Podcasty warte posłuchania

Data Engineering Podcasty

Podcasty warte posłuchania

Skrócona instrukcja obsługi