A high volume of data can contain a high volume of useful information. That fact is well understood by the software world. Unfortunately, it is not a simple process to surface useful information from this high volume of data. A human analyst needs to understand the business, formulate a question, and determine what metrics could The post <a rel="nofollow" href="https://softwareengineeringdaily.com/2020/03/17/sisu-data-with-peter-bailis/">Sisu Data with Peter Bailis</a> appeared first on <a rel="nofollow" href="https://softwareengineeringdaily.com">Software Engineering Daily</a>.

A high volume of data can contain a high volume of useful information. That fact is well understood by the software world. Unfortunately, it is not a simple process to surface useful information from this high volume of data. A human analyst needs to understand the business, formulate a question, and determine what metrics could The post Sisu Data with Peter Bailis appeared first on Software Engineering Daily.

Data – Software Engineering Daily

Sisu Data with Peter Bailis

Physical places have a large amount of latent data. Pick any location on a map, and think about all of the questions you could ask about that location. What businesses are at that location? How many cars pass through it? What is the soil composition? How much is the land on that location worth? The The post <a rel="nofollow" href="https://softwareengineeringdaily.com/2020/03/16/location-data-with-ryan-fox-squire/">Location Data with Ryan Fox Squire</a> appeared first on <a rel="nofollow" href="https://softwareengineeringdaily.com">Software Engineering Daily</a>.

Physical places have a large amount of latent data. Pick any location on a map, and think about all of the questions you could ask about that location. What businesses are at that location? How many cars pass through it? What is the soil composition? How much is the land on that location worth? The The post Location Data with Ryan Fox Squire appeared first on Software Engineering Daily.

Location Data with Ryan Fox Squire

Lyft is a ridesharing company that generates a high volume of data every day.  This data includes ride history, pricing information, mapping, routing, and financial transactions. The data is stored across a variety of different databases, data lakes, and queueing systems, and is processed at scale in order to generate machine learning models, reports, and The post <a rel="nofollow" href="https://softwareengineeringdaily.com/2020/03/12/flyte-lyft-data-processing-platform-with-allyson-gale-and-ketan-umare/">Flyte: Lyft Data Processing Platform with Allyson Gale and Ketan Umare</a> appeared first on <a rel="nofollow" href="https://softwareengineeringdaily.com">Software Engineering Daily</a>.

Lyft is a ridesharing company that generates a high volume of data every day.  This data includes ride history, pricing information, mapping, routing, and financial transactions. The data is stored across a variety of different databases, data lakes, and queueing systems, and is processed at scale in order to generate machine learning models, reports, and The post Flyte: Lyft Data Processing Platform with Allyson Gale and Ketan Umare appeared first on Software Engineering Daily.

Flyte: Lyft Data Processing Platform with Allyson Gale and Ketan Umare

GraphQL is a system that allows frontend engineers to make requests across multiple data sources using a simple query format. In GraphQL, a frontend developer does not have to worry about the request logic for individual backend services. The frontend developer only needs to know how to issue GraphQL requests from the client, and these The post <a rel="nofollow" href="https://softwareengineeringdaily.com/2020/03/10/onegraph-graphql-tooling-with-sean-grove/">OneGraph: GraphQL Tooling with Sean Grove</a> appeared first on <a rel="nofollow" href="https://softwareengineeringdaily.com">Software Engineering Daily</a>.

GraphQL is a system that allows frontend engineers to make requests across multiple data sources using a simple query format. In GraphQL, a frontend developer does not have to worry about the request logic for individual backend services. The frontend developer only needs to know how to issue GraphQL requests from the client, and these The post OneGraph: GraphQL Tooling with Sean Grove appeared first on Software Engineering Daily.

OneGraph: GraphQL Tooling with Sean Grove

A data warehouse serves the purpose of providing low latency queries for high volumes of data. A data warehouse is often part of a data pipeline, which moves data through different areas of infrastructure in order to build applications such as machine learning models, dashboards, and reports. Modern data pipelines are often associated with the The post <a rel="nofollow" href="https://softwareengineeringdaily.com/2020/03/09/dbt-data-build-tool-with-tristan-handy/">DBT: Data Build Tool with Tristan Handy</a> appeared first on <a rel="nofollow" href="https://softwareengineeringdaily.com">Software Engineering Daily</a>.

A data warehouse serves the purpose of providing low latency queries for high volumes of data. A data warehouse is often part of a data pipeline, which moves data through different areas of infrastructure in order to build applications such as machine learning models, dashboards, and reports. Modern data pipelines are often associated with the The post DBT: Data Build Tool with Tristan Handy appeared first on Software Engineering Daily.

DBT: Data Build Tool with Tristan Handy

At Airbnb, infrastructure management is standardized across the organization. Platform engineering teams build tools that allow the other teams throughout the organization to work more effectively. A platform engineering team handles problems such as continuous integration, observability, and service discovery. Other teams throughout a company use the tools that a platform engineering team builds. For The post <a rel="nofollow" href="https://softwareengineeringdaily.com/2020/02/24/infrastructure-management-with-joey-parsons/">Infrastructure Management with Joey Parsons</a> appeared first on <a rel="nofollow" href="https://softwareengineeringdaily.com">Software Engineering Daily</a>.

At Airbnb, infrastructure management is standardized across the organization. Platform engineering teams build tools that allow the other teams throughout the organization to work more effectively. A platform engineering team handles problems such as continuous integration, observability, and service discovery. Other teams throughout a company use the tools that a platform engineering team builds. For The post Infrastructure Management with Joey Parsons appeared first on Software Engineering Daily.

Infrastructure Management with Joey Parsons

In a modern data platform, distributed streaming systems are used to read data coming off of an application in real-time. There are a wide variety of streaming systems, including Kafka Streams, Apache Samza, Apache Flink, Spark Streaming, and more.  When Eric Anderson joined the show back in 2016, he was working at Google on Google The post <a rel="nofollow" href="https://softwareengineeringdaily.com/2020/02/20/data-infrastructure-investing-with-eric-anderson/">Data Infrastructure Investing with Eric Anderson</a> appeared first on <a rel="nofollow" href="https://softwareengineeringdaily.com">Software Engineering Daily</a>.

In a modern data platform, distributed streaming systems are used to read data coming off of an application in real-time. There are a wide variety of streaming systems, including Kafka Streams, Apache Samza, Apache Flink, Spark Streaming, and more.  When Eric Anderson joined the show back in 2016, he was working at Google on Google The post Data Infrastructure Investing with Eric Anderson appeared first on Software Engineering Daily.

Data Infrastructure Investing with Eric Anderson

Distributed stream processing frameworks are used to rapidly ingest and aggregate large volumes of incoming data. These frameworks often require the application developer to write imperative logic describing how that data should be processed.  For example, a high volume of clickstream data that is getting buffered to Kafka needs to have a stream processing system The post <a rel="nofollow" href="https://softwareengineeringdaily.com/2020/02/19/materialize-streaming-sql-on-timely-data-with-arjun-narayan-and-frank-mcsherry/">Materialize: Streaming SQL on Timely Data with Arjun Narayan and Frank McSherry</a> appeared first on <a rel="nofollow" href="https://softwareengineeringdaily.com">Software Engineering Daily</a>.

Distributed stream processing frameworks are used to rapidly ingest and aggregate large volumes of incoming data. These frameworks often require the application developer to write imperative logic describing how that data should be processed.  For example, a high volume of clickstream data that is getting buffered to Kafka needs to have a stream processing system The post Materialize: Streaming SQL on Timely Data with Arjun Narayan and Frank McSherry appeared first on Software Engineering Daily.

Materialize: Streaming SQL on Timely Data with Arjun Narayan and Frank McSherry

A data pipeline is a series of steps that takes large data sets and creates usable results from them. At the beginning of a data pipeline, a data set might be pulled from a database, a distributed file system, or a Kafka topic. Throughout a data pipeline, different data sets are joined, filtered, and statistically The post <a rel="nofollow" href="https://softwareengineeringdaily.com/2020/02/17/great-expectations-data-pipeline-testing-with-abe-gong/">Great Expectations: Data Pipeline Testing with Abe Gong</a> appeared first on <a rel="nofollow" href="https://softwareengineeringdaily.com">Software Engineering Daily</a>.

A data pipeline is a series of steps that takes large data sets and creates usable results from them. At the beginning of a data pipeline, a data set might be pulled from a database, a distributed file system, or a Kafka topic. Throughout a data pipeline, different data sets are joined, filtered, and statistically The post Great Expectations: Data Pipeline Testing with Abe Gong appeared first on Software Engineering Daily.

Great Expectations: Data Pipeline Testing with Abe Gong

A data warehouse provides low latency access to large volumes of data.  A data warehouse is a crucial piece of infrastructure for a large company, because it can be used to answer complex questions involving a large number of data points. But a data warehouse usually cannot hold all of a company’s data at any The post <a rel="nofollow" href="https://softwareengineeringdaily.com/2020/02/14/data-warehouse-etl-with-matthew-scullion/">Data Warehouse ETL with Matthew Scullion</a> appeared first on <a rel="nofollow" href="https://softwareengineeringdaily.com">Software Engineering Daily</a>.

A data warehouse provides low latency access to large volumes of data.  A data warehouse is a crucial piece of infrastructure for a large company, because it can be used to answer complex questions involving a large number of data points. But a data warehouse usually cannot hold all of a company’s data at any The post Data Warehouse ETL with Matthew Scullion appeared first on Software Engineering Daily.

Data Warehouse ETL with Matthew Scullion

Distributed stream processing systems are used to read large volumes of data and perform operations across those data streams.  These stream processing systems often build off of the MapReduce algorithm for collecting and aggregating large volumes of data, but instead of processing a calculation over a single large batch of data, they process data on The post <a rel="nofollow" href="https://softwareengineeringdaily.com/2020/02/12/flink-and-beam-stream-processing-with-maximilian-michels/">Flink and BEAM Stream Processing with Maximilian Michels</a> appeared first on <a rel="nofollow" href="https://softwareengineeringdaily.com">Software Engineering Daily</a>.

Distributed stream processing systems are used to read large volumes of data and perform operations across those data streams.  These stream processing systems often build off of the MapReduce algorithm for collecting and aggregating large volumes of data, but instead of processing a calculation over a single large batch of data, they process data on The post Flink and BEAM Stream Processing with Maximilian Michels appeared first on Software Engineering Daily.

Flink and BEAM Stream Processing with Maximilian Michels

Large companies generate large volumes of data. This data gets dumped into a data lake for long-term storage, then pulled into memory for processing and analysis. Once it is in memory, it is often read into a dashboard, which presents a human with a visualization of the data.  The end-user who is consuming this data The post <a rel="nofollow" href="https://softwareengineeringdaily.com/2020/02/11/druid-analytics-with-jad-nauous/">Druid Analytics with Jad Naous</a> appeared first on <a rel="nofollow" href="https://softwareengineeringdaily.com">Software Engineering Daily</a>.

Large companies generate large volumes of data. This data gets dumped into a data lake for long-term storage, then pulled into memory for processing and analysis. Once it is in memory, it is often read into a dashboard, which presents a human with a visualization of the data.  The end-user who is consuming this data The post Druid Analytics with Jad Naous appeared first on Software Engineering Daily.

Druid Analytics with Jad Naous

Data infrastructure has been transformed over the last fifteen years.  The open source Hadoop project led to the creation of multiple companies based around commercializing the MapReduce algorithm and Hadoop distributed file system. Cheap cloud storage popularized the usage of data lakes. Cheap cloud servers led to wide experimentation for data tools. Apache Spark emerged The post <a rel="nofollow" href="https://softwareengineeringdaily.com/2020/02/10/the-data-exchange-with-ben-lorica/">The Data Exchange with Ben Lorica</a> appeared first on <a rel="nofollow" href="https://softwareengineeringdaily.com">Software Engineering Daily</a>.

Data infrastructure has been transformed over the last fifteen years.  The open source Hadoop project led to the creation of multiple companies based around commercializing the MapReduce algorithm and Hadoop distributed file system. Cheap cloud storage popularized the usage of data lakes. Cheap cloud servers led to wide experimentation for data tools. Apache Spark emerged The post The Data Exchange with Ben Lorica appeared first on Software Engineering Daily.

The Data Exchange with Ben Lorica

A data platform contains all of the data that a company has accumulated over the years. Across a data platform, there is a multitude of data sources: databases, a data lake, data warehouses, a distributed queue like Kafka, and external data sources like Salesforce and Zendesk. A user of the data platform often has a The post <a rel="nofollow" href="https://softwareengineeringdaily.com/2020/02/07/presto-with-justin-borgman/">Presto with Justin Borgman</a> appeared first on <a rel="nofollow" href="https://softwareengineeringdaily.com">Software Engineering Daily</a>.

A data platform contains all of the data that a company has accumulated over the years. Across a data platform, there is a multitude of data sources: databases, a data lake, data warehouses, a distributed queue like Kafka, and external data sources like Salesforce and Zendesk. A user of the data platform often has a The post Presto with Justin Borgman appeared first on Software Engineering Daily.

Presto with Justin Borgman

Nubank is a popular bank that is based in Brazil. Nubank has more than 20 million customers, and has accumulated a high volume of data over the six years since it was started. Mobile computing and cloud computing have given rise to “challenger banks” that operate more like software companies. When a software company reaches The post <a rel="nofollow" href="https://softwareengineeringdaily.com/2020/02/06/nubank-data-engineering-with-sujith-nair/">Nubank Data Engineering with Sujith Nair</a> appeared first on <a rel="nofollow" href="https://softwareengineeringdaily.com">Software Engineering Daily</a>.

Nubank is a popular bank that is based in Brazil. Nubank has more than 20 million customers, and has accumulated a high volume of data over the six years since it was started. Mobile computing and cloud computing have given rise to “challenger banks” that operate more like software companies. When a software company reaches The post Nubank Data Engineering with Sujith Nair appeared first on Software Engineering Daily.

Nubank Data Engineering with Sujith Nair

Databases and data engineering episodes of Software Engineering Daily

hubhopper

Data – Software Engineering Daily

hubhopper