Wednesday, May 22, 2024
From the WireTechnology

Modern Data Engineering. Platform Specific Tools and Advanced… | by 💡Mike Shakhomirov | Nov, 2023

In the article “Modern Data Engineering: Platform Specific Tools and Advanced Techniques” by Mike Shakhomirov, you will explore the evolving landscape of modern data engineering and its impact on data engineers. The article will discuss crucial factors that affect data engineers and how to leverage this knowledge to power advanced analytics pipelines and achieve operational excellence. Additionally, the article dives into popular data engineering questions, such as the definition of modern data engineering, the efficiency of data pipelines, and the technological requirements for operational excellence. By delving into these topics, you will gain valuable insights into the trends and techniques shaping the field of data engineering.

Platform Specific Tools and Advanced Techniques

Welcome to an article discussing modern data engineering and the tools and techniques that can help fuel advanced data pipelines and enable operational excellence. As a friendly guide, we will explore popular data engineering questions, discuss modern trends, and delve into the differences between ELT and ETL approaches. So let’s get started!

Modern data engineering (DE). What is it?

Data engineering, also known as DE, is a field focused on designing and managing data pipelines that facilitate the flow of data from various sources to data consumers, such as data analysts and business intelligence teams. It involves transforming raw data into a format suitable for analysis, processing large volumes of data efficiently, and ensuring data quality and integrity.

Does your DE work well enough to fuel advanced data pipelines and Business intelligence (BI)?

The success of advanced data pipelines and business intelligence depends heavily on the effectiveness of the underlying data engineering processes. Without a robust and efficient data engineering framework, these pipelines can become bottlenecked and hinder the generation of valuable insights. It is crucial to evaluate the performance of your data engineering processes and make adjustments as needed to support the needs of your organization.

Are your data pipelines efficient?

Efficiency is a key factor in ensuring the smooth flow of data through your pipelines. Inefficient data pipelines can result in delays, increased costs, and decreased productivity. It is important to assess the efficiency of your pipelines regularly and identify areas for improvement. This may involve optimizing data transformations, improving data processing speed, and implementing automated monitoring and alerting systems.

What is required from the technological point of view to enable operational excellence?

To achieve operational excellence in data engineering, several technological aspects need to be considered. These include:

  • Data Warehousing: Choosing the right data warehouse solution, such as Snowflake, BigQuery, Redshift, or Firebolt, that aligns with your organizational needs and offers scalability, performance, and ease of management.

  • ELT Approach: Embracing the ELT (Extract, Load, Transform) approach, where data transformations are performed directly in the data warehouse. This approach leverages the power of modern data warehouse technologies and simplifies the data engineering process.

  • Data Lakes: Integrating data lakes into your data infrastructure, especially when dealing with unstructured data. Data lakes provide a scalable and flexible storage solution for various data types and enable advanced analytics and machine learning.

  • Data Connectors and API Integrations: Utilizing managed solutions like Fivetran and Stitch, which simplify the process of integrating third-party APIs with your data ecosystem. Leveraging existing data connectors can save time and effort in building connectors from scratch.

  • ETL Frameworks: Exploring the growing number of ETL (Extract, Transform, Load) frameworks available, such as Dataform and DBT. These frameworks provide powerful tools for managing and automating data transformations, ensuring data quality, and simplifying the data engineering workflow.

  • Data Infrastructure as Code: Adopting the practice of treating data infrastructure as code, using tools like Terraform or AWS CloudFormation. This allows for version control, reproducibility, and scalability of data infrastructure components.

  • Data Mesh and Decentralized Data Management: Considering the principles of Data Mesh, an emerging paradigm that emphasizes decentralized data ownership and management. This approach can help democratize data access, improve data quality, and foster collaboration across different teams and departments.

  • Democratization of Business Intelligence Pipelines using AI: Exploring the use of AI-powered tools and techniques to democratize access to business intelligence pipelines. These tools can automate data analysis, generate insights, and empower non-technical users to gain valuable insights from data.

  • Focus on Data Literacy: Prioritizing data literacy within your organization by promoting a data-driven culture, providing training and resources to improve data literacy skills, and ensuring that data is accessible and understandable to all stakeholders.

Modern data engineering trends

As technology evolves, so does the field of data engineering. Here are some key trends that are shaping modern data engineering practices:


Traditionally, data engineering followed the ETL (Extract, Transform, Load) approach, where data from various sources was extracted, transformed outside the data warehouse, and then loaded into the warehouse. However, there has been a shift towards the ELT (Extract, Load, Transform) approach, where data is first loaded into the data warehouse, and then transformed within the warehouse. This shift is largely driven by the increasing power and scalability of modern data warehouse platforms, such as Snowflake, which allow for efficient and performant data transformations within the warehouse itself.

Simplified data connectors and API integrations

Managed solutions, like Fivetran and Stitch, have emerged to simplify the process of integrating third-party APIs with data ecosystems. These solutions provide pre-built connectors and handle the management and maintenance of these connectors, reducing the burden on data engineering teams.

ETL frameworks explosion

ETL frameworks, such as Dataform and DBT, have gained popularity due to their ability to streamline and automate data transformations. These frameworks provide a layer of abstraction that simplifies the development, testing, and deployment of data transformations, increasing productivity and ensuring data quality.

Data infrastructure as code

Treating data infrastructure as code has become a best practice in modern data engineering. Tools like Terraform and AWS CloudFormation allow for the provisioning and management of data infrastructure components through code. This approach enables version control, scalability, and reproducibility of data infrastructure.

Data Mesh and decentralized data management

Data Mesh is an emerging paradigm in data engineering that promotes decentralized data ownership and management. Instead of relying on a central data team, data ownership and responsibility are distributed across the organization. This approach fosters collaboration, improves data quality, and enables domain experts to have greater control over their own data.

Democratization of Business intelligence pipelines using AI

The use of AI-powered tools and techniques is making it easier for non-technical users to access and derive insights from data. These tools automate data analysis, generate reports and visualizations, and provide recommendations, empowering business users to make data-driven decisions.

Focus on data literacy

Data literacy is becoming increasingly important in the data-driven world. Organizations are focusing on improving data literacy skills across their workforce through training programs, resources, and data visualization tools. This emphasis on data literacy ensures that all stakeholders can effectively leverage data to drive decision-making.

In conclusion, modern data engineering demands us to adapt to evolving trends and embrace platform-specific tools and advanced techniques. By understanding the nuances of DE, evaluating the efficiency of our data pipelines, and leveraging the right technological solutions, we can enable operational excellence and unlock the full potential of our data. So, keep exploring, innovating, and using data engineering to drive success in your organization.