Introduction to Data Engineering

Introduction

1. Knowledge and Experience
2. What are the topics we will cover?

Chapter 1 - The Theory.

1. What Is a Data Pipeline?
2. Data Pipelines built with Passion and Creativity
3. Storage and File Types
4. Access
5. Repeatable
6. Resilient
7. Scalable
8. In Summary

Chapter 2 - Data Pipeline Basics

1. Project Structure
2. Data Pipeline Code Structure
3. Code Readability and Organization
4. Tests.
5. Documentation
6. Containerzation
7. Architecture First
8. Review

Chapter 3 - Pipeline Architecture

1. Architecture Applied to Data
2. Data Size and Velocity
3. Calculating Compute Requirements
4. Calculating Storage Requirements
5. Understanding the End Result
6. Understanding Cost
7. Code Architecture
8. Batch vs Streaming Architecture
9. Puzzle Pieces
10. Summary

Chapter 4 - Storage

1. Access Patterns
2. SQL/NoSQL Databases vs Files.
3. File Types
4. Row vs Columnar Storage.
5. Common file types in data engineering.
6. Parquet.
7. Avro.
8. Orc.
9. CSV / Flat-file.
10. JSON
11. Compression.
12. Storage location.
13. Partitions.

Chapter 5 - Compute and Resources

1. Overview
2. RAM/Memory
3. CPU/Cores
4. Storage
5. Cluster/Nodes

Chapter 6 - Mastering SQL

1. Introduction To SQL
2. Does the type of database matter?
3. The fundamentals of SQL/Databases.
4. OLTP vs. OLAP
5. Table design/layout.
6. Table Design in Real Life.
7. Understanding Indexing Basics.
8. How to write fast/tune queries.
9. Where to look for common problems.
10. SQL Fundementals
11. Python + SQL
12. SQL Summary

Chapter 7 - Data Warehousing / Data Lakes

1. Data Warehouse vs Data Lake vs Lake House
2. Data Modeling in Data Warehouses, Data Lakes, and Lake Houses.
3. Facts and Dimensions.
4. Constraints and Schema.
5. Data Types.
6. Column Names.
7. The Role of ID’s in a Data Warehouses or Data Lake.
8. CDC / History Tracking.
9. Summary

Chapter 8 - Data Modeling

1. Data Types and Schema.
2. Data Types.
3. Example
4. Data Size.
5. Constraints.
6. Data Definitions.
7. Modeling Data Logically.
8. Logical data models lead to physical relationships.
9. Grain of Data.
10. Uniqueness of Data.
11. Access Patterns.
12. Example
13. Talking to the Business.
14. Normal Forms.
15. De-Duplication of Data.
16. Join Integrity.
17. Keys - Primary and Foreign.
18. The Idea Behind Keys.
19. Relational Databases (SQL) vs Data Lake (File Based) Modeling.
20. The number of Fact tables and Dimensions and normalization.
21. File size and table size matter in the new File-Based Data Lakes.
22. Partitions vs Indexes.
23. Walking the data model line between old and new.

Chapter 9 - Data Quality

1. What is Data Quality.
2. Reasoning about data.
3. Double meanings.
4. Data value quality.
5. Measures of Data Quality.
6. Correct Header or Column Names.
7. Correct File Formatting.
8. Correct data types.
9. Values ranges and values integrity.
10. Data Quality Applied

Chapter 10 - DevOps for Data Engineers

1. DevOps applied to Data Engineering
2. Dockerfiles and Docker-compose.
3. Unit Testing.
4. CI/CD.
5. Automation is the name of the game.
6. CI for Data Engineering

Conclusion