The Data Engineering Roadmap
Here’s a roadmap that can help you get better at data engineering:
Programming Languages
Learn SQL and a few programming languages like Python, Java, and Scala.
Processing Techniques
Learn batch processing tools like Spark and Hadoop and stream processing tools like Flink and Kafka.
Databases
Focus on both relational and non-relational databases. Some examples are MySQL, Postgres, MongoDB, Cassandra, and Redis.
Messaging Platforms
Master the use of platforms like Kafka, RabbitMQ, and Pulsar.
Data Lakes and Warehouses
Learn about various data lake and warehousing solutions such as Snowflake, Hive, S3, Redshift, and Clickhouse. Also, learn about Normalization, Denormalization, and OLTP vs OLAP.
Cloud Computing Platforms
Master the use of cloud platforms like AWS, Azure, Docker, and K8S
Storage Systems
Learn about the key storage systems like S3, Azure Data Lake, and HDFS
Orchestration Tools
Learn about orchestration tools like Airflow, Jenkins, and Luigi
Automation and Deployments
Learn automation tools such as Jenkins, Github Actions, and Terraform.
Frontend and Dashboarding
Master the use of tools like Jupyter Notebooks, PowerBI, Tableau, and Plotty
Over to you: What else will you add to the Data Engineering Roadmap?
