Data Engineering trends 2024. What to expect!

Here's what Google Trends told me about Data Engineering in 2023

In my last live stream we looked at Google trends for 2023.

I like to do this every January, because it tells you the trend of the previous year and that can tell what is coming in the next.

What I have learned is that the only things really growing 2023 were:

🚀 Everything ChatGPT, LLMs, GenAI related. This topic is growing like crazy.

⬆️ Data Governance is growing now that companies data initiatives mature.
⬆️ Dagster growing rapidly and almost reached Airflow in trends (very surprising)
⬆️ MLOps going up as well
⬆️ Low code and No code environments
⬆️ dbt cloud and dbt core are growing

➡️ Lakehouse is slowly going up but over the last months very stable
➡️ Apache Iceberg
➡️ Apache Airflow
➡️ AWS and Azure are stable, but very close together GCP is 3rd place (a lot behind)

🔻 Spark is actually on a downward trend ( I don’t know why. Databricks too)

If you want to see more then check out the recording on YouTube: https://youtube.com/live/FzyhNMWaP_k

Excited to hear your thoughts in the comments :)

Andreas