Why Big Data is Hard?
Yes, Big Data is hard. It's not hard only because of the number of technologies a good data engineering team needs to master technologies such as Spark , Flink , and Kafka Streams (Batch and Streaming), Hadoop , HDFS , and Hive if you have a DW legacy(most likely you do) and the Data Science part of it with Discovery and Execution at Scale. There are needs for different kinds of storage and Design/Modeling, and thats, not even the hard part. The technology landscape gets bigger and bigger as time pass. We have many specializations such as Frontend/Mobile engineering, Backend Engineering, Architecture, DevOps (Which is a movement, not a department, but all companies decide is a role, so you know what I mean), QA(a dying one? ), Product, Management and Data Engineering which often has Data Scientists working with Data Engineers. To some degree, Data Engineering and Data Science have the same issues as Product has today. Unfortunately, the product folks still too much about project m...
