People in the Big Data field are used to a changing landscape. The year 2017 was no exception as it brought many changes in our way of processing, understanding and interacting with data. These are some of the highlights of what happened:
If you’ve worked with any relational database system (DB2, Oracle, Postgres) or even with traditional Hadoop/Hive, you know that most ingestion processes work in batches where data is processed one block or time unit at a time, be it monthly, daily or hourly. Streaming tools existed previously (Spark has been doing this for some time), but 2017 is definitely the year where stream processing gained traction. Apache Kafka also reached the 1.0 milestone and gained exactly once semantics, which took the tools’ reliability one step further. These two tools (Kafka + Spark) are becoming the core of many business processes and we will definitely see them around for quite some time.
Switch from Data Analytics to Artificial Intelligence
We had already seen Advanced Data Analytics being used to analyze complex datasets and obtain forecast values, build decision trees, detect patterns or groups (cluster) within, etc. In 2017 we began to see massive uses for image processing, with the popularization of libraries like TensorFlow (which also reached the 1.0 milestone in 2017) or Keras (which reached 2.0 and got Tensorflow integration). Also, the term “Artificial Intelligence” is gaining more and more acceptance as we see computers begin to imitate art or create music. Some people even began to see this as a threat, such as Tesla’s CEO Elon Musk address to the US National Governors Association. Will we reach the Singularity anytime soon?
Personal assistants are already very present in the market (Apple’s Siri or Amazon’s Alexa are probably the best known), but in 2017 two factors contributed greatly to their advance. First, a huge effort was done by many of the companies behind assistants to deploy them internationally, thus significantly expanding the user base. Secondly, many of these companies also started supplying SDK’s or other cloud-based tools to allow third parties to be able to create their own voice-based virtual assistants without having to create them from scratch, so we will definitely see a boom in their number and quality.
Finally, although we didn’t see many real-world applications, we did notice a general consolidation around Industry 4.0. The necessary technology is already available and many of the ideas presented during 2017 will surely see use cases in 2018. As always, Datatons will be there trying to help you all in the Digital Transformation!