Important Changes About Nested Fields — Spark 3.1.1 release comes with a lot of new things! Check out, for example, all the Kubernetes-related tasks done for this release. As you can probably imagine from the title of this post we are not going to talk about Kubernetes but Nested Fields. If you have been working with…

Differences between array sorting techniques in Spark 3.0 — Earlier last year(2020) I had the need to sort an array, and I found that there were two functions, very similar in name, but different in functionality. These are array_sort and sort_array. Which one to use? …

In this post, I’ll break down the new output of the explain command. But first, let’s see how it was before 3.0. Some people might have never used explain, but it is really useful when you want to know what’s really happening under the hood. Sometimes we write beautiful code…

Background I’ve been working in Big Data now for 5 years, so when I decided to get the certification I was already familiar with some terms like Data Warehouse, networks, caches, performance, etc. I also knew some AWS services but more in relation to the Big Data world. Let’s start with…