Apache Avro is a popular data serialization system that relies on schemas. The official Avro documentation can be found here: http://avro.apache.org/docs/current/. This post walks through an example of serializing and…
Loading text files in Spark is a very common task, and luckily it is easy to do. Below are a few examples of loading a text file (located on the…
Depending on the data you load into Hive/HDFS, some of your fields might be empty. Having Hive interpret those empty fields as nulls can be very convenient. It is easy…
Often times it is useful to know the input file name you are processing in a Hive query. This is a common if useful metadata is stored in the file…
Often when working with files in S3, you need information about all the items in a particular S3 bucket. Below is an example class that extends the AmazonS3Client class to…
Currently AWS CLI doesn’t provide support for UNIX wildcards in a command’s “path” argument. However, it is quite easy to replicate this functionality using the --exclude and --include parameters available…