Loading Data from a Database into Spark
One of the great features of Spark is the variety of data sources it can read from. Loading data from a database into Spark using JDBC requires 3 major steps.…
0 Comments
January 25, 2021
Set AWS Credentials in Cloudera Quickstart Docker Container
Cloudera’s Quickstart Image is a fantastic way to get started quickly with the big data ecosystem. With software such as Hadoop, Spark, Hive, Pig, Impala, and Hue already set up,…
January 25, 2021
What is a UUID?
UUID stands for Universally Unique Identifier. UUIDs are used as IDs (to identify) unique objects or records. These are very common in a big data environment where coordinating unique IDs…
January 25, 2021
How to Copy local files to S3 with AWS CLI
AWS CLI has made working with S3 very easy. Once you get AWS CLI installed you might ask “How do I start copying local files to S3?” The syntax for…
January 25, 2021
Copy all Files in S3 Bucket to Local with AWS CLI
The AWS CLI makes working with files in S3 very easy. However, the file globbing available on most Unix/Linux systems is not quite as easy to use with the AWS…
January 25, 2021
How to do Total Order Sorting in Hadoop MapReduce
Being able to sort by all keys in a data set is a common need in the world of big data. Those familiar with Hive or relational databases know that…
January 25, 2021
How to Create a Custom Writable for Hadoop
If you have gone through other Hadoop MapReduce examples, you will have noticed the use of “Writable” data types such as LongWritable, IntWritable, Text, etc… All values in used in…
January 25, 2021