Spark hands-on with Databricks

Date: 10th Feb, 2021

I started spark hands-on once I finished with basic architecture and working of spark core I needed to install spark on my system but somehow couldn't manage hence thought of starting on databricks, which I find really amazing , your spark setup is just few clicks away and we can try spark with many resources:

1. RDS

2. S3

3. Databricks file system

4. Upload file from local system

Set up working environment:

Step1: Create Cluster

Select Cluster from LHS menu and fill up the required settings as per your version availability. Spark 3.0 was latest when I was trying hence I selected spark 3.0 as shown in below picture:

Once filled up click on Create Cluster button on top and wait for few minutes...

Once cluster has been created add Data 

Step2: Add Data source

We have so many options as a data source, I chose to upload file from my local to get my hands dirty with Movie dataset (http://media.sundog-soft.com/es/ml-100k.zip)