Posts

Create Git branch - GitHub

  Steps to create a branch and push files: 1. first clone master branch  git clone repo_url 2. create branch  git branch branch_name 3. push branch to remote git push -u origin branch_name Here your branch will be visible in branches, next you want to add on something in your branch and push to remote. Let's move to step 4> 4. Now to start work on your own branch, first you need to switch to your branch git checkout mnt_code-revamp-v1 5. to check your current working branch, it will show you all files and mark * on current one git branch -a git branch --show-current (Git 2.22 and above) 5.1 Add data/ files to local repository and then push to git branch Add file  : git add .  commit those changes:  git commit -m requirements.txt Push file to branch:  git push requirements.txt 5.2  Add folder to local repository and then push to git branch git add * git commit .....pop for enter a message ....write message and hit Esc>:wq! Enter git status 5....

Spark hands-on with Databricks

Image
Date: 10th Feb, 2021 I started spark hands-on once I finished with basic architecture and working of spark core I needed to install spark on my system but somehow couldn't manage hence thought of starting on databricks, which I find really amazing , your spark setup is just few clicks away and we can try spark with many resources: 1. RDS 2. S3 3. Databricks file system 4. Upload file from local system Set up working environment: Step1: Create Cluster Select Cluster from LHS menu and fill up the required settings as per your version availability. Spark 3.0 was latest when I was trying hence I selected spark 3.0 as shown in below picture: Once filled up click on Create Cluster button on top and wait for few minutes... Once cluster has been created add Data  Step2: Add Data source We have so many options as a data source, I chose to upload file from my local to get my hands dirty with Movie dataset (http://media.sundog-soft.com/es/ml-100k.zip)