Applications

Site created with Notion, Super & Cluster

← Back

Amazon EC2

Reference: https://medium.com/@josemarcialportilla/getting-spark-python-and-jupyter-notebook-running-on-amazon-ec2-dec599e1c297

Amazon Elastic Compute Cloud (Amazon EC2)

Web service providing resizable cloud computing; sort of like a virtual machine (VM)

Quick Checks

• Verify that instances are turned off to limit usage
• Verify security for ports

Workflow

1. Create EC2 Ubuntu instance on AWS
2. Connect to EC2 instance via PuTTY SSH client on Windows
3. Setup instance with applicable Python libraries, including Jupyter access, Spark & Hadoop
4. Access Jupyter Notebook for data operations
5. Terminate EC2 instance when complete

EC2 Setup Guide

Create EC2 Instance

1. Amazon Machine Image (AMI)
• Preference is an Ubuntu Server
2. Instance Type
• CPU/Memory: Specify as applicable to project requirements
3. Instance Configuration
• Number of Instances: 1, unless intent is to deploy to cluster of instances
• Storage: 8 GB General Purpose SSD (Default)
• Tag Instance

Key: name (ex. myinstance)

Value: webserver (ex. mymachine)

Note that these values are case-sensitive.

• Security Group Configuration

Create a new security group

Type: Set to specified security profile, or leave at All traffic, though it is not recommended.

4. Review Instance
• Confirm, and Launch
5. Key Pair
• Create new, or select existing, as applicable
• Key pair name: Specify unique name
• Download Key Pair .pem file

IMPORTANT: Verify this .pem file is downloaded before closing this dialog box.

6. Launch Instances

SSH Setup

The intent of this step is to remotely connect to command line of EC2 instance with an SSH (Secure Shell Connection) for Windows.

Step-by-Step Reference Guide

Connect to your Linux instance from Windows using PuTTY

After you launch your instance, you can connect to it and use it the way that you'd use a computer sitting in front of you. The following instructions explain how to connect to your instance using PuTTY, a free SSH client for Windows.

docs.aws.amazon.com

Collect Support Files

Download PuTTY

• PuTTY allows for creating a secure shell instance to the EC2 instance.

Download PuTTY: latest release (0.74)

This page contains download links for the latest released version of PuTTY. Currently this is 0.74, released on 2020-06-27. When new releases come out, this page will update to contain the latest, so this is a good page to bookmark or link to. Alternatively, here is a permanent link to the 0.74 release.

www.chiark.greenend.org.uk

• Binaries to download:
• putty.exe
• puttygen.exe

Convert Key Via PuTTYgen

Since PuTTY does not natively support the .pem format generated by EC2, so it needs to be converted by using PuTTYgen.

1. Run PuTTYgen.exe
2. Select Type of key to generate: RSA

RSA is a private-key cryptosystem for an SSL/TLS session

3. Load .pem file
4. Save private key
• Name: Specify unique
• Passphrase: Specify as needed for security profile
• Save in .ppk file format
5. Close PuTTYgen.exe

Configure SSH Client Via PuTTY

With the refined key, use PuTTY to configure the access to the EC2 instance.

1. Run PuTTY.exe

Session Configuration:

1. Host Name

Format: user_name@public_dns_name

user_name: ubuntu, for Ubuntu Server (check docs for other server usernames

public_dns_name: Public DNS from EC2 console

2. Connection type: SSH
3. Port: 22
4. Category > Connection > SSH > Auth

Browse and load .ppk file

2. Open to start SSH Client.
3. Trust the Security Alert dialog box that appears since the server being connected to is the intended one.

Access EC2 Instance With SSH Client

With PuTTY initialized and opened, shell client opens as follows:

When complete with use of EC2 instance, close shell and terminate instance on AWS Console.

Actions > Instance State > Terminate

Instance termination is critical for when the instance is no longer needed so as not to log active usage of charged services.

Jupyter & Spark Setup Guide

Overview

The intent of this process is to set up Anaconda Python, Jupyter, Spark, and Hadoop within the EC2 Instance.

Reference Step-by-Step Guide

Getting Spark, Python, and Jupyter Notebook running on Amazon EC2

You should now have successfully connected to the command line of your virtual Ubuntu instance running on EC2. The rest of the guide will tell you commands to put into this terminal. Next we will download and install Anaconda for our Python. You can replace the version numbers here with whatever version you prefer (2 or 3).

medium.com

Anaconda & Python

Install Anaconda

This process installs Anaconda and the Jupyter notebook with its own Python and all accompanying libraries to the specified directory.

1. Access the EC2 instance with PuTTY.
2. Specify version of Anaconda from the following link of available versions to acquire the weblink: https://repo.anaconda.com/archive/
Copy

ubuntu@ip:~$ wget <weblink_to_anaconda.sh_installer>
ubuntu@ip:~$ bash <anaconda.sh_installer>

3. Confirm install to /home/ubuntu/anaconda3
4. Press Enter key to navigate through agreement and enter yes for agreement acceptance.
5. Enter yes for installer to prepend the Anaconda3 install location to PATH in /home/ubuntu/.bashrc.

Python Versions

While the newly installed Anaconda has its own version of Python, Ubuntu already comes packaged with Python, so the following will confirm which Python is being actively utilized.

ubuntu@ip:~$ which python

When checking the version of Python, the default Ubuntu version is located at /usr/bin/python, while Anaconda's version is located at /home/ubuntu/anaconda3/bin/python. The following changes the active Python to the Anaconda version.

ubuntu@ip:~$ source .bashrc

Jupyter Notebook

Configuration

While Jupyter comes with Anaconda, it needs to be configured for use within this EC2 Ubuntu environment.

1. Create the Jupyter Notebook configuration file.
Copy

ubuntu@ip:~$ jupyter notebook --generate-config

Note that the configuration file is created and stored at /home/ubuntu/.jupyter/jupyter_notebook_config.py

2. Create a directory for storing certifications.
Copy

ubuntu@ip:~$ jupyter notebook --generate-config

3. Generate the applicable certification files.
Copy

ubuntu@ip:~$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem

4. Populate the fields for the newly created certification file/s, as applicable.
Copy

Country Name (2 letter code) [AU]: 
State or Province Name (full name) [Some-State]: 
Locality Name (e.g., city) []:
Organization Name (e.g., company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (e.g., section) []: 
Common Name (e.g., server FQDN or YOUR name) []: 
Email Address []: 

Document the location where this .pem file is located on the EC2 instance; it will be needed in the next step.

Permissions

Before attempting to access the Jupyter Notebook, note that the ownership of the mycert.pem file needs to be addressed by way of the following. Change directory to the certs directory and input the following to change the group ownership of the certification file to the 'ubuntu' user.

ubuntu@ip:~$ sudo chown ubuntu:root mycert.pem

The above is noted in several troubleshooting documentation sites, as noted below. The Jupyter Notebook has issues being accessed via the browser and notes the following error. However the above step has shown to resolve the permission issue required to bypass the error.

Supporting Frameworks & Libraries

In support of the Conda environment, the following need to be installed on the EC2 instance, if not already accessible:

• Java
• Scala
• pip for Conda
• py4j
• findspark
• Spark
• Hadoop

pip for Conda

1. Install Anaconda's version of pip
Copy

ubuntu@ip:~$ conda install pip

2. Confirm pip installation
Copy

ubuntu@ip:~$ which pip

It should be installed to the /home/ubuntu/anaconda3/bin/pip directory.

• Confirm which Python libraries are installed to Conda
Copy

ubuntu@ip:~$ conda list

py4j Library

This library allows Python to connect to Java

• Install py4j library
Copy

ubuntu@ip:~$ pip install py4j

findspark Library

This library makes pyspark importable as a regular library.

• Install findspark Library
Copy

ubuntu@ip:~$ pip install -q findspark

Jupyter Notebook Utilization

Once applicable libraries and configurations are setup, access the Jupyter Notebook with

ubuntu@ip:~$ jupyter notebook

Creating a new notebook should provide the opportunity to test the configuration and installation for using Spark.

Close Jupyter Notebook

Back in the shell, press CTRL + C to exit Jupyter Notebook.

Enter y to confirm notebook shutdown

Terminate EC2 Instance

Right-click on instance in AWS Console and Terminate.

On This Page

• Amazon EC2
• Quick Checks
• Workflow
• EC2 Setup Guide
• Create EC2 Instance
• SSH Setup
• Collect Support Files
• Convert Key Via PuTTYgen
• Configure SSH Client Via PuTTY
• Access EC2 Instance With SSH Client
• Jupyter & Spark Setup Guide
• Overview
• Anaconda & Python
• Install Anaconda
• Python Versions
• Jupyter Notebook
• Configuration
• Permissions
• Supporting Frameworks & Libraries
• pip for Conda
• py4j Library
• findspark Library
• Jupyter Notebook Utilization
• Close Jupyter Notebook
• Terminate EC2 Instance