MyCgDoc
MyCgDoc

Amazon EC2 Instance

About

← Back

Amazon EC2

Reference: https://medium.com/@josemarcialportilla/getting-spark-python-and-jupyter-notebook-running-on-amazon-ec2-dec599e1c297

Amazon Elastic Compute Cloud (Amazon EC2)

Web service providing resizable cloud computing; sort of like a virtual machine (VM)

Quick Checks

  • Verify that instances are turned off to limit usage
  • Verify security for ports

Workflow

  1. Create EC2 Ubuntu instance on AWS
  2. Connect to EC2 instance via PuTTY SSH client on Windows
  3. Setup instance with applicable Python libraries, including Jupyter access, Spark & Hadoop
  4. Access Jupyter Notebook for data operations
  5. Terminate EC2 instance when complete

EC2 Setup Guide

❗
There are many guides available online documenting similar processes and they may differ in configurations and successful deployment. Regardless, the following is my reference that I have been able to use to set up an EC2 Ubuntu instance for use with Spark.

Create EC2 Instance

  1. Amazon Machine Image (AMI)
    • Preference is an Ubuntu Server
  2. Instance Type
    • CPU/Memory: Specify as applicable to project requirements
  3. Instance Configuration
    • Number of Instances: 1, unless intent is to deploy to cluster of instances
    • Storage: 8 GB General Purpose SSD (Default)
    • Tag Instance

      • Key: name (ex. myinstance)

      Value: webserver (ex. mymachine)

      Note that these values are case-sensitive.

    • Security Group Configuration

      • Create a new security group

      Type: Set to specified security profile, or leave at All traffic, though it is not recommended.

  4. Review Instance
    • Confirm, and Launch
  5. Key Pair
    • Create new, or select existing, as applicable
    • Key pair name: Specify unique name
    • Download Key Pair .pem file

      • IMPORTANT: Verify this .pem file is downloaded before closing this dialog box.

  6. Launch Instances

SSH Setup

The intent of this step is to remotely connect to command line of EC2 instance with an SSH (Secure Shell Connection) for Windows.

Step-by-Step Reference Guide

Collect Support Files

Download PuTTY

  • PuTTY allows for creating a secure shell instance to the EC2 instance.
  • Binaries to download:
    • putty.exe
    • puttygen.exe

Collect EC2 information from AWS Dashboard

  • Instance ID
  • Public DNS
  • Private Key (.pem file)
  • Enable inbound SSH traffic

    • This operation is dependent on the security profile specified, but if All traffic was selected in instance creation, then this has been enabled.

Convert Key Via PuTTYgen

Since PuTTY does not natively support the .pem format generated by EC2, so it needs to be converted by using PuTTYgen.

  1. Run PuTTYgen.exe
  2. Select Type of key to generate: RSA

    3. RSA is a private-key cryptosystem for an SSL/TLS session

  3. Load .pem file
  4. Save private key
    • Name: Specify unique
    • Passphrase: Specify as needed for security profile
    • Save in .ppk file format
  5. Close PuTTYgen.exe

Configure SSH Client Via PuTTY

With the refined key, use PuTTY to configure the access to the EC2 instance.

  1. Run PuTTY.exe

      2. Session Configuration:

    1. Host Name

      2. Format: user_name@public_dns_name

      user_name: ubuntu, for Ubuntu Server (check docs for other server usernames

      public_dns_name: Public DNS from EC2 console

    2. Connection type: SSH
    3. Port: 22
    4. Category > Connection > SSH > Auth

      5. Browse and load .ppk file

  2. Open to start SSH Client.
  3. Trust the Security Alert dialog box that appears since the server being connected to is the intended one.

Access EC2 Instance With SSH Client

With PuTTY initialized and opened, shell client opens as follows:

Using username "ubuntu".
Authenticating with public key "imported-openssh-key"
Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-1029-aws x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Sun Feb  7 17:01:21 UTC 2021

  System load:  0.0               Processes:             99
  Usage of /:   16.8% of 7.69GB   Users logged in:       0
  Memory usage: 19%               IPv4 address for eth0: ???.??.??.???
  Swap usage:   0%

1 update can be installed immediately.
0 of these updates are security updates.
To see these additional updates run: apt list --upgradable


The list of available updates is more than a week old.
To check for new updates run: sudo apt update


The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@ip-???-??-??-???:~$ python3
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()

When complete with use of EC2 instance, close shell and terminate instance on AWS Console.

Actions > Instance State > Terminate

Instance termination is critical for when the instance is no longer needed so as not to log active usage of charged services.

Jupyter & Spark Setup Guide

Overview

The intent of this process is to set up Anaconda Python, Jupyter, Spark, and Hadoop within the EC2 Instance.

Reference Step-by-Step Guide

Anaconda & Python

Install Anaconda

This process installs Anaconda and the Jupyter notebook with its own Python and all accompanying libraries to the specified directory.

  1. Access the EC2 instance with PuTTY.
  2. Specify version of Anaconda from the following link of available versions to acquire the weblink: https://repo.anaconda.com/archive/
    3. ubuntu@ip:~$ wget <weblink_to_anaconda.sh_installer>
ubuntu@ip:~$ bash <anaconda.sh_installer>
  3. Confirm install to /home/ubuntu/anaconda3
  4. Press Enter key to navigate through agreement and enter yes for agreement acceptance.
  5. Enter yes for installer to prepend the Anaconda3 install location to PATH in /home/ubuntu/.bashrc.

Python Versions

While the newly installed Anaconda has its own version of Python, Ubuntu already comes packaged with Python, so the following will confirm which Python is being actively utilized.

ubuntu@ip:~$ which python

When checking the version of Python, the default Ubuntu version is located at /usr/bin/python, while Anaconda's version is located at /home/ubuntu/anaconda3/bin/python. The following changes the active Python to the Anaconda version.

ubuntu@ip:~$ source .bashrc

Jupyter Notebook

Configuration

While Jupyter comes with Anaconda, it needs to be configured for use within this EC2 Ubuntu environment.

  1. Create the Jupyter Notebook configuration file.
    2. ubuntu@ip:~$ jupyter notebook --generate-config

    Note that the configuration file is created and stored at /home/ubuntu/.jupyter/jupyter_notebook_config.py

  2. Create a directory for storing certifications.
    3. ubuntu@ip:~$ jupyter notebook --generate-config
  3. Generate the applicable certification files.
    4. ubuntu@ip:~$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem
  4. Populate the fields for the newly created certification file/s, as applicable.
    5. Country Name (2 letter code) [AU]: 
State or Province Name (full name) [Some-State]: 
Locality Name (e.g., city) []:
Organization Name (e.g., company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (e.g., section) []: 
Common Name (e.g., server FQDN or YOUR name) []: 
Email Address []:

    Document the location where this .pem file is located on the EC2 instance; it will be needed in the next step.

Permissions

Before attempting to access the Jupyter Notebook, note that the ownership of the mycert.pem file needs to be addressed by way of the following. Change directory to the certs directory and input the following to change the group ownership of the certification file to the 'ubuntu' user.

ubuntu@ip:~$ sudo chown ubuntu:root mycert.pem

The above is noted in several troubleshooting documentation sites, as noted below. The Jupyter Notebook has issues being accessed via the browser and notes the following error. However the above step has shown to resolve the permission issue required to bypass the error.

Error:

Permission Error: [Errno 13] Permission denied

Resources:

Supporting Frameworks & Libraries

In support of the Conda environment, the following need to be installed on the EC2 instance, if not already accessible:

  • Java
  • Scala
  • pip for Conda
  • py4j
  • findspark
  • Spark
  • Hadoop

pip for Conda

  1. Install Anaconda's version of pip
    2. ubuntu@ip:~$ conda install pip
  2. Confirm pip installation
    3. ubuntu@ip:~$ which pip

    It should be installed to the /home/ubuntu/anaconda3/bin/pip directory.

  • Confirm which Python libraries are installed to Conda 
    • ubuntu@ip:~$ conda list

py4j Library

This library allows Python to connect to Java

  • Install py4j library
    • ubuntu@ip:~$ pip install py4j

findspark Library

This library makes pyspark importable as a regular library.

  • Install findspark Library
    • ubuntu@ip:~$ pip install -q findspark

Jupyter Notebook Utilization

Once applicable libraries and configurations are setup, access the Jupyter Notebook with

ubuntu@ip:~$ jupyter notebook

and by accessing the following web address:

https://<EC2_instance_public_ip>:8888

Creating a new notebook should provide the opportunity to test the configuration and installation for using Spark.

image

Close Jupyter Notebook

Back in the shell, press CTRL + C to exit Jupyter Notebook.

Enter y to confirm notebook shutdown

Terminate EC2 Instance

Right-click on instance in AWS Console and Terminate.