Intermediate Data Engineering with Python

Course Overview

Chapter 1. Introduction to Apache Spark

• What is Apache Spark
• The Spark Platform
• Spark vs Hadoop's MapReduce (MR)
• Common Spark Use Cases
• Languages Supported by Spark
• Running Spark on a Cluster
• The Spark Application Architecture
• The Driver Process
• The Executor and Worker Processes
• Spark Shell
• Jupyter Notebook Shell Environment
• Spark Applications
• The spark-submit Tool
• The spark-submit Tool Configuration
• Interfaces with Data Storage Systems
• Project Tungsten
• The Resilient Distributed Dataset (RDD)
• Datasets and DataFrames
• Spark SQL, DataFrames, and Catalyst Optimizer
• Spark Machine Learning Library
• GraphX
• Extending Spark Environment with Custom Modules and Files
• Summary

Chapter 2. The Spark Shell

• The Spark Shell
• The Spark v.2 + Command-Line Shells
• The Spark Shell UI
• Spark Shell Options
• Getting Help
• Jupyter Notebook Shell Environment
• Example of a Jupyter Notebook Web UI (Databricks Cloud)
• The Spark Context (sc) and Spark Session (spark)
• Creating a Spark Session Object in Spark Applications
• The Shell Spark Context Object (sc)
• The Shell Spark Session Object (spark)
• Loading Files
• Saving Files
• Summary

Chapter 3. Spark RDDs

• The Resilient Distributed Dataset (RDD)
• Ways to Create an RDD
• Supported Data Types
• RDD Operations
• RDDs are Immutable
• Spark Actions
• RDD Transformations
• Other RDD Operations
• Chaining RDD Operations
• RDD Lineage
• The Big Picture
• What May Go Wrong
• Miscellaneous Pair RDD Operations
• RDD Caching
• Summary

Chapter 4. Introduction to Spark SQL

• What is Spark SQL?
• Uniform Data Access with Spark SQL
• Hive Integration
• Hive Interface
• Integration with BI Tools
• What is a DataFrame?
• Creating a DataFrame in PySpark
• Commonly Used DataFrame Methods and Properties in PySpark
• Grouping and Aggregation in PySpark
• The "DataFrame to RDD" Bridge in PySpark
• The SQLContext Object
• Examples of Spark SQL / DataFrame (PySpark Example)
• Converting an RDD to a DataFrame Example
• Example of Reading / Writing a JSON File
• Using JDBC Sources
• JDBC Connection Example
• Performance, Scalability, and Fault-tolerance of Spark SQL
• Summary

Chapter 5. Overview of the Amazon Web Services (AWS)

• Amazon Web Services
• The History of AWS
• The Initial Iteration of Moving amazon.com to AWS
• The AWS (Simplified) Service Stack
• Accessing AWS
• Direct Connect
• Shared Responsibility Model
• Trusted Advisor
• The AWS Distributed Architecture
• AWS Services
• Managed vs Unmanaged Amazon Services
• Amazon Resource Name (ARN)
• Compute and Networking Services
• Elastic Compute Cloud (EC2)
• AWS Lambda
• Auto Scaling
• Elastic Load Balancing (ELB)
• Virtual Private Cloud (VPC)
• Route53 Domain Name System
• Elastic Beanstalk
• Security and Identity Services
• Identity and Access Management (IAM)
• AWS Directory Service
• AWS Certificate Manager
• AWS Key Management Service (KMS)
• Storage and Content Delivery
• Elastic Block Storage (EBS)
• Simple Storage Service (S3)
• Glacier
• CloudFront Content Delivery Service
• Database Services
• Relational Database Service (RDS)
• DynamoDB
• Amazon ElastiCache
• Redshift
• Messaging Services
• Simple Queue Service (SQS)
• Simple Notifications Service (SNS)
• Simple Email Service (SES)
• AWS Monitoring with CloudWatch
• Other Services Example
• Summary

Chapter 6. Introduction to AWS Glue

• What is AWS Glue?
• AWS Glue Components
• AWS Glue Components (Cont'd)
• Managing Notebooks
• AWS Glue Components (Cont'd)
• Putting it Together: The AWS Glue Environment Architecture
• AWS Glue Main Activities
• Additional Glue Services
• When To Use AWS Glue?
• Integration with other AWS Services
• Summary

Chapter 7. Introduction to Apache Spark

• What is Apache Spark
• The Spark Platform
• Uniform Data Access with Spark SQL
• Common Spark Use Cases
• Languages Supported by Spark
• Running Spark on a Cluster
• The Spark Application Architecture
• The Driver Process
• The Executor and Worker Processes
• Spark Shell
• Jupyter Notebook Shell Environment
• Interfaces with Data Storage Systems
• The Resilient Distributed Dataset (RDD)
• Datasets and DataFrames
• Data Partitioning
• Data Partitioning Diagram
• Summary

Chapter 8. AWS Glue PySpark Extensions

• AWS Glue and Spark
• The DynamicFrame Object
• The DynamicFrame API
• The GlueContext Object
• Glue Transforms
• A Sample Glue PySpark Script
• Using PySpark
• AWS Glue PySpark SDK
• Summary

Lab Exercises

Lab 1. Learning the Databricks Community Cloud Lab Environment
Lab 2. Data Visualization and EDA with pandas and seaborn
Lab 3. Correlating Cause and Effect
Lab 4. Learning PySpark Shell Environment
Lab 5. Understanding Spark DataFrames
Lab 6. Learning the PySpark DataFrame API
Lab 7. Data Repair and Normalization in PySpark
Lab 8. Working with Parquet File Format in PySpark and pandas
Lab 9. AWS Glue Overview
Lab 10. AWS Glue Crawlers and Classifiers
Lab 11. Creating an S3 Bucket for AWS Glue ETL Script Output
Lab 12. Creating and Working with Glue Scripts Using Dev Endpoints
Lab 13. Using PySpark API Directly
Lab 14. Understanding AWS Glue ETL Jobs

Prerequisites

Participants must have practical experience coding in Python or another modern programming language. Knowledge of AWS Management Console is desirable but not necessary. The students are expected to be able to quickly learn the new material and reinforce the knowledge of a learned topic by doing programming exercises (labs).