Kafka for Application Developers

Course Overview

Chapter 1. Introduction to Kafka

• Messaging Architectures – What is Messaging?
• Messaging Architectures – Steps to Messaging
• Messaging Architectures – Messaging Models
• What is Kafka?
• What is Kafka? (Contd.)
• Kafka Overview
• Kafka Overview (Contd.)
• Need for Kafka
• When to Use Kafka?
• Kafka Architecture
• Core concepts in Kafka
• Kafka Topic
• Kafka Partitions
• Kafka Producer
• Kafka Consumer
• Kafka Broker
• Kafka Cluster
• Why Kafka Cluster?
• Sample Multi-Broker Cluster
• Overview of ZooKeeper
• Kafka Cluster & ZooKeeper
• Who Uses Kafka?
• Summary

Chapter 2. Using Apache Kafka

• Installing Apache Kafka
• Configuration Files
• Starting Kafka
• Using Kafka Command Line Client Tools
• Setting up a Multi-Broker Cluster
• Using Multi-Broker Cluster
• Kafka Cluster Planning
• Kafka Cluster Planning – Producer/Consumer Throughput
• Kafka Cluster Planning – Number of Brokers (and ZooKeepers)
• Kafka Cluster Planning – Sizing for Topics and Partitions
• Kafka Cluster Planning – Sizing for Storage
• Kafka Connect
• Kafka Connect – Configuration Files
• Using Kafka Connect to Import/Export Data
• Creating a Spring Boot Producer
• Adding Kafka dependency to pom.xml
• Defining a Spring Boot Service to Send Message(s)
• Defining a Spring Boot Controller
• Testing the Spring Boot Producer
• Creating a Nodejs Consumer
• Summary

Chapter 3. Building Data Pipelines

• Building Data Pipelines
• Considerations When Building Data Pipelines
• Timeliness
• Reliability
• High and Varying Throughput
• High and Varying Throughput (Contd.)
• Data Formats
• Data Formats (Contd.)
• Transformations
• Transformations (Contd.)
• Security
• Failure Handling
• Coupling and Agility
• Ad-hoc Pipelines
• Loss of Metadata
• Extreme Processing
• Kafka Connect Versus Producer and Consumer
• Kafka Connect Versus Producer and Consumer (Contd.)
• Summary

Chapter 4. Integrating Kafka with Other Systems

• Introduction to Kafka Integration
• Kafka Connect
• Kafka Connect (Contd.)
• Running Kafka Connect
• Key Configurations for Connect workers:
• Kafka Connect API
• Kafka Connect Example – File Source
• Kafka Connect Example – File Sink
• Kafka Connector Example – MySQL to Elasticsearch
• Kafka Connector Example – MySQL to Elasticsearch (Contd.)
• Write the data to Elasticsearch
• Building Custom Connectors
• Kafka Connect – Connectors
• Kafka Connect - Tasks
• Kafka Connect - Workers
• Kafka Connect – Workers (Contd.)
• Kafka Connect - Converters and Connect’s data model
• Kafka Connect - Offset management
• Alternatives to Kafka Connect
• Alternatives to Kafka Connect (Contd.)
• Introduction to Storm
• Other Components of Spark
• Integrating Storm with Kafka
• Integrating Storm with Kafka – Sample Code
• Integrating Storm with Kafka
• Introduction to Hadoop
• Hadoop Components
• Integrating Hadoop with Kafka
• Hadoop Consumers
• Hadoop Consumers (Contd.)
• Hadoop Consumers (Contd.)
• Hadoop Consumers – Produce Topic
• Hadoop Consumers – Fetch Generated Topic
• Kafka at Uber
• Kafka at Uber (Contd.)
• Kafka at LinkedIn
• Kafka at LinkedIn – Core Kafka Services
• Kafka at LinkedIn – Core Kafka Services (Contd.)
• Kafka at LinkedIn – Libraries
• Kafka at LinkedIn – Monitoring and Stream Processing
• Summary

Chapter 5. Kafka and Schema Management

• Evolving Schema
• Protobuf (Protocol Buffers) Overview
• Avro Overview
• Managing Data Evolution Using Schemas
• Confluent Platform
• Confluent Schema Registry
• Schema Change and Backward Compatibility
• Collaborating over Schema Change
• Handling Unreadable Messages
• Deleting Data
• Segregating Public and Private Topics
• Summary

Chapter 6. Kafka Streams and KSQL

• What Kafka can be used for?
• What Kafka can be used for? (Contd.)
• What Exactly is Kafka?
• The APIs for Stream Processing
• Kafka: A Streaming Platform
• What is KSQL?
• What is KSQL? (Contd.)
• Starting KSQL
• Using the KSQL CLI
• KSQL Data Types
• Review the Structure of an Existing STREAM
• Query the STREAM
• KSQL Functions
• Writing to a Topic
• KSQL Table vs. Stream
• KSQL JOIN
• Windows in KSQL Queries
• Miscellaneous KSQL Commands
• Summary

Chapter 7. KSQL UDF and Deployment

• KSQL Custom Functions
• KSQL UDF/UDAF
• Implement a Custom Function
• Creating UDF and UDAF
• Creating UDF and UDAF (Contd.)
• UDFs and Null Handling
• UDFs and Null Handling (Contd.)
• Sample UDF Class
• Build Engine
• UDAF
• UDAF Sample Class
• Supported Types
• Deploying Custom Functions
• Using Custom Functions
• Summary

Lab Exercises

Lab 1. Kafka Basics
Lab 2. Kafka Multiple Brokers and Import/Export Messages
Lab 3. Apache Kafka with Java
Lab 4. Apache Kafka with Node.js
Lab 5. Kafka Integration With Spark
Lab 6. KSQL Create and Deploy UDF

Prerequisites

Basic understanding of messaging, cloud, development, architecture and virtualization would be beneficial.