Data Infrastructure with Apache Kafka E-Learning Training 

In de Data Infrastructuur met Apache Kafka LearningKIT verkent u Apache Kafka, integreert u Kafka met Python en gebruikt u consumentengroepen, integreert u Apache Kafka met Spark en gebruikt u Kafka met Cassandra en Confluent. U verkent de Kafka architectuur voor event streaming, het opzetten van topics, het creëren van brokers en het afhandelen van berichten. U leert ook hoe u met Kafka berichten kunt produceren en consumeren, en hoe u Kafka-brokerconfiguraties kunt aanpassen. Deze cursus richt zich ook op prestatieoptimalisatie van Kafka en gestructureerde streaming met Apache Spark, waaronder het bouwen van Spark-applicaties voor het verwerken van gegevens die naar Kafka-onderwerpen worden gestreamd met behulp van Data Frames en de integratie van Kafka met Spark en Cassandra voor NoSQL-gegevens.

Deze LearningKit met meer dan 7 leeruren is verdeeld in drie sporen:

Cursusinhoud

Track 1: Intro to Data Infrastructure

In this track, the focus will be on data infrastructure in an organization, data mesh architecture, data tools, messaging platforms, and data stores.
Courses (½ hour +):

Setting up the Data Infrastructure in an Organization

Course: 46 Minutes

• Course Overview
• Data Infrastructure in an Organization
• Data Mesh Architecture
• Tools for Data Management
• Messaging Platforms
• Data Stores
• Course Summary

Track 2: Apache Kafka

In this track, the focus will be on Apache Kafka and Apache Spark. Apache Kafka is a popular event streaming platform used by Fortune 100 companies for both real-time and batch data processing. Apache Spark is a powerful distributed data processing engine that can handle petabytes of data by chunking that data and dividing across a cluster of resources.
Courses (7 hours +)

Processing Data: Getting Started with Apache Kafka

Course: 1 Hour, 32 Minutes

• Course Overview
• Work with Real-time Data
• Stream Events with Kafka
• Kafka Topics
• Downloading and Installing Kafka
• Creating a Topic, Producer, and Consumer with Kafka
• Working with Multiple Kafka Topics
• Configuring a Multi-node Kafka Cluster
• Monitoring a Kafka Cluster
• Using Partitions and Replicas with Kafka
• Course Summary

Processing Data: Integrating Kafka with Python & Using Consumer Groups

Course: 1 Hour, 24 Minutes

• Course Overview
• Developing Kafka Producers and Consumers in Python
• Processing Messages at the Consumer
• Tweaking Kafka Broker Configurations
• Defining Automatic Topic Creation in Kafka
• Generating Fake Data for Kafka Consumption
• Setting a Destination Partition in Kafka
• Kafka Consumer Groups
• Creating and Using Consumer Groups in Kafka
• Working with Consumer Groups and Partitions in Kafka
• Kafka Configuration
• Course Summary

Processing Data: Introducing Apache Spark

Course: 1 Hour, 44 Minutes

• Course Overview
• Apache Spark
• Apache Spark Architecture
• Structured Streaming in Apache Spark
• Downloading and Installing Spark
• Deploying a Spark Cluster
• Launching a Spark Job
• Monitoring Spark Apps with the Web UI
• Configuring a Spark Cluster
• Building a Spark Streaming App
• Running Apps on a Standalone Cluster
• Running Apps on Spark Local
• Course Summary

Processing Data: Integrating Kafka with Apache Spark

Course: 1 Hour, 46 Minutes

• Course Overview
• Integrating Spark with Kafka
• Transforming Kafka Messages with PySpark
• Reading from Multiple Kafka Topics
• Setting up a Producer and Consumer with Kafka
• Publishing to Kafka from PySpark
• Transforming Data with Spark SQL
• Aggregations on Streaming Data
• Exploring Grouping and Ordering
• Defining Window Operations
• Creating Tumbling and Sliding Windows
• Course Summary

Processing Data: Using Kafka with Cassandra & Confluent

Course: 42 Minutes

• Course Overview
• Installing and Setting up Apache Cassandra
• Integrating Spark with Kafka and Cassandra
• Confluent and Kafka
• Setting up the Confluent Platform
• Working with Kafka Using Confluent
• Course Summary

Assessment:

• Final Exam: Apache Kafka