Course Outline

Introduction

  • Apache Beam vs MapReduce, Spark Streaming, Kafka Streaming, Storm and Flink

Installing and Configuring Apache Beam

Overview of Apache Beam Features and Architecture

  • Beam Model, SDKs, Beam Pipeline Runners
  • Distributed processing back-ends

Understanding the Apache Beam Programming Model

  • How a pipeline is executed

Running a sample pipeline

  • Preparing a WordCount pipeline
  • Executing the Pipeline locally

Designing a Pipeline

  • Planning the structure, choosing the transforms, and determining the input and output methods

Creating the Pipeline

  • Writing the driver program and defining the pipeline
  • Using Apache Beam classes
  • Data sets, transforms, I/O, data encoding, etc.

Executing the Pipeline

  • Executing the pipeline locally, on remote machines, and on a public cloud
  • Choosing a runner
  • Runner-specific configurations

Testing and Debugging Apache Beam

  • Using type hints to emulate static typing
  • Managing Python Pipeline Dependencies

Processing Bounded and Unbounded Datasets

  • Windowing and Triggers

Making Your Pipelines Reusable and Maintainable

Create New Data Sources and Sinks

  • Apache Beam Source and Sink API

Integrating Apache Beam with other Big Data Systems

  • Apache Hadoop, Apache Spark, Apache Kafka

Troubleshooting

Summary and Conclusion

Requirements

  • Experience with Python Programming.
  • Experience with the Linux command line.

Audience

  • Developers
  14 Hours
 

Need help picking the right course?
Call us +971 4871 6715

Testimonials

Recalling/reviewing keypoints of the topics discussed.

Paolo Angelo Gaton - SMS Global Technologies Inc.

Sufficient hands on, trainer is knowledgable

Chris Tan

The trainer was passionate and well-known what he said I appreciate his help and answers all our questions and suggested cases.

The lab exercises. Applying the theory from the first day in subsequent days.

- Dell

The trainer was passionate and well-known what he said I appreciate his help and answers all our questions and suggested cases.

Related Courses

Samza for Stream Processing

  14 hours
Apache Samza is an open-source near-realtime, asynchronous computational framework for stream processing.  It uses Apache Kafka for messaging, and Apache Hadoop YARN for fault tolerance, processor isolation, security, and resource
Read More...

Tigon: Real-time Streaming for the Real World

  14 hours
Tigon is an open-source, real-time, low-latency, high-throughput, native YARN, stream processing framework that sits on top of HDFS and HBase for persistence. Tigon applications address use cases such as network intrusion detection and analytics,
Read More...

Real-Time Stream Processing with MapR

  7 hours
In this instructor-led, live training, participants will learn the core concepts behind MapR Stream Architecture as they develop a real-time streaming application. By the end of this training, participants will be able to build producer and
Read More...

Stream Processing with Kafka Streams

  7 hours
Kafka Streams is a client-side library for building applications and microservices whose data is passed to and from a Kafka messaging system. Traditionally, Apache Kafka has relied on Apache Spark or Apache Storm to process data between message
Read More...

A Practical Introduction to Stream Processing

  21 hours
Stream Processing refers to the real-time processing of "data in motion", that is, performing computations on data as it is being received. Such data is read as continuous streams from data sources such as sensor events, website user
Read More...

Building Kafka Solutions with Confluent

  14 hours
This instructor-led, live training (online or onsite) is aimed at engineers who wish to use Confluent (a distribution of Kafka) to build and manage a real-time data processing platform for their applications. By the end of this training,
Read More...

Apache Kafka for Python Programmers

  7 hours
Apache Kafka is an open-source stream-processing platform that provides a fast, reliable, and low-latency platform for handling real-time data analytics. Apache Kafka can be integrated with available programming languages such as Python. This
Read More...

Apache Flink Fundamentals

  28 hours
Apache Flink is an open-source framework for scalable stream and batch data processing. This instructor-led, live training introduces the principles and approaches behind distributed stream and batch data processing, and walks participants
Read More...

Apache NiFi for Administrators

  21 hours
Apache NiFi (Hortonworks DataFlow) is a real-time integrated data logistics and simple event processing platform that enables the moving, tracking and automation of data between systems. It is written using flow-based programming and provides a
Read More...

Apache NiFi for Developers

  7 hours
Apache NiFi (Hortonworks DataFlow) is a real-time integrated data logistics and simple event processing platform that enables the moving, tracking and automation of data between systems. It is written using flow-based programming and provides a
Read More...

Apache Storm

  28 hours
Apache Storm is a distributed, real-time computation engine used for enabling real-time business intelligence. It does so by enabling applications to reliably process unbounded streams of data (a.k.a. stream processing). "Storm is for
Read More...

Apache Apex: Processing Big Data-in-Motion

  21 hours
Apache Apex is a YARN-native platform that unifies stream and batch processing. It processes big data-in-motion in a way that is scalable, performant, fault-tolerant, stateful, secure, distributed, and easily operable. This instructor-led, live
Read More...

Apache Ignite for Developers

  14 hours
Apache Ignite is an in-memory computing platform that sits between the application and data layer to improve speed, scale, and availability. In this instructor-led, live training, participants will learn the principles behind persistent and pure
Read More...

Confluent KSQL

  7 hours
Confluent KSQL is a stream processing framework built on top of Apache Kafka. It enables real-time data processing using SQL operations. This instructor-led, live training (online or onsite) is aimed at developers who wish to implement Apache
Read More...

Spark Streaming with Python and Kafka

  7 hours
Apache Spark Streaming is a scalable, open source stream processing system that allows users to process real-time data from supported sources. Spark Streaming enables fault-tolerant processing of data streams. This instructor-led, live
Read More...