Course Outline
Introduction
Overview of Spark Streaming Features and Architecture
- Supported data sources
- Core APIs
Preparing the Environment
- Dependencies
- Spark and streaming context
- Connecting to Kafka
Processing Messages
- Parsing inbound messages as JSON
- ETL processes
- Starting the streaming context
Performing a Windowed Stream Processing
- Slide interval
- Checkpoint delivery configuration
- Launching the environment
Prototyping the Processing Code
- Connecting to a Kafka topic
- Retrieving JSON from data source using Paw
- Variations and additional processing
Streaming the Code
- Job control variables
- Defining values to match
- Functions and conditions
Acquiring Stream Output
- Counters
- Kafka output (matched and non-matched)
Troubleshooting
Summary and Conclusion
Requirements
- Experience with Python and Apache Kafka
- Familiarity with stream-processing platforms
Audience
- Data engineers
- Data scientists
- Programmers
Testimonials
I preferred the exercise and learning about the nooks and crannies of Python.
Connor Brierley-Green
Joey has an infectious enthusiasm about programming. And he was very good at adapting to our needs and interests on the fly.
Randy Enkin
Many examples made me easy to understand.
Lingmin Cao
Fact that customization was taken seriously.
jurgen linsen
I did like the exercises.
Office for National Statistics
I liked the helpful and very kind.
Natalia Machrowicz
We did practical exercises (the scripts we wrote can be used in our everyday work). It made the course very interesting. I also liked the way the trainer shared his knowledge. He did it in a very accessible way.
Malwina Sawa
* Enjoyable exercises. * Quickly moved into more advanced topics. * Trainer was friendly and easy to get on with. * Customized course for needs of team.
Matthew Lucas
I enjoyed the felixibility to add specific topics into the course / lessons.
Marc Ammann
In-depth coverage of machine learning topics, particularly neural networks. Demystified a lot of the topic.
Sacha Nandlall
The case studies helped us understand how we can apply Python in the industry. Really appreciated the trainer's help during the exercises.
Rajiv Dhingra - TCS
As we are PHP developers, he understood the situation and allowed us to slowly map things between. I liked the examples and the humor he added.
Soumya Tyagi - TCS
I genuinely enjoyed the lots of labs and practices.
Vivian Feng - Destination Canada
The exercises/labs were tailored to our own organizational needs.
- Destination Canada
I generally liked the subject matter.
- Destination Canada
The trainer was sharing real word experiences, it's nice to learn from real professional.
- Fednot
The trainer was excellent, He was always ready to answer my questions and share as much knowledge as he could.
Fahad Malalla - Tatweer Petroleum
1:1 very intensive but learnt a lot.
Karen Dyke - BT
I mostly enjoyed the subject.
- Proximus
The way the exercises were organized : all on own tempo and Antonio there to help you further.
- Proximus
I liked the sufficient and very detailed reading materials and examples (slides).
- HC Consumer Finance Philippines, Inc.
I genuinely liked the na.
- HC Consumer Finance Philippines, Inc.
What I like the most about the training is that everything in the course outline is something that will be useful for our projects.
Joanna Marie Escueta - Aarki, Inc.
The overview/the recommendations
frddy de meersman - Proximus
Labs
- Proximus
The informal exchanges we had during the lectures really helped me deepen my understanding of the subject
- Explore
practice tasks
Pawel Kozikowski - GE Medical Systems Polska Sp. Zoo
Recap of previous day, trainer very knowledgable in answering questions
Mateusz Jaros - GE Medical Systems Polska Sp. Zoo
It gave me a broad overview of the possibilites
- GE Medical Systems Polska Sp. Zoo
really kind, good approach to trainees, helpful
- GE Medical Systems Polska Sp. Zoo
I like pace of the training. It was good and we were able to cover many aspects of programming language. Trainer was able to show many applications of Python in very informative way. Trainer sent to us many scripts and micro-programs for furher reference which is very useful. I like, that we started training with some technical remarks and setting up virtual environment.
Bartosz Rosiek - GE Medical Systems Polska Sp. Zoo
I thought John was very knowledgeable and able to diseminate information in a very understandable way.
- Crux Product Design
John was a very friendly and knowledgeable trainer and was keen to adapt the course to our requests.
- Crux Product Design
Gaining a better understanding of object oriented programming as this is a key difference to programming in Matlab (which I am much more familiar with). The training should hopefully be very useful!
- Crux Product Design
knew his subject well
Albert JACOB - Proximus
The exercises combined with the experienced help of the trainer
- Proximus
The fact that we could practice a lot. Even though for me being a newbe the pace was to fast and explanation too few. However, probably due to the mixed knowkedge level of the students attending the class.
- Proximus
Trainer obviously had a great holistic understanding of programming.
- Crux Product Design
the last day. generation part
- Accenture Inc
The topics referring to NLG. The team was able to learn something new in the end with topics that were interesting but it was only in the last day. There were also more hands on activities than slides which was good.
- Accenture Inc
I enjoyed the sentinal analysis/ data science aspect of the course.
Jake Hamilton - Scottish Government
pace and explanations
- Centric IT Solutions Lithuania
The trainer was great! If he would have more time I think we could have learned a lot more.
Zarim Jei Serrano - Cloudstaff Philippines, Inc.
Exercises
Vince Christian Henson - Cloudstaff Philippines, Inc.
It makes the trick. A good introduction (and more) to python.
jean-christophe GOLDBERG - Proximus
* Organization * Trainer's expertise with the subject
- ENGIE- 101 Arch Street
Teaching style and ability of the trainer to overcome unforeseen obstacles and adopt to circumstances. Broad knowledge and experience of the trainer
ASML
Overall good intro to Python. The format of using Jupyter notebook and live examples on the projector was good for following along with the exercises.
ASML
Very good approach to memorize/repeat the key topics. Very nice “warm-up” exercises.
I like that it focuses more on the how-to of the different text summarization methods
Related Courses
Scaling Data Analysis with Python and Dask
14 hoursDask is a flexible and high-performance Python library for parallel computing. It scales and accelerates big data processing with other Python-based data science libraries, such as Pandas, Numpy, and Scikit-Learn. This instructor-led, live
Data Analysis with Python, Pandas, and Numpy
14 hoursPandas is a Python package that provides data structures for working with structured (tabular, multidimensional, potentially heterogeneous) and time series data.
Accelerating Python Pandas Workflows with Modin
14 hoursModin is a parallel data frame system designed to speed up Pandas workflows. It can be used to handle large datasets, leveraging Ray or Dask as the backend framework for distributed computing in Python. This instructor-led, live training (online
Machine Learning with Python and Pandas
14 hoursPandas is a Python library for data manipulation and analysis. Using Pandas, users can perform predictive analysis through machine learning. This instructor-led, live training (online or onsite) is aimed at data scientists who wish to use Pandas
FARM (FastAPI, React, and MongoDB) Full Stack Development
14 hoursFARM (FastAPI, React, and MongoDB) is similar to MERN, but performs faster with Python and FastAPI replacing Node.js and Express as the backend. FastAPI is a high-performance Python web framework used by top companies, such as Microsoft, Uber, and
Developing APIs with Python and FastAPI
14 hoursFastAPI is an open source, high-performance web framework for building APIs with Python. It is used by many large companies, such as Uber, Netflix, and Microsoft. This instructor-led, live training (online or onsite) is aimed at developers who
Web application development with Flask
14 hoursThis practical course is addressed to Python developers that want to create and maintain their first web applications. It is also addressed to people who are already familiar with other web frameworks such as Django or Web2py, and want to learn
Advanced Flask
14 hoursFlask is a micro-framework for developing web applications in Python. Unlike other frameworks, Flask does not have any dependencies on external libraries, making it lightweight and fast. This instructor-led, live training (online or onsite) is
Build REST APIs with Python and Flask
14 hoursFlask is a micro-framework for developing web services in Python. Flask, unlike other frameworks, does not have any dependencies on external libraries, making it lightweight and fast. This instructor-led, live training (online or onsite) is aimed
Kivy: Building Android Apps with Python
7 hoursKivy is an open-source cross-platform graphical user interface library written in Python, which allows multi-touch application development for a wide selection of devices. In this instructor-led, live training participants will learn how to
Game Development with PyGame
7 hoursPyGame is an open source library of Python modules for developing game applications and programs. It is lightweight, easy to use, and compatible with any operating system or platform. This instructor-led, live training (online or onsite) is aimed
GUI Programming with Python and PyQt
21 hoursPyQt is a cross-platform library for developing GUIs (graphical user interfaces) for Python applications. It interfaces Python with the Qt GUI toolkit. This instructor-led, live training (online or onsite) is aimed at persons who wish to program
Scientific Computing with Python SciPy
7 hoursSciPy is an open source Python library for scientific, mathematical, and technical computing. It is built on the NumPy extension, providing a wide range of functionalities for performing complex numerical operations. This instructor-led, live
GUI Programming with Python and Tkinter
14 hoursTkinter is the most commonly used Python GUI (Graphical User Interface) programming toolkit and it is the standard GUI package for Python. Tkinter is an object-oriented layer wrapped over the TK GUI toolkit. This instructor-led, live
Web Development with Web2Py
28 hoursWeb2py is a python based free open source full-stack framework for rapid development of fast, scalable, secure and portable database-driven web-based applications. Audience This course is directed at Engineers and Developers using web2py as a