Apache Spark 2：Data Processing and Real-Time Analytics

Romeo Kienzler Md. Rezaul Karim Sridhar Alla Siamak Amirghodsi Meenakshi Rajendran Broderick Hall Shuen Mei

更新时间：2021-07-02 15:15:26

coverpage

Title Page

Apache Spark 2: Data Processing and Real-Time Analytics

About Packt

Why Subscribe?

Packt.com

Contributors

About the Authors

Packt Is Searching for Authors Like You

Preface

Who This Book Is For

What This Book Covers

To Get the Most out of This Book

Download the Example Code Files

Conventions Used

Get in Touch

Reviews

A First Taste and What's New in Apache Spark V2

Spark machine learning

Spark Streaming

Spark SQL

Spark graph processing

Extended ecosystem

What's new in Apache Spark V2?

Cluster design

Cluster management

Local

Standalone

Apache YARN

Apache Mesos

Cloud-based deployments

Performance

The cluster structure

Hadoop Distributed File System

Data locality

Memory

Coding

Cloud

Summary

Apache Spark Streaming

Overview

Errors and recovery

Checkpointing

Streaming sources

TCP stream

File streams

Flume

Kafka

Summary

Structured Streaming

The concept of continuous applications

True unification - same code same engine

Windowing

How streaming engines use windowing

How Apache Spark improves windowing

Increased performance with good old friends

How transparent fault tolerance and exactly-once delivery guarantee is achieved

Replayable sources can replay streams from a given offset

Idempotent sinks prevent data duplication

State versioning guarantees consistent results after reruns

Example - connection to a MQTT message broker

Controlling continuous applications