Skip to main content

Ctrl+K

Introduction

Spark overview

Getting Started with Spark
When To Use Spark
Guidance on Spark Sessions
Example Spark Sessions
Configuration Hierarchy and spark-defaults.conf
Data Types in Spark
Creating DataFrames Manually
Reading and Writing Data in Spark
Data Storage

Introduction to PySpark

Introduction to PySpark
Returning Data from Cluster to Driver
Reference columns by name: F.col()

Introduction to sparklyr

Introduction to sparklyr
Using Spark functions in sparklyr

Spark functions

Union two DataFrames with different columns
Add leading zeros with lpad()
Sampling: an overview
Pivot tables in Spark
Rounding differences in Python, R and Spark
Window Functions in Spark
Cross Joins
Arrays Functions in PySpark
Date functions
Writing Data
Set Spark Job Description
Median in Spark

Understanding and Optimising Spark

Ideas for optimising Spark code
Spark Application and UI
Groups not Loops
Shuffling
Optimising Joins
Salted Joins
Managing Partitions
Persisting in Spark
Caching
Checkpoints and Staging Tables
Garbage Collection
Spark DataFrames Are Not Ordered
Exercises

Analysis in Spark

Big data workflow
Interpolation in Spark
Logistic Regression
Spark and Visualisation
Flags in Spark
Data binning
Calculating Cramér’s V from a Spark DataFrame

Testing and Debugging

Understanding and Handling Spark Errors
Unit Testing in Spark

Ancillary Topics

Naming Conflicts in Module Imports
Pydoop: HDFS to pandas
Creating a SQL view in HDFS
Pandas UDFs

Book version 2022.6

Repository
Open issue

.md

Set Spark Job Description

Contents

Further Resources

Set Spark Job Description#

Warning

This section is under construction.

library(sparklyr)

Further Resources#

Spark at the ONS Articles:

Spark Application and UI

Contents

Further Resources

By Analysis Standards and Pipelines in Quality and Improvement, Office for National Statistics