How to find the date of the first occurrence of a specified weekday after a given date.

PySpark @ Freshers.in

PySpark, the Python API for Apache Spark, offers a plethora of functions for handling big data efficiently. One such function is next_day, a tool essential for date and time manipulation. In this article, we’ll delve into the intricacies of the next_day function, showcasing its utility through practical examples.┬áThe next_day function in PySpark is a powerful tool for manipulating dates and times. By understanding its application through examples, data professionals can leverage this functionality to efficiently handle date-related queries in their datasets.

Understanding next_day

The next_day function in PySpark is used to find the date of the first occurrence of a specified weekday after a given date. It takes two arguments:

  1. A column containing date values.
  2. A string specifying the weekday.

The function returns a new column with dates corresponding to the next occurrence of the specified weekday.

Syntax

from pyspark.sql.functions import next_day
new_df = df.withColumn("next_specified_day", next_day(df["date_column"], "weekday"))

Practical example

To illustrate the usage of next_day, let’s consider a dataset with employee names and their respective joining dates. We aim to find the next Monday after their joining date.

Sample data

Assume we have the following data in a DataFrame named employee_df:

Name JoiningDate
Sachin 2023-03-10
Manju 2023-03-11
Ram 2023-03-12
Raju 2023-03-13
David 2023-03-14
Wilson 2023-03-15
Code Implementation
from pyspark.sql import SparkSession
from pyspark.sql.functions import next_day
from pyspark.sql.types import *
# Initialize Spark Session
spark = SparkSession.builder.appName("NextDayExample").getOrCreate()
# Sample data
data = [("Sachin", "2023-03-10"),
        ("Manju", "2023-03-11"),
        ("Ram", "2023-03-12"),
        ("Raju", "2023-03-13"),
        ("David", "2023-03-14"),
        ("Wilson", "2023-03-15")]
# Define schema
schema = StructType([
    StructField("Name", StringType(), True),
    StructField("JoiningDate", StringType(), True)
])
# Create DataFrame
employee_df = spark.createDataFrame(data, schema)
employee_df = employee_df.withColumn("JoiningDate", employee_df["JoiningDate"].cast(DateType()))
# Use next_day function
employee_df_with_next_monday = employee_df.withColumn("NextMonday", next_day(employee_df["JoiningDate"], "Monday"))
# Show results
employee_df_with_next_monday.show()

Output

The output will display the original data along with a new column, NextMonday, showing the date of the next Monday after each employee’s joining date.

+------+-----------+----------+
|  Name|JoiningDate|NextMonday|
+------+-----------+----------+
|Sachin| 2023-03-10|2023-03-13|
| Manju| 2023-03-11|2023-03-13|
|   Ram| 2023-03-12|2023-03-13|
|  Raju| 2023-03-13|2023-03-20|
| David| 2023-03-14|2023-03-20|
|Wilson| 2023-03-15|2023-03-20|
+------+-----------+----------+
Author: user