Extracting hour component from timestamps using PySpark

This article focuses on the hour function, offering practical examples and scenarios to highlight its relevance. The hour function in PySpark extracts the hour component from a given timestamp.

Example of extracting the hour component from a series of timestamps:

from pyspark.sql import SparkSession
from pyspark.sql.functions import hour
spark = SparkSession.builder \
    .appName("PySpark Hour Function") \
    .getOrCreate()
data = [("2023-04-21 12:34:56",), ("2023-04-21 00:10:15",), ("2023-04-21 23:59:59",)]
df = spark.createDataFrame(data, ["timestamps"])
df.withColumn("hour_component", hour(df["timestamps"])).show()

Use case: Analyzing web traffic

Imagine a situation where you’re analyzing web traffic to discern the peak hours. The hour function can assist in extracting hours from timestamps, enabling better aggregation and visualization:

web_traffic_data = [
    ("2023-04-21 12:15:30", 100),
    ("2023-04-21 12:45:15", 120),
    ("2023-04-21 13:05:10", 110),
    ("2023-04-21 14:25:45", 95)
]
df_traffic = spark.createDataFrame(web_traffic_data, ["timestamps", "hits"])
# Extracting hour component
df_traffic = df_traffic.withColumn("hour", hour(df_traffic["timestamps"]))
# Aggregating based on hour to get total hits
df_traffic.groupBy("hour").sum("hits").orderBy("hour").show()

Output

+----+---------+
|hour|sum(hits)|
+----+---------+
|  12|      220|
|  13|      110|
|  14|       95|
+----+---------+

From the above data, it’s clear that the website has the highest traffic during the 12 PM hour.

When to use hour?

Temporal analysis: Whether you’re analyzing sales data, website hits, or any time-stamped records, the hour function can segment data on an hourly basis.

Log analysis: For IT admins and system maintainers, extracting the hour from logs can be pivotal for detecting patterns or anomalies.

Scheduling: In scenarios where resource scheduling or planning is involved, the hour function can assist in time-based segmentation.

Spark important urls to refer

1. Spark Examples
2. PySpark Blogs
3. Bigdata Blogs
4. Spark Interview Questions
5. Official Page