PySpark-Pandas Series.apply()
apply() function, which allows users to apply custom functions to each element of a Series. In this article, we’ll explore the capabilities of Series.apply() through a practical example and delve into its significance in data transformation tasks.
Significance of Series.apply():
- Flexibility:
apply()allows users to define and apply any custom function to each element, offering unparalleled flexibility. - Efficiency: Leveraging vectorized operations in Pandas ensures efficient computation, even for large datasets.
- Readability:
apply()enhances code readability by encapsulating complex transformations into concise functions.
Usage:
- Data Cleaning: Applying custom cleaning functions to standardize data formats or handle missing values.
- Feature Engineering: Creating new features based on existing ones using user-defined transformation logic.
- Statistical Analysis: Applying statistical functions to compute summary statistics or derive new insights from data.
Considerations:
- Vectorized Alternatives: Where possible, prefer Pandas’ built-in vectorized functions for improved performance.
- Performance Optimization: Avoid inefficient operations within custom functions to optimize computation time.
- Type Consistency: Ensure consistency in data types returned by the custom function to prevent unexpected behavior.
Sample code
from pyspark.sql import SparkSessionfrom pyspark.sql.functions import col, udffrom pyspark.sql.types import IntegerType# Create a SparkSessionspark = SparkSession.builder \ .appName("Learning @ Freshers.in : Pandas API on Spark-Series.apply()") \ .getOrCreate()# Sample datadata = [("London", 20), ("New York", 21), ("Helsinki", 12)]columns = ['city', 'numbers']# Create a DataFramedf = spark.createDataFrame(data, schema=columns)# Define a custom functiondef square(x): return x ** 2# Register the custom function as a Spark UDFsquare_udf = udf(square, IntegerType())# Apply the function using the UDFresult_df = df.withColumn('squared_numbers', square_udf(col('numbers')))# Show the resultresult_df.show()Output
+--------+-------+---------------+| city|numbers|squared_numbers|+--------+-------+---------------+| London| 20| 400||New York| 21| 441||Helsinki| 12| 144|+--------+-------+---------------+- We import the necessary modules from PySpark.
- Sample data is defined as a list of tuples.
- A Spark DataFrame is created using
createDataFrame. - A custom function
square()is defined to square each element. - The function is registered as a Spark UDF (User Defined Function) using
udf. - The UDF is applied to the ‘numbers’ column using
withColumn. - Finally, the transformed DataFrame is displayed using
show().
Spark important urls to refer