Posted in spark

Binary Operator Functions in Pandas API on Spark – 2

   March 28, 2024
Spark_Pandas_Freshers_in

The fusion of Spark’s distributed computing prowess with the intuitive functionalities of Pandas unleashes unparalleled capabilities for handling massive datasets efficiently. One of the key features that empowers this synergy is the support for binary operator functions within the Pandas API on Spark. These functions, including Series.rmul(), Series.rsub(), Series.rtruediv(), Series.sub(), and Series.truediv(), enable users to perform element-wise operations seamlessly across distributed data. In this article, we’ll delve into each of these functions, explore their applications, and demonstrate their usage with illustrative examples.

1. Series.rmul(other) in Spark

The Series.rmul() function calculates the reverse multiplication of two series element-wise. It multiplies each element of the second series by the corresponding element of the first series, yielding a new series with the result. This function is particularly useful when you need to compute the product of two datasets in a distributed manner.

# Example of Series.rmul()
import pandas as pd
from pyspark.sql import SparkSession
# Create a SparkSession
spark = SparkSession.builder.appName("Learning @ Freshers.in Pandas API on Spark").getOrCreate()
# Sample data
data1 = {'A': [1, 2, 3, 4, 5]}
data2 = {'A': [10, 20, 30, 40, 50]}
df1 = spark.createDataFrame(pd.DataFrame(data1))
df2 = spark.createDataFrame(pd.DataFrame(data2))
# Convert DataFrames to Pandas Series
series1 = df1.select('A').toPandas()['A']
series2 = df2.select('A').toPandas()['A']
# Perform reverse multiplication
result = series1.rmul(series2)
# Print the result
print("Result of reverse multiplication:")
print(result)
Python

Output:

Result of reverse multiplication:
0     10
1     40
2     90
3    160
4    250
Name: A, dtype: int64
Bash

2. Series.rsub(other) in Spark

The Series.rsub() function computes the reverse subtraction of two series element-wise. It subtracts each element of the second series from the corresponding element of the first series, generating a new series with the result. This function is valuable for scenarios where you need to determine the difference between two datasets.

# Example of Series.rsub()
# Assume the series1 and series2 are defined from the previous example
# Perform reverse subtraction
result = series1.rsub(series2)
# Print the result
print("Result of reverse subtraction:")
print(result)
Python

Output:

Result of reverse subtraction:
0     9
1    18
2    27
3    36
4    45
Name: A, dtype: int64
Python

3. Series.rtruediv(other)

The Series.rtruediv() function calculates the reverse floating-point division of two series element-wise. It divides each element of the second series by the corresponding element of the first series, yielding a new series with the result. This function is beneficial for performing division operations with a distributed dataset.

# Example of Series.rtruediv()
# Assume the series1 and series2 are defined from the previous example
# Perform reverse division
result = series1.rtruediv(series2)
# Print the result
print("Result of reverse division:")
print(result)
Python

Output:

Result of reverse division:
0    10.000000
1    10.000000
2    10.000000
3    10.000000
4    10.000000
Name: A, dtype: float64
Bash

4. Series.sub(other)

The Series.sub() function computes the subtraction of two series element-wise. It subtracts each element of the second series from the corresponding element of the first series, generating a new series with the result. This function is useful for calculating the difference between datasets.

# Example of Series.sub()
# Assume the series1 and series2 are defined from the previous example
# Perform subtraction
result = series1.sub(series2)
# Print the result
print("Result of subtraction:")
print(result)
Python

Output:

Result of subtraction:
0    -9
1   -18
2   -27
3   -36
4   -45
Name: A, dtype: int64
Bash

5. Series.truediv(other)

The Series.truediv() function computes the floating-point division of two series element-wise. It divides each element of the first series by the corresponding element of the second series, yielding a new series with the result. This function is essential for performing division operations across distributed datasets.

# Example of Series.truediv()
# Assume the series1 and series2 are defined from the previous example
# Perform division
result = series1.truediv(series2)
# Print the result
print("Result of division:")
print(result)
Python

Output:

Result of division:
0    0.1
1    0.1
2    0.1
3    0.1
4    0.1
Name: A, dtype: float64
Bash
Post Views: 2
Author: user
Website