PySpark : How to number up to the nearest integer

PySpark @


In PySpark, the ceil() function is used to round a number up to the nearest integer. This function is a part of the pyspark.sql.functions module, and it can be used on both column and numeric expressions.

Here is an example of using the ceil() function in PySpark:

from pyspark.sql import SparkSession
from pyspark.sql.functions import ceil

# Create a SparkSession
spark = SparkSession.builder.appName("Ceil Example").getOrCreate()

# Create a DataFrame with some sample data
data = [(1.2,), (2.5,), (3.7,), (4.9,)]
df = spark.createDataFrame(data, ["num"])

# Use the ceil() function to round the numbers up
df =["num"]).alias("rounded_num"))

# Show the result
This code creates a SparkSession and a DataFrame with a single column “num” containing some sample decimal numbers. Then it uses the ceil() function to round these numbers up to the nearest integer and create a new column “rounded_num” with the result. The DataFrame is then displayed and show the rounded number.

The output of this code will be:

|          2|
|          3|
|          4|
|          5|

The Ceil function rounds up the decimal number to nearest integer.

Author: user

Leave a Reply