PySpark : Generating a 64-bit hash value in PySpark

PySpark @

Introduction to 64-bit Hashing

A hash function is a function that can be used to map data of arbitrary size to fixed-size values. The values returned by a hash function are called hash codes, hash values, or simply hashes.

When we say a hash value is a “signed 64-bit” value, it means the hash function outputs a 64-bit integer that can represent both positive and negative numbers. In computing, a 64-bit integer can represent a vast range of numbers, from -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807.

A 64-bit hash function can be useful in a variety of scenarios, particularly when working with large data sets. It can be used for quickly comparing complex data structures, indexing data, and checking data integrity.

Use of 64-bit Hashing in PySpark

While PySpark does not provide a direct function for 64-bit hashing, it does provide a function hash() that returns a hash as an integer, which is usually a 32-bit hash. For a 64-bit hash, we can consider using the murmur3 hash function from Python’s mmh3 library, which produces a 128-bit hash and can be trimmed down to 64-bit. You can install the library using pip:

pip install mmh3

Here is an example of how to generate a 64-bit hash value in PySpark:

from pyspark.sql import SparkSession
from pyspark.sql.functions import udf
from pyspark.sql.types import LongType
import mmh3

#Create a Spark session
spark = SparkSession.builder.appName(" Learning for 64-bit Hashing in PySpark ").getOrCreate()

#Creating sample data
data = [("Sachin",), ("Ramesh",), ("Babu",)]
df = spark.createDataFrame(data, ["Name"])

#Function to generate 64-bit hash
def hash_64(input):
    return mmh3.hash64(input.encode('utf-8'))[0]

#Create a UDF for the 64-bit hash function
hash_64_udf = udf(lambda z: hash_64(z), LongType())

#Apply the UDF to the DataFrame
df_hashed = df.withColumn("Name_hashed", hash_64_udf(df['Name']))

#Show the DataFrame

In this example, we create a Spark session and a DataFrame df with a single column “Name”. Then, we define the function hash_64 to generate a 64-bit hash of an input string. After that, we create a user-defined function (UDF) hash_64_udf using PySpark SQL functions. Finally, we apply this UDF to the column “Name” in the DataFrame df and create a new DataFrame df_hashed with the 64-bit hashed values of the names.

Advantages and Drawbacks of 64-bit Hashing


  1. Large Range: A 64-bit hash value has a very large range of possible values, which can help reduce hash collisions (different inputs producing the same hash output).
  2. Fast Comparison and Lookup: Hashing can turn time-consuming operations such as string comparison into a simple integer comparison, which can significantly speed up certain operations like data lookups.
  3. Data Integrity Checks: Hash values can provide a quick way to check if data has been altered.


  1. Collisions: While the possibility is reduced, hash collisions can still occur where different inputs produce the same hash output.
  2. Not for Security: A hash value is not meant for security purposes. It can be reverse-engineered to get the original input.
  3. Data Loss: Hashing is a one-way function. Once data is hashed, it cannot be converted back to the original input.

Spark important urls to refer

  1. Spark Examples
  2. PySpark Blogs
  3. Bigdata Blogs
  4. Spark Interview Questions
  5. Official Page
Author: user

Leave a Reply