Lateral Join is a powerful feature in Snowflake that allows you to join a table…
Category: article
Snowflake : Retrieve the SQL script used to create a specific database, schema, table, view, materialized view
In Snowflake, you can use the “GET_DDL” function to retrieve the SQL script used to create a specific database, schema,…
PySpark : Understanding the ‘take’ Action in PySpark with Examples. [Retrieves a specified number of elements from the beginning of an RDD or DataFrame]
In this article, we will focus on the ‘take’ action, which is commonly used in PySpark operations. We’ll provide a…
AWS Glue : Handling Errors and Retries in AWS Glue
AWS Glue is a fully managed ETL service that simplifies and automates data processing tasks. While AWS Glue is designed…
Redshift : Redshift Unload getting the file as filename appended with 000 – how to get as actual filename.
Unfortunately, Redshift’s UNLOAD command appends a part number (like 000) to the output file names by default, and there’s no…
DBT : Restarting job from failure step in DBT on DBT cloud
To restart a job from a failure step in DBT on DBT Cloud, you can follow these steps: Go to…
DBT : Best Practices for Restartable dbt Jobs: Tips for Resilient Data Pipelines
To ensure restartability in dbt jobs, you can use a combination of incremental models, snapshots, and custom materializations. Additionally, it’s…
DBT : Organizing DBT Models in Subdirectories: A Guide to YAML Configuration
DBT (Data Build Tool) is an essential tool for data engineers and analysts to build, test, and document data pipelines…
DBT : Converting S3 Paths with DBT Macros Based on Environment Variables
In data engineering, it is common to work with cloud-based storage systems such as Amazon S3. Often, the location of…
DBT : Demystifying the DBT Model: A Comprehensive Guide
Data Build Tool (DBT) has become an indispensable tool for data engineers and analysts in modern data environments. It enables…
Python : ZIP file from an S3 bucket, split it into smaller ZIP files, and save those directly to the S3 bucket
To read a large ZIP file from an S3 bucket, split it into smaller ZIP files, and save those directly…