You can create a new table in BigQuery from the results of a query by…
Category: article
BigQuery : How do you optimize query performance in BigQuery
There are several ways to optimize query performance in BigQuery: Partitioning: Partitioning a table by a column can significantly improve…
DBT : How does DBT handle performance optimization and data scalability
DBT does not handle performance optimization and data scalability directly. However, it can be used in conjunction with other tools…
DBT : DBT’s way of handling versioning of data models.
DBT uses a versioning system called “Incremental Modeling” which allows to version data models by maintaining the history of changes…
DBT : DBTs way of handling testing and validation of data models ?
DBT uses a testing framework called “Snapshot Testing” which allows to take snapshots of the data in the database tables…
PySpark : Explain in detail whether Apache Spark SQL lazy or not ?
Yes, Apache Spark SQL is lazy. In Spark, the concept of “laziness” refers to the fact that computations are not…
Python : Program that copies a file from one location to another
This program uses the shutil.copy2() function to copy the file. The shutil.copy2() function is similar to the shutil.copy() function, but…
Python : Program to get all the files with full path, modified after a specific date.
This program uses the os.walk() function to iterate through all files and directories in the specified root directory. For each…
DBT : Learn what is dbt docs. Explain with example
dbt (data build tool) is an open-source command line tool that helps data analysts and engineers write, test, and organize…
PySpark : Generate a sequence number based on a specific order of the DataFrame
You can also use the row_number() function with over() clause to generate a sequence number based on a specific order…
PySpark : Generates a unique and increasing 64-bit integer ID for each row in a DataFrame
pyspark.sql.functions.monotonically_increasing_id A column that produces 64-bit integers with a monotonic increase. The created ID is assured to be both singular…