pyspark caching

Innocent Iguana answered on September 19, 2021 Popularity 5/10 Helpfulness 6/10

answer pyspark caching

related cache pyspark

pyspark caching

Comment

# Caching dataframe
df.persist(storageLevel=pyspark.StorageLevel.MEMORY_AND_DISK) # cache a fataframe
df.cache() # Alternative : cache a fataframe
df.is_cached # Check if the dataframe is cached
df.storageLevel # How the dataframe is cached (useDisk, useMemory, useOffHeap, deserialized, replication)
df.unpersist() # uncache a fataframe
# Caching table
df.createOrReplaceTempView('df') # Register dataframe as table
spark.catalog.cacheTable('df') # Cache the table
spark.catalog.isCached(tableName='df') # Check if the table is cached
spark.catalog.uncacheTable('df') # Uncache table
spark.catalog.clearCache() # Clear all cache

# Visualize cache operations, query plans at >>>>>> localhost:4040

# Logging for inspecting
import logging
logging.basicConfig(stream=sys.stdout, level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s') # DEBUG, 
logging.info("Hello %s", "world") # INFO level print
logging.debug("Hello, take %d", 2) # DEBUG level print
ENABLED = False
t = timer()
logger.info("No action here.")
elapsed1 = t - timer()
if ENABLED: # Managing processor usage leakage
    logger.info("df has %d rows.", df.count())
elapsed2 = t - timer()

# Log columns of text_df as debug message
logging.deug("text_df columns: %s", df1.columns)
# Log whether table1 is cached as info message
logging.info("table1 is cached: %s", spark.catalog.isCached(tableName="table1"))
# Log first row of text_df as warning message
logging.warning("The first row of text_df:\n %s", df1.first())
# Log selected columns of text_df as error message
logging.error("Selected columns: %s", df1.select("id", "word"))
logging.disable(logging.DEBUG) # Turn off logging

# Query plan analysis
df.cache()
df.explain() # Analysis on datafrmae (use bottom-up approach)
spark.sql('EXPLAIN SELECT * FROM df').first() # Analysis using sql

Popularity 5/10 Helpfulness 6/10 Language python

Source: Grepper

Tags: caching pyspark python

Link to this answer
Share Copy Link

Contributed on Feb 23 2024

Innocent Iguana

0 Answers Avg Quality 2/10

Closely Related Answers

cache pyspark

Comment

df_cache = spark.read.csv(["data.csv"],header=True,inferSchema=True)
df.cache().count()

Popularity 9/10 Helpfulness 2/10 Language python

Source: Grepper

Tags: python pyspark pysp

Link to this answer
Share Copy Link

Contributed on Sep 19 2021

Sore Stork

0 Answers Avg Quality 2/10

pyspark caching

Contents

More Related Answers

pyspark caching

Closely Related Answers

cache pyspark

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.