Pass system property to spark-submit and read file from classpath or custom path Passing System Properties to Spark Submit and Reading Files Dynamically Spark applications often need to read data from different locations depending on the env 2 min read 06-10-2024 11
get latest schema for partitionned parquet dataframe Dynamically Obtaining the Latest Schema for Partitioned Parquet Data Frames in Spark In the world of big data using Spark and Parquet files for data storage is 2 min read 05-09-2024 7
How to get Total Storage Memory of all Spark executors via Custom Listener Monitoring Spark Executor Memory Usage with Custom Listeners Understanding how your Spark executors are using memory is crucial for performance tuning and resou 3 min read 04-09-2024 15