Categories
Mastering Development

spark.sql.files.maxPartitionBytes not limiting max size of written partitions

I’m trying to copy parquet data from another s3 bucket to my s3 bucket. I want to limit the size of each partition to a max of 128 MB. I thought by default spark.sql.files.maxPartitionBytes would have been set to 128 MB, but when I look at the partition files in s3 after my copy I […]