tstore.archive.ts.writers package#
Submodules#
tstore.archive.ts.writers.pyarrow module#
Created on Mon Apr 8 17:26:02 2024.
@author: ghiggi
- tstore.archive.ts.writers.pyarrow.estimate_row_group_size(df, size='200MB')[source]#
Estimate row_group_size parameter based on the desired row group memory size.
row_group_size is a Parquet argument controlling the number of rows in each Apache Parquet File Row Group.
- tstore.archive.ts.writers.pyarrow.write_partitioned_dataset(base_dir, table, partitioning=None, row_group_size='400MB', max_file_size='2GB', compression='snappy', compression_level=None, max_open_files=0, use_threads=True)[source]#
Write partitioned Parquet Dataset.
https://arrow.apache.org/docs/python/generated/pyarrow.dataset.write_dataset.html
Module contents#
Created on Mon Jun 12 22:23:06 2023.
@author: ghiggi