Data Compression

Data compression encodes data using fewer bits than the original representation, reducing storage space and transmission bandwidth. Compression is either lossless (the original can be perfectly reconstructed - required for structured data) or lossy (some precision is sacrificed for greater reduction - acceptable for images or audio).

In data pipelines, compression dramatically reduces storage costs and transfer times for large datasets. Parquet applies columnar compression automatically; CSV files are commonly compressed with gzip or zstandard. A 100MB CSV file might compress to 10-20MB depending on data characteristics and field repetition.

When working with APIs, HTTP compression (gzip or Brotli) reduces response size transparently - most HTTP clients decompress automatically. For datasets, always choose a compressed format like gzip-compressed CSV or Parquet for anything larger than a few megabytes to reduce storage costs and delivery time.