Cloud Storage

Cloud storage is a service that stores files and data on remote servers managed by a cloud provider, accessible over the internet via API. Major providers include Amazon S3, Google Cloud Storage, and Azure Blob Storage. Files are organized into buckets or containers.

For data pipelines, cloud storage serves as the durable landing zone for raw files, processed datasets, and archived data. API responses can be stored as JSON files in S3, datasets can be delivered to a customer's bucket, and pipeline intermediates can be staged between processing steps.

Cloud storage is infinitely scalable, highly available, and low-cost for data at rest. Most modern data lakehouse architectures use cloud storage - particularly Parquet files on S3 - as the primary data layer, with compute engines like Spark, Athena, or BigQuery querying the files directly without loading them into a database first.