Parquet — Columnar Table
A small employee table as Apache Parquet — the columnar format at the heart of modern data lakes and analytics. For testing Parquet readers (pandas, Spark, DuckDB) and conversion.
Files for data engineering
Example and test files tagged for data engineering.
ORC — Columnar Table
The same employee table as Apache ORC — the columnar format common in the Hive/Hadoop ecosystem. For testing ORC readers and Parquet↔ORC conversion.
Feather — Arrow IPC Table
The same table as Feather (Arrow IPC file) — the zero-copy on-disk form of an Apache Arrow table. For testing Arrow readers and fast columnar interchange.
Avro — Row Binary + Schema
The same records as Apache Avro — a compact row-based binary format that embeds its own schema, widely used in Kafka pipelines. For testing Avro decoders and schema evolution.
HDF5 — Hierarchical Scientific Data
A small HDF5 file with a compound 'employees' dataset and a numeric 'readings' grid — the hierarchical format used across science and ML. For testing h5py/HDF5 readers and conversion.
E-commerce Products (CSV, 200 rows)
A realistic e-commerce product catalogue (200 rows) — part of a relational dataset (products, customers, orders) with CSV, JSON, SQL, and Parquet twins for testing joins, imports, and conversion.
E-commerce Products (Parquet, 200 rows)
The e-commerce products table as Apache Parquet — the columnar twin, for testing analytics engines (pandas, DuckDB, Spark).
E-commerce Customers (CSV, 500 rows)
A realistic e-commerce customer directory (500 rows) — part of a relational dataset (products, customers, orders) with CSV, JSON, SQL, and Parquet twins for testing joins, imports, and conversion.
E-commerce Orders (CSV, 2000 rows)
A realistic e-commerce order lines (customer_id → customers, product_id → products) (2000 rows) — part of a relational dataset (products, customers, orders) with CSV, JSON, SQL, and Parquet twins for testing joins, imports, and conversion.
E-commerce Orders (Parquet, 2000 rows)
The e-commerce orders table as Apache Parquet — the columnar twin, for testing analytics engines (pandas, DuckDB, Spark).