Guides /
Parquet or CSV for market data? How to choose for NIFTY options files
Why Parquet files are smaller and faster to load than CSV, when CSV still makes sense, and how to read Parquet in pandas, polars, DuckDB and R.
Every pack downloads as Parquet by default, and the order page also offers CSV. Which should you use? For most research work Parquet is the better choice: the files are much smaller, load several times faster and keep their column types. CSV still has its place, mainly in spreadsheets and older tools.
Why Parquet is smaller and faster
- Columnar. Values of one column are stored together, so a repeated value such as the expiry date or the symbol compresses to almost nothing.
- Typed. Prices are stored as numbers and timestamps as timestamps, so nothing has to be parsed from text.
- Selective reads. You can load only the columns, and with filters only the rows, you need.
Reading Parquet in your tool
# pandas
import pandas as pd
df = pd.read_parquet("file.parquet", columns=["datetime", "strike_price", "right", "close"])
# polars
import polars as pl
df = pl.read_parquet("file.parquet")
# DuckDB, straight from disk, many files at once
import duckdb
duckdb.sql("SELECT count(*) FROM 'data/**/*.parquet'").show()# R
library(arrow)
df <- read_parquet("file.parquet")When CSV is the right call
- Spreadsheet work: Excel and Google Sheets open CSV directly.
- Tools that cannot read Parquet, such as some older trading platforms and backtesting apps.
- Quick inspection with command-line tools.
If you use CSV, parse the timestamp column explicitly and keep the time zone in mind: all times in our files are Indian Standard Time, labelled at the start of the bar.