Hypaxel for Python
Write and read large XLSX files in Python with flat memory: stream rows, records and DataFrames to FastAPI, Django or Flask, and read them back fast.
v0.3.0 unreleasedLast updated View as Markdown
Writing large Excel files in Python is slow with pure-Python libraries, and pandas.to_excel holds a lot of memory while it works. Hypaxel for Python writes .xlsx files as a forward-only stream: rows are converted, serialized and compressed in native code (the Hypaxel Rust core, through PyO3) as you write them, so memory stays flat whether you export 10 rows or 10 million. It accepts the data you already have, from tuples and dicts to dataclasses, pydantic models and DataFrames, and it can stream a download from FastAPI, Django or Flask while your query is still running. It also reads .xlsx files as a stream, into rows, records or pandas and polars DataFrames, in bounded memory.
- Package
- hypaxel
- Requires
- CPython 3.10+
- Engine
- Rust core via PyO3; writer and streaming reader
- Web
- FastAPI / Starlette XlsxResponse, Django, Flask
- Registry
- PyPI (not published yet)
- License
- Apache-2.0
Install#
pip install hypaxel
pip install "hypaxel[pandas]" # extras: pandas, polars, starletteNot published yethypaxel is not on PyPI yet. The command above is what installation will look like once it is.
The package is not published yet. Until then, build it from source: see Building from source. Once it is, optional extras add integrations:
pip install "hypaxel[pandas]" # extras: pandas, polars, starletteQuick start#
import datetime as dt
import hypaxel
from hypaxel import Formula, Styled
with hypaxel.Writer("report.xlsx") as wb:
header = wb.add_style(bold=True, font_color="#FFFFFF", fill="#305496", align="center")
money = wb.add_style(num_format="#,##0.00")
ws = wb.add_worksheet("Sales", freeze_rows=1, column_widths=[20, 12, 14])
ws.write_row(["Product", "Date", "Revenue"], style=header)
ws.write_rows([
("Widget", dt.date(2024, 5, 1), Styled(1234.5, money)),
("Gadget", dt.date(2024, 5, 2), Styled(99.99, money)),
])
ws.write_row(["Total", None, Styled(Formula("SUM(C2:C3)"), money)])One-liners:
hypaxel.to_xlsx(df, "orders.xlsx") # pandas or polars DataFrame
data: bytes = hypaxel.to_xlsx(orders) # list of dicts / dataclasses / models
hypaxel.to_xlsx(df, "out.xlsx", column_formats={"amount": "#,##0.00"})Supported values#
date, datetime (timezone-aware ones are written as local time), time, timedelta, Decimal, numpy scalars, bool and huge ints are written as real Excel types. NaN, None, NaT and pd.NA become empty cells.
Streaming web responses#
FastAPI and Starlette#
from hypaxel.starlette import XlsxResponse
@app.get("/orders.xlsx")
def orders(): # sync generator: runs in Starlette's thread pool
return XlsxResponse(hypaxel.stream_records(load_orders()), filename="orders.xlsx")
@app.get("/live.xlsx")
async def live(): # async source, e.g. an async DB cursor
rows = await conn.stream("SELECT id, customer, amount FROM orders")
return XlsxResponse(hypaxel.astream_rows(rows, header=["Id", "Customer", "Amount"]), filename="live.xlsx")Django#
from django.http import StreamingHttpResponse
def export(request):
response = StreamingHttpResponse(
hypaxel.stream_records(Order.objects.values("id", "customer", "amount").iterator()),
content_type=hypaxel.CONTENT_TYPE,
)
response["Content-Disposition"] = 'attachment; filename="orders.xlsx"'
return responseFlask#
Response(hypaxel.stream_rows(rows, header=[...]), mimetype=hypaxel.CONTENT_TYPE)Streaming generators#
| Generator | Source |
|---|---|
stream_rows(rows, header=...) | iterable of rows |
stream_records(records) | iterable of dicts, dataclasses, models, … |
stream_dataframe(df) | pandas or polars DataFrame |
astream_rows(rows, header=...) | async or sync iterable of rows |
astream_records(records) | async or sync iterable of records |
Custom layouts#
For full control (several sheets, styles, merges), use an in-memory writer and drain():
def export():
wb = hypaxel.Writer() # dest=None → in memory
ws = wb.add_worksheet("Data")
for batch in batches():
ws.write_rows(batch)
if chunk := wb.drain(): # roughly every 64 KiB of compressed output
yield chunk
yield wb.finish()Reading#
import hypaxel
for row in hypaxel.iter_rows("orders.xlsx", "Orders"): # sheet name or 0-based index
print(row) # ['Widget', 3, 9.99, datetime(...)]
records = hypaxel.read_records("orders.xlsx") # [{"Product": "Widget", ...}, ...]
df = hypaxel.read_dataframe("orders.xlsx") # pandas; library="polars" for polars
with hypaxel.open_workbook(upload_bytes) as wb: # path, bytes or binary file object
print(wb.sheet_names, wb.sheets, wb.date_1904)
for row in wb.rows("Orders", skip_empty=True):
...iter_rows and iter_records yield one row at a time; read_rows and read_records return lists. For several sheets, or several iterators over one workbook, open it once with open_workbook, which returns a Reader and works as a context manager.
- Rows are lists:
row[i]is columni + 1, and a row ends at its last non-empty cell. Missing rows come back as[], soenumerate(rows, 1)gives Excel row numbers. Passskip_empty=Trueto drop them. - Records:
iter_recordsandread_recordsuse a header row (header_row=1). Empty header cells becomecolumn_<n>, repeated names get_2,_3and so on, and rows without values are skipped. - DataFrames:
read_dataframe(source, sheet=0, *, header=True, library="pandas"). polars needs one type per column, so a column mixing text and dates arrives as text.
Values you get back#
Values are str, int (integral numbers, as openpyxl returns them), float, bool, datetime.datetime for dates, datetime.time for times of day, datetime.timedelta for [h]:mm durations, error strings such as "#N/A", and None. Formulas yield their cached result. Dates are recognised from the cell’s number format, in both date systems; times are rounded to the millisecond.
Memory, errors and untrusted uploads#
- Memory: rows are streamed. A path is read straight from disk; bytes and file objects are held in memory once. Beyond that, memory is bounded by the workbook’s shared string table and one row.
- Untrusted uploads:
max_shared_strings_bytes=(default 1 GiB) caps the shared string table. - Errors:
FileNotFoundErrorfor missing files andXlsxErrorfor everything else. Messages name the problem, for example a legacy.xlsor encrypted workbook, a sheet that does not exist (with the sheets that do), or a damaged part.
Records and DataFrames#
@dataclass
class Order:
id: int
customer: str
amount: float
placed: dt.date
wb = hypaxel.Writer("orders.xlsx")
hypaxel.write_records(wb, "Orders", orders, column_formats={"amount": "#,##0.00", "placed": "yyyy-mm-dd"})
hypaxel.write_dataframe(wb, "Summary", summary_df, index=True)
wb.finish()The first record defines the columns. write_records accepts dicts, dataclasses, named tuples, pydantic models, objects with __dict__ or __slots__, and tuples. Every table gets a bold, frozen header row, an auto filter, and column widths estimated from the headers.
DataFrames are converted column-wise in chunks, so memory stays bounded even for very large frames.
Tables, row grouping and auto-fit#
ws = wb.add_worksheet("Regions", autofit=True, row_outline_levels=1, outline_summary_above=True)
ws.write_row(["Region", "Revenue"])
ws.write_row(["North", 300])
ws.write_row(["Store A", 120], outline_level=1)
ws.write_row(["Store B", 180], outline_level=1, hidden=True)
ws.add_table(1, 1, ws.row, 2, columns=["Region", "Revenue"], name="Regions", style="TableStyleMedium9")- Auto-fit:
autofit=Trueholds back the sheet’s output untilautofit_sample_rowsrows (default 1,000, at most 8 MiB of XML) have been measured, then writes the widths and streams as usual. Explicitcolumn_widthswin; formulas and merged cells are not measured. - Row grouping:
row_outline_levelsdeclares the deepest grouping level up front, because Excel reads it before the rows.write_rowandbegin_rowacceptoutline_level,hiddenandcollapsed. - Tables:
columnsmust equal the header texts. Table names are unique in the workbook; tables cannot overlap each other, merged cells or anautofilterrange.
API overview#
| Call | Notes |
|---|---|
Writer(dest=None, *, compression_level=1, shared_strings=False, flush_threshold=65536, date_1904=False) | dest: path, binary file object, or None (in memory) |
Writer.add_style(bold=, italic=, underline=, strikethrough=, font_name=, font_size=, font_color=, fill=, border=, border_color=, border_bottom=, align=, valign=, wrap=, indent=, rotation=, num_format=) | returns a Style. Identical styles are shared |
Writer.add_worksheet(name, *, freeze_rows=, freeze_columns=, column_widths=, hidden=, show_grid_lines=, tab_color=, autofit=, autofit_sample_rows=, autofit_max_width=, row_outline_levels=, outline_summary_above=) | returns a Worksheet. The previous sheet is completed |
Writer.drain() / Writer.finish() | in-memory chunks / complete the workbook |
Worksheet.write_row(values, style=None, *, height=, outline_level=, hidden=, collapsed=), write_rows(rows), write_columns(columns), write_dicts(records, keys) | bulk writers, with the loops running in Rust |
Worksheet.write(value), skip_cells(n), end_row(), begin_row(height=, style=, *, outline_level=, hidden=, collapsed=), skip_rows(n) | cell-by-cell control. A begun row is filled by the next write |
Worksheet.merge_cells(r1, c1, r2, c2), autofilter(r1, c1, r2, c2) | ranges (1-based, inclusive) |
Worksheet.add_table(r1, c1, r2, c2, *, columns=, name=, style="TableStyleMedium2", header_row=, autofilter=, row_stripes=, column_stripes=, first_column=, last_column=) | Excel table; columns must equal the header texts |
Formula("SUM(A1:A9)"), Styled(value, style) | special cell values |
open_workbook(source, *, max_shared_strings_bytes=1 GiB) → Reader | source: path, bytes or binary file object; a context manager |
Reader.sheet_names, Reader.sheets, Reader.date_1904, Reader.rows(sheet=0, *, skip_empty=False), Reader.close() | rows iterates lists; several iterators can be open at once |
iter_rows / read_rows, iter_records / read_records, read_dataframe | one-call readers; see Reading |
Options#
Writer options are keyword arguments of hypaxel.Writer:
| Option | Default | Meaning |
|---|---|---|
compression_level | 1 | Deflate level |
shared_strings | False | Use a shared string table instead of inline strings |
flush_threshold | 65536 | Compressed bytes buffered before output is produced |
date_1904 | False | Use the 1904 date system |
Notes and limits#
- Write order: rows are written top to bottom, and one worksheet at a time. Column widths and frozen panes are set in
add_worksheet. - Formulas are stored without cached results; Excel recalculates them when the file opens.
- Long text: text over 32,767 characters raises
XlsxError. The offending cell is left empty and the file stays valid. - GIL: the GIL is held while writing (values are read from Python objects), so use threads for parallel exports of separate files.
- Reading: cached formula results are read, not formula text. Styles (other than date detection), merged ranges, comments and hyperlinks are not read.
Performance#
On a Linux Xeon (2.1 GHz) with Python 3.13, Hypaxel writes 100,000 rows × 10 columns in about 0.1 s, about 50 times faster than XlsxWriter and 95 times faster than openpyxl. DataFrames are about 30 times faster than pandas.to_excel and polars.write_excel, with 10 to 30 times less memory. A million rows take about 0.9 s with write_rows and 2.5 to 3.4 s from a DataFrame, scaling linearly.
- Hypaxel write_rows0.1 s, 1.1×
- Hypaxel stream_rows0.09 s, best
- XlsxWriter (constant_memory)5.15 s, 57×
- openpyxl (write_only)9.48 s, 105×
- Hypaxel pandas DataFrame*0.27 s, 3.0×
- pandas.to_excel (xlsxwriter)*8.68 s, 96×
- Hypaxel polars DataFrame*0.21 s, 2.3×
- polars.write_excel*6.15 s, 68×
| Case | Time | Throughput | Peak Python memory |
|---|---|---|---|
| Hypaxel write_rows | 0.1 s | 1,000,000 rows/s | 5 MB |
| Hypaxel stream_rows | 0.09 s | 1,080,000 rows/s | 11 MB |
| XlsxWriter (constant_memory) | 5.15 s | 19,400 rows/s | 7 MB |
| openpyxl (write_only) | 9.48 s | 10,500 rows/s | 7 MB |
| Hypaxel pandas DataFrame* | 0.27 s | 371,000 rows/s | 16 MB |
| pandas.to_excel (xlsxwriter)* | 8.68 s | 11,500 rows/s | 147 MB |
| Hypaxel polars DataFrame* | 0.21 s | 475,000 rows/s | 8 MB |
| polars.write_excel* | 6.15 s | 16,300 rows/s | 265 MB |
Reading#
On an Apple M1 Max with Python 3.13 and macOS, iter_rows reads 100,000 rows × 10 columns in 0.13 s with a 27 MB peak RSS, about 27 times faster than openpyxl and 3.5 times faster than python-calamine. read_dataframe builds a pandas DataFrame 3.5 to 4 times faster than pandas.read_excel(engine="calamine"), and a polars one faster than fastexcel. At 1M rows (55 MB file), iter_rows takes 1.35 s with a 68 MB peak, against 4.7 s and 862 MB for python-calamine. Every reader runs in its own process, and all of them return the same 1,000,010 non-empty cells.
- Hypaxel iter_rows0.13 s, best
- python-calamine iter_rows0.47 s, 3.6×
- openpyxl (read_only)3.58 s, 28×
- Hypaxel read_dataframe (pandas)0.2 s, 1.5×
- pandas.read_excel (calamine)0.74 s, 5.7×
- pandas.read_excel (openpyxl)4.36 s, 34×
- Hypaxel read_dataframe (polars)0.18 s, 1.4×
- fastexcel to_polars0.24 s, 1.8×
| Case | Time | Peak RSS |
|---|---|---|
| Hypaxel iter_rows | 0.13 s | 27 MB |
| python-calamine iter_rows | 0.47 s | 105 MB |
| openpyxl (read_only) | 3.58 s | 63 MB |
| Hypaxel read_dataframe (pandas) | 0.2 s | 182 MB |
| pandas.read_excel (calamine) | 0.74 s | 315 MB |
| pandas.read_excel (openpyxl) | 4.36 s | 201 MB |
| Hypaxel read_dataframe (polars) | 0.18 s | 143 MB |
| fastexcel to_polars | 0.24 s | 188 MB |
See Performance for methodology and reproduction, and Benchmarks for the full tables.
Building from source#
The source is at github.com/vensas/hypaxel, in the python/ directory. Report bugs at github.com/vensas/hypaxel/issues.
cd python
uv venv && uv pip install maturin pytest pytest-asyncio pandas polars numpy openpyxl xlsxwriter fastapi httpx pydantic
# read benchmark extras: uv pip install python-calamine fastexcel pyarrow
.venv/bin/maturin develop --release
.venv/bin/python -m pytest