Skip to content
Docs: Hypaxel for PythonMenu

Hypaxel for Python

Write and read large XLSX files in Python with flat memory: stream rows, records and DataFrames to FastAPI, Django or Flask, and read them back fast.

v0.3.0 unreleasedLast updated View as Markdown

Writing large Excel files in Python is slow with pure-Python libraries, and pandas.to_excel holds a lot of memory while it works. Hypaxel for Python writes .xlsx files as a forward-only stream: rows are converted, serialized and compressed in native code (the Hypaxel Rust core, through PyO3) as you write them, so memory stays flat whether you export 10 rows or 10 million. It accepts the data you already have, from tuples and dicts to dataclasses, pydantic models and DataFrames, and it can stream a download from FastAPI, Django or Flask while your query is still running. It also reads .xlsx files as a stream, into rows, records or pandas and polars DataFrames, in bounded memory.

Compatibilityv0.3.0 unreleased
Package
hypaxel
Requires
CPython 3.10+
Engine
Rust core via PyO3; writer and streaming reader
Web
FastAPI / Starlette XlsxResponse, Django, Flask
Registry
PyPI (not published yet)
License
Apache-2.0

Install#

Install · PyPI
pip install hypaxel
pip install "hypaxel[pandas]"   # extras: pandas, polars, starlette

Not published yethypaxel is not on PyPI yet. The command above is what installation will look like once it is.

The package is not published yet. Until then, build it from source: see Building from source. Once it is, optional extras add integrations:

pip install "hypaxel[pandas]"       # extras: pandas, polars, starlette

Quick start#

import datetime as dt
import hypaxel
from hypaxel import Formula, Styled

with hypaxel.Writer("report.xlsx") as wb:
    header = wb.add_style(bold=True, font_color="#FFFFFF", fill="#305496", align="center")
    money = wb.add_style(num_format="#,##0.00")

    ws = wb.add_worksheet("Sales", freeze_rows=1, column_widths=[20, 12, 14])
    ws.write_row(["Product", "Date", "Revenue"], style=header)
    ws.write_rows([
        ("Widget", dt.date(2024, 5, 1), Styled(1234.5, money)),
        ("Gadget", dt.date(2024, 5, 2), Styled(99.99, money)),
    ])
    ws.write_row(["Total", None, Styled(Formula("SUM(C2:C3)"), money)])

One-liners:

hypaxel.to_xlsx(df, "orders.xlsx")                       # pandas or polars DataFrame
data: bytes = hypaxel.to_xlsx(orders)                    # list of dicts / dataclasses / models
hypaxel.to_xlsx(df, "out.xlsx", column_formats={"amount": "#,##0.00"})

Supported values#

date, datetime (timezone-aware ones are written as local time), time, timedelta, Decimal, numpy scalars, bool and huge ints are written as real Excel types. NaN, None, NaT and pd.NA become empty cells.

Streaming web responses#

FastAPI and Starlette#

from hypaxel.starlette import XlsxResponse

@app.get("/orders.xlsx")
def orders():                                  # sync generator: runs in Starlette's thread pool
    return XlsxResponse(hypaxel.stream_records(load_orders()), filename="orders.xlsx")

@app.get("/live.xlsx")
async def live():                              # async source, e.g. an async DB cursor
    rows = await conn.stream("SELECT id, customer, amount FROM orders")
    return XlsxResponse(hypaxel.astream_rows(rows, header=["Id", "Customer", "Amount"]), filename="live.xlsx")

Django#

from django.http import StreamingHttpResponse

def export(request):
    response = StreamingHttpResponse(
        hypaxel.stream_records(Order.objects.values("id", "customer", "amount").iterator()),
        content_type=hypaxel.CONTENT_TYPE,
    )
    response["Content-Disposition"] = 'attachment; filename="orders.xlsx"'
    return response

Flask#

Response(hypaxel.stream_rows(rows, header=[...]), mimetype=hypaxel.CONTENT_TYPE)

Streaming generators#

GeneratorSource
stream_rows(rows, header=...)iterable of rows
stream_records(records)iterable of dicts, dataclasses, models, …
stream_dataframe(df)pandas or polars DataFrame
astream_rows(rows, header=...)async or sync iterable of rows
astream_records(records)async or sync iterable of records

Custom layouts#

For full control (several sheets, styles, merges), use an in-memory writer and drain():

def export():
    wb = hypaxel.Writer()                      # dest=None → in memory
    ws = wb.add_worksheet("Data")
    for batch in batches():
        ws.write_rows(batch)
        if chunk := wb.drain():                # roughly every 64 KiB of compressed output
            yield chunk
    yield wb.finish()

Reading#

import hypaxel

for row in hypaxel.iter_rows("orders.xlsx", "Orders"):      # sheet name or 0-based index
    print(row)                                               # ['Widget', 3, 9.99, datetime(...)]

records = hypaxel.read_records("orders.xlsx")                # [{"Product": "Widget", ...}, ...]
df = hypaxel.read_dataframe("orders.xlsx")                   # pandas; library="polars" for polars

with hypaxel.open_workbook(upload_bytes) as wb:             # path, bytes or binary file object
    print(wb.sheet_names, wb.sheets, wb.date_1904)
    for row in wb.rows("Orders", skip_empty=True):
        ...

iter_rows and iter_records yield one row at a time; read_rows and read_records return lists. For several sheets, or several iterators over one workbook, open it once with open_workbook, which returns a Reader and works as a context manager.

  • Rows are lists: row[i] is column i + 1, and a row ends at its last non-empty cell. Missing rows come back as [], so enumerate(rows, 1) gives Excel row numbers. Pass skip_empty=True to drop them.
  • Records: iter_records and read_records use a header row (header_row=1). Empty header cells become column_<n>, repeated names get _2, _3 and so on, and rows without values are skipped.
  • DataFrames: read_dataframe(source, sheet=0, *, header=True, library="pandas"). polars needs one type per column, so a column mixing text and dates arrives as text.

Values you get back#

Values are str, int (integral numbers, as openpyxl returns them), float, bool, datetime.datetime for dates, datetime.time for times of day, datetime.timedelta for [h]:mm durations, error strings such as "#N/A", and None. Formulas yield their cached result. Dates are recognised from the cell’s number format, in both date systems; times are rounded to the millisecond.

Memory, errors and untrusted uploads#

  • Memory: rows are streamed. A path is read straight from disk; bytes and file objects are held in memory once. Beyond that, memory is bounded by the workbook’s shared string table and one row.
  • Untrusted uploads: max_shared_strings_bytes= (default 1 GiB) caps the shared string table.
  • Errors: FileNotFoundError for missing files and XlsxError for everything else. Messages name the problem, for example a legacy .xls or encrypted workbook, a sheet that does not exist (with the sheets that do), or a damaged part.

Records and DataFrames#

@dataclass
class Order:
    id: int
    customer: str
    amount: float
    placed: dt.date

wb = hypaxel.Writer("orders.xlsx")
hypaxel.write_records(wb, "Orders", orders, column_formats={"amount": "#,##0.00", "placed": "yyyy-mm-dd"})
hypaxel.write_dataframe(wb, "Summary", summary_df, index=True)
wb.finish()

The first record defines the columns. write_records accepts dicts, dataclasses, named tuples, pydantic models, objects with __dict__ or __slots__, and tuples. Every table gets a bold, frozen header row, an auto filter, and column widths estimated from the headers.

DataFrames are converted column-wise in chunks, so memory stays bounded even for very large frames.

Tables, row grouping and auto-fit#

ws = wb.add_worksheet("Regions", autofit=True, row_outline_levels=1, outline_summary_above=True)
ws.write_row(["Region", "Revenue"])
ws.write_row(["North", 300])
ws.write_row(["Store A", 120], outline_level=1)
ws.write_row(["Store B", 180], outline_level=1, hidden=True)
ws.add_table(1, 1, ws.row, 2, columns=["Region", "Revenue"], name="Regions", style="TableStyleMedium9")
  • Auto-fit: autofit=True holds back the sheet’s output until autofit_sample_rows rows (default 1,000, at most 8 MiB of XML) have been measured, then writes the widths and streams as usual. Explicit column_widths win; formulas and merged cells are not measured.
  • Row grouping: row_outline_levels declares the deepest grouping level up front, because Excel reads it before the rows. write_row and begin_row accept outline_level, hidden and collapsed.
  • Tables: columns must equal the header texts. Table names are unique in the workbook; tables cannot overlap each other, merged cells or an autofilter range.

API overview#

CallNotes
Writer(dest=None, *, compression_level=1, shared_strings=False, flush_threshold=65536, date_1904=False)dest: path, binary file object, or None (in memory)
Writer.add_style(bold=, italic=, underline=, strikethrough=, font_name=, font_size=, font_color=, fill=, border=, border_color=, border_bottom=, align=, valign=, wrap=, indent=, rotation=, num_format=)returns a Style. Identical styles are shared
Writer.add_worksheet(name, *, freeze_rows=, freeze_columns=, column_widths=, hidden=, show_grid_lines=, tab_color=, autofit=, autofit_sample_rows=, autofit_max_width=, row_outline_levels=, outline_summary_above=)returns a Worksheet. The previous sheet is completed
Writer.drain() / Writer.finish()in-memory chunks / complete the workbook
Worksheet.write_row(values, style=None, *, height=, outline_level=, hidden=, collapsed=), write_rows(rows), write_columns(columns), write_dicts(records, keys)bulk writers, with the loops running in Rust
Worksheet.write(value), skip_cells(n), end_row(), begin_row(height=, style=, *, outline_level=, hidden=, collapsed=), skip_rows(n)cell-by-cell control. A begun row is filled by the next write
Worksheet.merge_cells(r1, c1, r2, c2), autofilter(r1, c1, r2, c2)ranges (1-based, inclusive)
Worksheet.add_table(r1, c1, r2, c2, *, columns=, name=, style="TableStyleMedium2", header_row=, autofilter=, row_stripes=, column_stripes=, first_column=, last_column=)Excel table; columns must equal the header texts
Formula("SUM(A1:A9)"), Styled(value, style)special cell values
open_workbook(source, *, max_shared_strings_bytes=1 GiB) → Readersource: path, bytes or binary file object; a context manager
Reader.sheet_names, Reader.sheets, Reader.date_1904, Reader.rows(sheet=0, *, skip_empty=False), Reader.close()rows iterates lists; several iterators can be open at once
iter_rows / read_rows, iter_records / read_records, read_dataframeone-call readers; see Reading

Options#

Writer options are keyword arguments of hypaxel.Writer:

OptionDefaultMeaning
compression_level1Deflate level
shared_stringsFalseUse a shared string table instead of inline strings
flush_threshold65536Compressed bytes buffered before output is produced
date_1904FalseUse the 1904 date system

Notes and limits#

  • Write order: rows are written top to bottom, and one worksheet at a time. Column widths and frozen panes are set in add_worksheet.
  • Formulas are stored without cached results; Excel recalculates them when the file opens.
  • Long text: text over 32,767 characters raises XlsxError. The offending cell is left empty and the file stays valid.
  • GIL: the GIL is held while writing (values are read from Python objects), so use threads for parallel exports of separate files.
  • Reading: cached formula results are read, not formula text. Styles (other than date detection), merged ranges, comments and hyperlinks are not read.

Performance#

On a Linux Xeon (2.1 GHz) with Python 3.13, Hypaxel writes 100,000 rows × 10 columns in about 0.1 s, about 50 times faster than XlsxWriter and 95 times faster than openpyxl. DataFrames are about 30 times faster than pandas.to_excel and polars.write_excel, with 10 to 30 times less memory. A million rows take about 0.9 s with write_rows and 2.5 to 3.4 s from a DataFrame, scaling linearly.

Python: 100,000 rowsXeon 2.1 GHz, Linux; Python 3.13. Compared with XlsxWriter (constant_memory), openpyxl (write_only), pandas, polars.
Timelower is better
  1. Hypaxel write_rows0.1 s, 1.1×
  2. Hypaxel stream_rows0.09 s, best
  3. XlsxWriter (constant_memory)5.15 s, 57×
  4. openpyxl (write_only)9.48 s, 105×
  5. Hypaxel pandas DataFrame*0.27 s, 3.0×
  6. pandas.to_excel (xlsxwriter)*8.68 s, 96×
  7. Hypaxel polars DataFrame*0.21 s, 2.3×
  8. polars.write_excel*6.15 s, 68×
Python: 100,000 rows
CaseTimeThroughputPeak Python memory
Hypaxel write_rows0.1 s1,000,000 rows/s5 MB
Hypaxel stream_rows0.09 s1,080,000 rows/s11 MB
XlsxWriter (constant_memory)5.15 s19,400 rows/s7 MB
openpyxl (write_only)9.48 s10,500 rows/s7 MB
Hypaxel pandas DataFrame*0.27 s371,000 rows/s16 MB
pandas.to_excel (xlsxwriter)*8.68 s11,500 rows/s147 MB
Hypaxel polars DataFrame*0.21 s475,000 rows/s8 MB
polars.write_excel*6.15 s16,300 rows/s265 MB

Methodology and notes · Benchmark source

Reading#

On an Apple M1 Max with Python 3.13 and macOS, iter_rows reads 100,000 rows × 10 columns in 0.13 s with a 27 MB peak RSS, about 27 times faster than openpyxl and 3.5 times faster than python-calamine. read_dataframe builds a pandas DataFrame 3.5 to 4 times faster than pandas.read_excel(engine="calamine"), and a polars one faster than fastexcel. At 1M rows (55 MB file), iter_rows takes 1.35 s with a 68 MB peak, against 4.7 s and 862 MB for python-calamine. Every reader runs in its own process, and all of them return the same 1,000,010 non-empty cells.

Python reading: 100,000 rowsApple M1 Max, macOS; Python 3.13. Compared with openpyxl 3.1.5, python-calamine 0.9.1, fastexcel 0.21.0, pandas 3.1.1, polars 2.1.0.
Timelower is better
  1. Hypaxel iter_rows0.13 s, best
  2. python-calamine iter_rows0.47 s, 3.6×
  3. openpyxl (read_only)3.58 s, 28×
  4. Hypaxel read_dataframe (pandas)0.2 s, 1.5×
  5. pandas.read_excel (calamine)0.74 s, 5.7×
  6. pandas.read_excel (openpyxl)4.36 s, 34×
  7. Hypaxel read_dataframe (polars)0.18 s, 1.4×
  8. fastexcel to_polars0.24 s, 1.8×
Python reading: 100,000 rows
CaseTimePeak RSS
Hypaxel iter_rows0.13 s27 MB
python-calamine iter_rows0.47 s105 MB
openpyxl (read_only)3.58 s63 MB
Hypaxel read_dataframe (pandas)0.2 s182 MB
pandas.read_excel (calamine)0.74 s315 MB
pandas.read_excel (openpyxl)4.36 s201 MB
Hypaxel read_dataframe (polars)0.18 s143 MB
fastexcel to_polars0.24 s188 MB

Methodology and notes · Benchmark source

See Performance for methodology and reproduction, and Benchmarks for the full tables.

Building from source#

The source is at github.com/vensas/hypaxel, in the python/ directory. Report bugs at github.com/vensas/hypaxel/issues.

cd python
uv venv && uv pip install maturin pytest pytest-asyncio pandas polars numpy openpyxl xlsxwriter fastapi httpx pydantic
# read benchmark extras: uv pip install python-calamine fastexcel pyarrow
.venv/bin/maturin develop --release
.venv/bin/python -m pytest