Exports and reproducibility bundles#
Any result, batch of results or data frame can be written in a format other software reads: CSV, XLSX, Parquet, Arrow or JSON. Each export carries the attribution and every required notice of the data it holds (Legal notices), and the manifest that says how the result was produced (Results and manifests). A reproducibility bundle goes one step further: one zip file from which anyone can check a result and rerun it.
import io
import tempfile
from pathlib import Path
import pandas as pd
from pyeconomics import run
from pyeconomics.exports import FORMATS, export, export_bytes
inputs = {"calculation": "irr", "cash_flows": [-100, 30, 40, 50]}
result = run("foundations.time_value", inputs)
assert FORMATS == ("csv", "xlsx", "parquet", "arrow", "json")
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "irr.csv"
export(result, "csv", path)
frame = pd.read_csv(path, comment="#")
assert frame.loc[0, "outputs.irr"] == result.outputs.irr
assert frame.loc[0, "inputs.cash_flows"] == "[-100,30,40,50]"
assert export_bytes(result, "json") == result.to_json().encode()
export(value, format, target) writes to a path or a binary stream, and
export_bytes(value, format) returns the bytes. The value is a Result, a
BatchResult, a Series or Table, what data.get resolved (the frame with
its source records) or identifier Records.
The formats#
Format |
What it holds |
|---|---|
|
A |
|
A |
|
The same columns, with the manifest and attributions in the schema metadata; needs pyarrow |
|
An Arrow IPC file (Feather version 2) with the same metadata; needs pyarrow |
|
Canonical JSON with the manifest embedded: a result’s own canonical bytes |
A result’s columns are those of Result.to_pandas(): inputs.<field>, then
outputs.<field>, then result_sha256, one row per run. A frame’s are date,
then realtime_start and realtime_end when it carries vintages, then value
or the table’s columns. A missing value is an empty cell, a date is its ISO 8601
text and an array its canonical JSON (in Parquet and Arrow it stays a list). A
number is written as its shortest round-trip text, so pandas.read_csv reads it
back equal from pandas 3.1. On pandas 3.0, whose default parser can miss a value’s
last bit, pass float_precision="round_trip"; pandas 3.1 deprecates that argument.
The CSV metadata block#
A CSV export starts with one # label: text line per item: the pyeconomics
version, the model and its version (or the data’s provider and id), the hashes,
then every attribution, notice and terms page of the data:
# pyeconomics: 1.0.0b1
# data: fred:UNRATE
# title: Unemployment Rate
# frame_sha256: 6f1c…
# attribution: Source: Bureau of Labor Statistics, retrieved from FRED®, Federal Reserve Bank of St. Louis.
# notice: This product uses the FRED® API but is not endorsed or certified by the Federal Reserve Bank of St. Louis.
# terms: https://fred.stlouisfed.org/legal/
"date","value"
"2024-01-31",0.037
pandas.read_csv(path, comment="#") skips the block and reads the rest; a line
read on its own is split at the first ": ". The block is part of the data’s
terms: keep it with the data, or show its attributions and notices wherever the
data appears.
Formula escaping#
A spreadsheet runs a cell that starts with =, +, - or @ as a formula, so
a hostile text could run code when a CSV is opened. Every text cell that starts
with one of those characters, a tab or a carriage return is written with a
leading ', in CSV and XLSX alike; numbers are never escaped, so -3 stays a
number while the text "-3" becomes '-3. XLSX stores every string as a string
and never a formula, and every CSV text cell is quoted.
from pyeconomics.exports.layout import escape_text
assert escape_text("=HYPERLINK(1)") == "'=HYPERLINK(1)"
assert escape_text("-0.5") == "'-0.5"
assert escape_text("net debt") == "net debt"
An XLSX cell holds at most 32,767 characters, so a longer text (a long array’s JSON, say) is refused with exit code 3 rather than cut: export it as CSV, Parquet, Arrow or JSON instead. openpyxl writes each sheet to a temporary file of its own, owner-only, which it deletes as soon as the workbook is saved.
Parquet and Arrow#
pyarrow is never a dependency of pyeconomics, not even through an extra; install
it beside pyeconomics (pip install pyarrow) to write Parquet or Arrow. Without
it, both raise MissingOptionalDependencyError naming pyarrow (exit code 8 on
the command line). The schema metadata holds three keys:
Key |
Value |
|---|---|
|
The manifest as canonical JSON (the key |
|
The attributions and notices, a canonical JSON list |
|
The pyeconomics version that wrote the file |
import json
import pyarrow.parquet as pq
data = export_bytes(result, "parquet")
metadata = pq.read_schema(io.BytesIO(data)).metadata
manifest = json.loads(metadata[b"pyeconomics.manifest"])
assert manifest["result_sha256"] == result.manifest.result_sha256
assert json.loads(metadata[b"pyeconomics.attributions"]) == []
Reproducibility bundles#
write_bundle(result, target, request=None) writes a zip of eight members:
Member |
What it holds |
|---|---|
|
The result’s canonical bytes |
|
The validated inputs the model ran on |
|
The bind request ( |
|
The manifest: versions, data records and hashes |
|
The attribution, notice and terms page of each source |
|
|
|
Reruns the model and compares |
|
What the members are, and how to check and rerun them |
The zip is deterministic: every member stored (not compressed), in sorted order, dated 1980-01-01 and with the same permissions, so equal results give equal bytes on every operating system.
import subprocess
import sys
import zipfile
from pyeconomics.exports import check_bundle, write_bundle
with tempfile.TemporaryDirectory() as directory:
bundle = Path(directory) / "irr.zip"
write_bundle(result, bundle)
report = check_bundle(bundle)
with zipfile.ZipFile(bundle) as archive:
archive.extractall(Path(directory) / "irr")
script = Path(directory) / "irr" / "reproduce.py"
done = subprocess.run(
[sys.executable, str(script)], capture_output=True, text=True, check=False
)
assert report.result_sha256 == result.manifest.result_sha256
assert done.returncode == 0
assert "identical to result.json" in done.stdout
reproduce.py uses only the standard library until it imports pyeconomics. It
checks that the installed version is the recorded one (exit code 2, with the
install command, when it is not), reruns pyeconomics.run(model_id, inputs) and
compares the rerun’s result_sha256 with the recorded one: exit code 0 when
they agree, 1 when not. It also says whether the rerun’s bytes equal
result.json, or which parts of the manifest differ: the dependency versions of
another environment, or the data records, which a rerun from the recorded inputs
never reads. A run that read revised data carries the revised_data warning its
bind added; the script adds that recorded warning back before it compares the
hashes, and says so. --out FILE writes the rerun’s canonical bytes.
python reproduce.py --refetch binds the model’s data again from
request.json, at the recorded date, vintage policy and profile, and reports
for each bound field whether the value read now agrees with the recorded input,
labelling revised data. It needs the network and the providers’ keys, as the
first run did.
Warning
reproduce.py is a program. Read it before you run a bundle someone else
made, as you would any script. pyeconomics bundle check verifies a bundle
without running anything from it.
Checking a bundle#
check_bundle(path) (and pyeconomics bundle check FILE) verifies a bundle
while executing nothing from it. Before it reads any member it refuses an
archive larger than 64 MiB, more than 64 members, a member name with a path
separator or .., a member the format does not have or the lack of one it
has, and a compressed or encrypted member. It then reads the four JSON members
only, strictly (no repeated key, no NaN), and checks that each is canonical
JSON, that the result’s result_sha256 and inputs_sha256 are the hashes of
its body and inputs, and that inputs.json and manifest.json are the
result’s own. It never reads reproduce.py. A failure says what is wrong and
exits with code 3.
On the command line#
models run and data get take --format csv, xlsx, parquet or arrow
beside json and table (The command line). A binary format (xlsx, parquet,
arrow) is written to standard output only when it is not a terminal: name a
file with --out, redirect the output or pipe it.
pyeconomics models run foundations.time_value --calculation irr --cash-flows "[-100, 30, 40, 50]" --format csv
pyeconomics models run foundations.time_value --calculation irr --cash-flows "[-100, 30, 40, 50]" --format parquet > irr.parquet
pyeconomics data get fred:UNRATE --format xlsx --out unrate.xlsx
pyeconomics models run macro.recession_probability --as-of 2024-06-28 --bundle recession.zip
pyeconomics bundle check recession.zip
models run ID --batch FILE runs the model once per row of a file: CSV with a
header row of field names (an empty cell leaves that field out), or JSON Lines
with one object per line. The file is read within the 16 MiB input limit and a
batch holds at most 100,000 rows; --format then writes the batch, one row per
run.
pyeconomics models run foundations.time_value --batch rows.csv --format csv --out results.csv
No export or bundle holds a key, a file path, a host name or anything else of the machine that wrote it: the manifest records versions, data records and hashes only.