Project Structure That Scales
import os, tempfile, textwrap
from pathlib import Path
DEMO 1 · Scaffold a standard project¶
Almost every good data project shares the same skeleton: raw data in one place, source code in another, notebooks for exploration, outputs kept separate, and a README plus pinned requirements at the top. Create it.
root = Path(tempfile.mkdtemp(prefix="project_"))
for folder in ["data/raw", "data/processed", "src", "notebooks", "outputs"]:
(root / folder).mkdir(parents=True, exist_ok=True)
(root / "README.md").write_text("# Sales Analysis\nRun: pip install -r requirements.txt, then python src/pipeline.py\n")
(root / "requirements.txt").write_text("pandas==2.2.0\nnumpy==1.26.0\n")
(root / ".gitignore").write_text("data/\noutputs/\n.venv/\n__pycache__/\n")
def tree(p, prefix=""):
for item in sorted(p.iterdir()):
print(prefix + "|- " + item.name + ("/" if item.is_dir() else ""))
if item.is_dir(): tree(item, prefix + " ")
tree(root)
|- .gitignore |- README.md |- data/ |- processed/ |- raw/ |- notebooks/ |- outputs/ |- requirements.txt |- src/
DEMO 2 · Separate configuration from code¶
Hard-coded numbers scattered through a script are impossible to reproduce or tweak safely. Put every knob, the seed, paths, thresholds, in one config block at the top. Change behavior by editing config, never by hunting through logic.
config = {
"seed": 42,
"input_file": "data/raw/sales.csv",
"output_file": "outputs/summary.csv",
"min_revenue": 100,
}
for k, v in config.items(): print(f"{k:14} = {v!r}")
print("\nOne place to see, and change, everything that controls the run.")
seed = 42 input_file = 'data/raw/sales.csv' output_file = 'outputs/summary.csv' min_revenue = 100 One place to see, and change, everything that controls the run.
DEMO 3 · Relative paths, never absolute¶
A path like /Users/you/Desktop/data.csv works only on your machine. Anchor paths to the project root instead, so the project runs anywhere. pathlib makes this clean.
PROJECT_ROOT = root # in a real script: Path(__file__).resolve().parent.parent
input_path = PROJECT_ROOT / config["input_file"]
output_path = PROJECT_ROOT / config["output_file"]
print("resolves anywhere the project is cloned:")
print(" input :", input_path.relative_to(PROJECT_ROOT))
print(" output:", output_path.relative_to(PROJECT_ROOT))
print("\nBad (breaks on any other machine): /Users/me/Desktop/sales.csv")
resolves anywhere the project is cloned: input : data/raw/sales.csv output: outputs/summary.csv Bad (breaks on any other machine): /Users/me/Desktop/sales.csv
DEMO 4 · A pipeline script that reruns identically¶
The reproducibility payoff: a script that reads its input, seeds its randomness, computes, and writes its output, deterministically. Run it twice and the outputs are byte-for-byte identical.
import numpy as np, pandas as pd, hashlib
# make a small input file
pd.DataFrame({"revenue": [120, 80, 300, 40, 150, 90]}).to_csv(input_path, index=False)
def pipeline(cfg):
rng = np.random.default_rng(cfg["seed"])
df = pd.read_csv(PROJECT_ROOT / cfg["input_file"])
big = df[df.revenue >= cfg["min_revenue"]].copy()
big["noise"] = rng.normal(0, 1, len(big)).round(4) # seeded, so reproducible
out = PROJECT_ROOT / cfg["output_file"]
big.to_csv(out, index=False)
return hashlib.sha256(out.read_bytes()).hexdigest()[:12]
h_first = pipeline(config)
h_second = pipeline(config)
print("output hash, run 1:", h_first)
print("output hash, run 2:", h_second)
print("identical:", h_first == h_second)
output hash, run 1: 177e1ee3c6e9 output hash, run 2: 177e1ee3c6e9 identical: True
DEMO 5 · The finished, runnable project¶
The whole thing: standard folders, a README, pinned requirements, a .gitignore that keeps data and outputs out of version control, and a deterministic pipeline. This is what 'reproducible' looks like on disk.
tree(root)
print("\n.gitignore keeps large/derived files out of git:")
print((root / ".gitignore").read_text())
|- .gitignore
|- README.md
|- data/
|- processed/
|- raw/
|- sales.csv
|- notebooks/
|- outputs/
|- summary.csv
|- requirements.txt
|- src/
.gitignore keeps large/derived files out of git:
data/
outputs/
.venv/
__pycache__/
Wrap-up¶
A predictable structure (data / src / notebooks / outputs), a single config block, relative paths, and a deterministic pipeline turn a project from personal to shareable. Notice the .gitignore already deciding what version control should and should not track, which is exactly where we go next.