←Back to the bookChapter 156 · Notebook 2 · Project Structure

Chapter 156 · Tools & Workflow · Notebook 2 of 5

Project Structure That Scales

A pile of notebooks named final_v3_REALLY.ipynb is where reproducibility goes to die. A predictable folder layout, config separated from code, and relative paths make a project something a stranger can pick up and run. We build one from scratch.

import os, tempfile, textwrap
from pathlib import Path

DEMO 1 · Scaffold a standard project

Almost every good data project shares the same skeleton: raw data in one place, source code in another, notebooks for exploration, outputs kept separate, and a README plus pinned requirements at the top. Create it.

root = Path(tempfile.mkdtemp(prefix="project_"))
for folder in ["data/raw", "data/processed", "src", "notebooks", "outputs"]:
    (root / folder).mkdir(parents=True, exist_ok=True)
(root / "README.md").write_text("# Sales Analysis\nRun: pip install -r requirements.txt, then python src/pipeline.py\n")
(root / "requirements.txt").write_text("pandas==2.2.0\nnumpy==1.26.0\n")
(root / ".gitignore").write_text("data/\noutputs/\n.venv/\n__pycache__/\n")

def tree(p, prefix=""):
    for item in sorted(p.iterdir()):
        print(prefix + "|- " + item.name + ("/" if item.is_dir() else ""))
        if item.is_dir(): tree(item, prefix + "   ")
tree(root)

|- .gitignore
|- README.md
|- data/
   |- processed/
   |- raw/
|- notebooks/
|- outputs/
|- requirements.txt
|- src/

DEMO 2 · Separate configuration from code

Hard-coded numbers scattered through a script are impossible to reproduce or tweak safely. Put every knob, the seed, paths, thresholds, in one config block at the top. Change behavior by editing config, never by hunting through logic.

config = {
    "seed": 42,
    "input_file": "data/raw/sales.csv",
    "output_file": "outputs/summary.csv",
    "min_revenue": 100,
}
for k, v in config.items(): print(f"{k:14} = {v!r}")
print("\nOne place to see, and change, everything that controls the run.")

seed           = 42
input_file     = 'data/raw/sales.csv'
output_file    = 'outputs/summary.csv'
min_revenue    = 100

One place to see, and change, everything that controls the run.

DEMO 3 · Relative paths, never absolute

A path like /Users/you/Desktop/data.csv works only on your machine. Anchor paths to the project root instead, so the project runs anywhere. pathlib makes this clean.

PROJECT_ROOT = root                    # in a real script: Path(__file__).resolve().parent.parent
input_path  = PROJECT_ROOT / config["input_file"]
output_path = PROJECT_ROOT / config["output_file"]
print("resolves anywhere the project is cloned:")
print("  input :", input_path.relative_to(PROJECT_ROOT))
print("  output:", output_path.relative_to(PROJECT_ROOT))
print("\nBad (breaks on any other machine):  /Users/me/Desktop/sales.csv")

resolves anywhere the project is cloned:
  input : data/raw/sales.csv
  output: outputs/summary.csv

Bad (breaks on any other machine):  /Users/me/Desktop/sales.csv

DEMO 4 · A pipeline script that reruns identically

The reproducibility payoff: a script that reads its input, seeds its randomness, computes, and writes its output, deterministically. Run it twice and the outputs are byte-for-byte identical.

import numpy as np, pandas as pd, hashlib
# make a small input file
pd.DataFrame({"revenue": [120, 80, 300, 40, 150, 90]}).to_csv(input_path, index=False)

def pipeline(cfg):
    rng = np.random.default_rng(cfg["seed"])
    df = pd.read_csv(PROJECT_ROOT / cfg["input_file"])
    big = df[df.revenue >= cfg["min_revenue"]].copy()
    big["noise"] = rng.normal(0, 1, len(big)).round(4)      # seeded, so reproducible
    out = PROJECT_ROOT / cfg["output_file"]
    big.to_csv(out, index=False)
    return hashlib.sha256(out.read_bytes()).hexdigest()[:12]

h_first  = pipeline(config)
h_second = pipeline(config)
print("output hash, run 1:", h_first)
print("output hash, run 2:", h_second)
print("identical:", h_first == h_second)

output hash, run 1: 177e1ee3c6e9
output hash, run 2: 177e1ee3c6e9
identical: True

DEMO 5 · The finished, runnable project

The whole thing: standard folders, a README, pinned requirements, a .gitignore that keeps data and outputs out of version control, and a deterministic pipeline. This is what 'reproducible' looks like on disk.

tree(root)
print("\n.gitignore keeps large/derived files out of git:")
print((root / ".gitignore").read_text())

|- .gitignore
|- README.md
|- data/
   |- processed/
   |- raw/
      |- sales.csv
|- notebooks/
|- outputs/
   |- summary.csv
|- requirements.txt
|- src/

.gitignore keeps large/derived files out of git:
data/
outputs/
.venv/
__pycache__/

Wrap-up

A predictable structure (data / src / notebooks / outputs), a single config block, relative paths, and a deterministic pipeline turn a project from personal to shareable. Notice the .gitignore already deciding what version control should and should not track, which is exactly where we go next.

←Back to the bookChapter 156 · Notebook 2 · Project Structure