File Handling & Serialization

1. Reading and writing text files

Variables live in memory and disappear when the program ends. Files are how data survives. Working with a file always follows three steps: open it, read or write, then close it.

Opening a file

open(path, mode, encoding=...) returns a file object. The file object keeps track of a position in the file, which moves forward as you read or write. The mode says what you intend to do:

Writing

with open("notes.txt", "w", encoding="utf-8") as f:
    print(f.write("tokens\n"))
    f.write("embeddings\n")

with open("notes.txt", "a", encoding="utf-8") as f:
    f.write("attention\n")

print(open("notes.txt", encoding="utf-8").read())

7
tokens
embeddings
attention

• write() returns the number of characters written (7 here, counting the newline).
• It writes exactly what you give it. Unlike print() it adds no newline, so put \n in yourself.
• writelines(lines) writes a list (or any iterable) of strings in one call, again without adding newlines.

Reading

open("notes.txt", "w", encoding="utf-8").write("tokens\nembeddings\nattention\n")

with open("notes.txt", encoding="utf-8") as f:
    for n, line in enumerate(f, start=1):
        print(n, repr(line))
    print("at the end:", repr(f.read()))

1 'tokens\n'
2 'embeddings\n'
3 'attention\n'
at the end: ''

Each line keeps its trailing \n; use line.rstrip("\n") to drop it. At the end of the file read() returns an empty string rather than raising an error, which is how loops know to stop.

Memory matters for big files. On a 50 MB test file of 100-character lines, read() held all 50,000,000 characters at once, while for line in f held one 100-character line at a time.

Closing, and why with is used

Writes don't go straight to disk. They collect in a buffer of 8,192 bytes and are written out when the buffer fills or the file is closed:

import os
f = open("buf.txt", "w", encoding="utf-8")
f.write("Hello world")
print("before close:", os.path.getsize("buf.txt"), "bytes")
f.close()
print("after close: ", os.path.getsize("buf.txt"), "bytes")

before close: 0 bytes
after close:  11 bytes

So a file that's never closed can lose its last writes. A with block closes the file automatically when the block ends, even if an exception happens inside it, which is why it's the standard way to open files.

Encoding

Text files are stored as bytes, and the encoding is the rule for converting between characters and bytes. Without encoding=, Python uses the operating system's default: UTF-8 on this Mac, but often cp1252 on Windows. A file containing "₹" or Hindi text can then fail to open on another machine. Always pass encoding="utf-8".

Key points

• Use with open(...) as f: so files are always closed.
• "w" erases an existing file as soon as it opens; "a" adds to it; "x" refuses to overwrite.
• Always specify encoding="utf-8" for text files.
• Loop over the file object for large files instead of calling read().

Common mistakes

• Opening an existing file with "w" by mistake. The contents are gone before your first write.
• Forgetting \n when writing lines, so everything ends up on one line.
• Writing a number directly: f.write(5) raises TypeError. Convert with str() or an f-string.

Resources

• CampusX Session 10, 14:50–26:55 — writing, "w" vs "a"
• CampusX Session 10, 33:30–39:30 — read, read(n), readline
• CampusX Session 10, 45:25–48:00 — the with statement
• CampusX Session 10, 50:10–61:30 — reading a large file in chunks
• Python tutorial: Reading and writing files

Practice

exercises/python/07/01_text_files.py — It writes two lines, appends a third and reads them back.

1 'tokens\n'
2 'embeddings\n'
3 'attention\n'
read() at the end gives ''

Try: change the second open from "a" to "w". Result: only attention is left; "w" emptied the file first.

2. CSV and JSON

Serialization means converting a Python object into a format that can be saved to a file or sent over a network. Deserialization is the reverse. A plain text file can only hold strings, so structured data needs a format. CSV and JSON are the two most common.

CSV

CSV (comma-separated values) stores a table: one row per line, cells separated by commas. It's what spreadsheets and databases export. Use the csv module rather than splitting on commas yourself, because it handles quoting (a comma inside a cell) correctly.

import csv

rows = [{"name": "Asha", "score": 92}, {"name": "Ravi", "score": 78}]

with open("scores.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["name", "score"])
    writer.writeheader()
    writer.writerows(rows)

print(open("scores.csv", encoding="utf-8").read())

with open("scores.csv", newline="", encoding="utf-8") as f:
    for row in csv.DictReader(f):
        print(row)

name,score
Asha,92
Ravi,78

{'name': 'Asha', 'score': '92'}
{'name': 'Ravi', 'score': '78'}

• Every value comes back as a string: the score is '92', not 92. Convert it yourself with int(row["score"]). (pandas' read_csv guesses types for you.)
• Open CSV files with newline="". The csv module writes its own line endings, and without this Windows shows a blank line between rows.
• csv.writer / csv.reader work with lists instead of dicts.

JSON

JSON (JavaScript Object Notation) stores nested data: objects (like dicts) and arrays (like lists), to any depth. Almost every web API uses it, and every language can read it.

import json

data = {"name": "Asha", "score": 92, "tags": ("python", "sql"), "active": True, "mentor": None}

with open("asha.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2)

print(open("asha.json", encoding="utf-8").read())

with open("asha.json", encoding="utf-8") as f:
    back = json.load(f)
print(back)

{
  "name": "Asha",
  "score": 92,
  "tags": [
    "python",
    "sql"
  ],
  "active": true,
  "mentor": null
}
{'name': 'Asha', 'score': 92, 'tags': ['python', 'sql'], 'active': True, 'mentor': None}

The s in dumps/loads stands for string. Useful arguments: indent=2 for readable output, ensure_ascii=False to keep non-English text as-is instead of \u escapes.

What survives a JSON round trip

import json, datetime
print(json.loads(json.dumps({1: "a", 2: "b"})))
print(json.dumps({"joined": datetime.date(2026, 9, 21)}, default=str))
json.dumps({"skills": {"python", "sql"}})

{'1': 'a', '2': 'b'}
{"joined": "2026-09-21"}
TypeError: Object of type set is not JSON serializable

default= is called for any object JSON can't handle. default=str is enough for dates; for your own classes, pass a function that returns a dict.

CSV or JSON?

Common mistakes

• Doing arithmetic on CSV values without converting them: "92" * 2 is "9292".
• Looking up a JSON dict with an int key after loading it: the key is now a string.
• Expecting a tuple or set back from JSON.

Resources

• CampusX Session 10, 76:08–82:30 — why text files can't hold lists and dicts
• CampusX Session 10, 86:00–96:30 — json.dump, json.load, tuples, nested dicts
• CampusX Session 10, 96:25–104:30 — saving your own objects with default=
• CampusX Session 27, 24:07–56:30 — CSV with pandas (Session 10 doesn't cover CSV)
• CampusX Session 27, 64:16–70:49 — JSON with pandas (read_json)
• Python docs: csv.DictReader
• Python docs: JSON ↔ Python conversion table

Practice

exercises/python/07/02_csv_json.py — It saves the same records as CSV and JSON and prints the type of each value that comes back.

CSV : '92' "('python', 'sql')"
JSON: 92 ['python', 'sql']

Try: make Asha's tags a set, {"python", "sql"}. Result: TypeError: Object of type set is not JSON serializable.

3. Pickle

pickle is Python's own serialization format. It saves almost any Python object, including instances of your own classes, sets and tuples, and restores them exactly. The result is bytes, so the file must be opened in binary mode ("wb" / "rb").

import pickle

class Student:
    def __init__(self, name, skills):
        self.name, self.skills = name, skills

data = {"student": Student("Asha", {"python", "sql"}), "scores": (92, 78)}

with open("data.pkl", "wb") as f:
    pickle.dump(data, f)

with open("data.pkl", "rb") as f:
    back = pickle.load(f)

print(type(back["student"]).__name__, back["student"].name, sorted(back["student"].skills), back["scores"])
print(open("data.pkl", "rb").read()[:12])

Student Asha ['python', 'sql'] (92, 78)
b'\x80\x04\x95g\x00\x00\x00\x00\x00\x00\x00}'

The object, the set and the tuple all came back unchanged. Like JSON, there's dumps/loads for working with bytes in memory instead of files.

Pros and cons

The class must exist when loading

A pickle stores an object's data and the name of its class, not the class's code. Loading it in a program where Student isn't defined fails:

AttributeError: Can't get attribute 'Student' on <module '__main__' (built-in)>

Renaming or moving the class breaks old pickle files in the same way, which is why pickle is a poor choice for long-term storage.

Security

Unpickling can execute code: an object can tell pickle to call any function while it's being loaded. This harmless example calls print; a malicious file could run a system command instead.

import pickle

class Surprise:
    def __reduce__(self):
        return (print, ("this line ran inside pickle.loads()",))

pickle.loads(pickle.dumps(Surprise()))

this line ran inside pickle.loads()

Never unpickle data from a source you don't trust: downloads, uploads, email attachments. The same applies to joblib.load, which uses pickle underneath. This is why model weights are increasingly shared in the safetensors format.

When to use what

• Pickle: your own short-lived caches or models, used only by Python, from a source you trust.
• JSON: anything shared, stored long-term, or read by other tools.

Resources

• CampusX Session 10, 106:19–109:30 — pickle.dump and the "wb" requirement
• CampusX Session 10, 109:30–112:30 — loading fails without the class, works with it
• CampusX Session 10, 112:52–113:54 — pickle vs JSON
• Python docs: pickle (see the warning at the top)

Practice

exercises/python/07/03_pickle.py — It pickles an object with a set inside, loads it back, and shows code running during a load.

Student ['python', 'sql'] (92, 78)
first bytes: b'\x80\x04\x95g\x00\x00\x00\x00\x00\x00\x00}'
this line ran inside pickle.loads()

Try: add import json and json.dumps(data) at the end. Result: TypeError: Object of type Student is not JSON serializable.

4. Working with directories (os, pathlib)

A path is a file's address. An absolute path starts at the top of the disk (/Users/darshan/data/a.csv). A relative path (data/a.csv) is resolved from the current working directory, the folder Python was started from, which is not necessarily the folder the script is in.

Python has two ways to work with paths: the older os / os.path functions, which use plain strings, and pathlib, which represents paths as objects. New code should use pathlib.

pathlib basics

from pathlib import Path

p = Path("project") / "data" / "scores.csv"      # / joins parts, on any OS
print(p)
print(p.name, p.stem, p.suffix, p.parent)
print(p.exists())

p.parent.mkdir(parents=True, exist_ok=True)       # create all missing folders
p.write_text("name,score\nAsha,92\n", encoding="utf-8")
print(p.exists(), p.read_text(encoding="utf-8").splitlines())

project/data/scores.csv
scores.csv scores .csv project/data
False
True ['name,score', 'Asha,92']

Listing and finding files

from pathlib import Path

data = Path("data")
(data / "raw").mkdir(parents=True)
for name in ["b.csv", "a.csv", "notes.txt", "raw/c.csv"]:
    (data / name).write_text("x", encoding="utf-8")

print(sorted(p.name for p in data.iterdir()))       # everything in the folder
print(sorted(p.name for p in data.glob("*.csv")))   # CSVs in this folder
print(sorted(p.name for p in data.glob("**/*.csv")))  # CSVs here and below

['a.csv', 'b.csv', 'notes.txt', 'raw']
['a.csv', 'b.csv']
['a.csv', 'b.csv', 'c.csv']

iterdir() and glob() return files in whatever order the operating system gives, so wrap them in sorted() when order matters. The practice script below shows it: on a Mac, glob("*.csv") returned ['a.csv', 'b.csv'], while the same code run in this page's browser runner returned ['b.csv', 'a.csv'].

Common operations: pathlib vs os

Relative paths and the working directory

A script that does open("data/scores.csv") works when run from its own folder and raises FileNotFoundError when run from anywhere else, because the path is resolved from the working directory. To make a path relative to the script itself, build it from the script's location:

BASE = Path(__file__).parent
scores = BASE / "data" / "scores.csv"

Also note that open() does not create missing folders: writing to no_such_folder/x.txt raises FileNotFoundError until the folder exists.

Key points

• Prefer pathlib; know the os equivalents for reading older code.
• Relative paths depend on where Python was started. Anchor them with Path(__file__).parent.
• Use mkdir(parents=True, exist_ok=True) before writing into a new folder.
• Sort directory listings if the order matters.

Resources

• CampusX Session 10, 32:00–33:00 — saving a file into another folder (done through the Colab file panel)
• Python docs: pathlib basic use
• Python docs: pathlib ↔ os equivalents
• Python docs: os, files and directories — the os functions in full

Practice

exercises/python/07/04_directories.py — It builds a small folder tree, lists and renames files, and shows the working-directory effect.

glob     : ['a.csv', 'b.csv']
sorted   : ['a.csv', 'b.csv']
parts    : a .csv data
after    : ['a.csv', 'b.csv', 'notes.md', 'raw']
relative : False

Try: change glob("*.csv") to glob("**/*.csv"). Result: c.csv from the raw subfolder is found too.

5. Logging to files

print() output disappears when the terminal closes and can't be filtered by importance. The logging module writes messages with a severity level, a source and a format, and can send them to a file, so there's a record to check after a script has run unattended.

Levels

The level set in the configuration is a threshold: messages below it are dropped.

Logging to a file

import logging

logging.basicConfig(
    filename="app.log",
    level=logging.INFO,
    format="%(asctime)s %(levelname)-8s %(name)s: %(message)s",
    encoding="utf-8",
)
log = logging.getLogger("etl")

log.debug("row 17 raw=%r", {"score": "92"})    # below INFO: dropped
log.info("loaded %d rows", 250)
log.warning("3 rows had no score")
try:
    1 / 0
except ZeroDivisionError:
    log.exception("average failed")            # ERROR level + traceback

print(open("app.log", encoding="utf-8").read())

2026-09-21 19:58:58,335 INFO     etl: loaded 250 rows
2026-09-21 19:58:58,335 WARNING  etl: 3 rows had no score
2026-09-21 19:58:58,335 ERROR    etl: average failed
Traceback (most recent call last):
  File "<string>", line 15, in <module>
ZeroDivisionError: division by zero

• basicConfig(filename=...) sends log lines to a file instead of the screen. The file is opened in append mode, so it grows across runs.
• log.exception() inside an except block logs at ERROR level and includes the full traceback.
• Pass values as arguments (log.info("loaded %d rows", n)) rather than building an f-string, so the text is only formatted if the message is actually written.
• Common format fields: %(asctime)s time, %(levelname)s level, %(name)s logger name, %(message)s the message.

Loggers per module

In a larger program, each module gets its own logger with log = logging.getLogger(__name__), so every line shows which module wrote it. Configuration (basicConfig) is done once, in the program's entry point.

Screen and file together

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s: %(message)s",
    handlers=[logging.FileHandler("app.log", encoding="utf-8"), logging.StreamHandler()],
)

Common mistakes

• Calling log.info() with no configuration. The default level is WARNING, so INFO messages go nowhere.
• Calling basicConfig twice. Only the first call takes effect; a second one is silently ignored unless you pass force=True. This also happens when an imported library configured logging first.
• Logging secrets such as API keys or passwords. Log files get copied into bug reports and chats.

Resources

• Python docs: Logging HOWTO, logging to a file
• Python docs: when to use logging — which level for which situation

CampusX Session 10 doesn't cover logging.

Practice

exercises/python/07/05_logging.py — It logs at four levels to app.log and prints the file.

INFO     etl: loaded 250 rows
WARNING  etl: 3 rows had no score
ERROR    etl: average failed
Traceback (most recent call last):
  File "…/exercises/python/07/05_logging.py", line 18, in <module>
    1 / 0
    ~~^~~
ZeroDivisionError: division by zero

Try: change level=logging.INFO to level=logging.DEBUG. Result: the DEBUG line appears as well.

Summary

Review questions

1. What happens to an existing file when it's opened with "w", and when exactly does it happen?
2. Why can data be lost if a file isn't closed?
3. A value read from a CSV file is "92". What is row["score"] * 2?
4. What do a tuple, a set and a dict with integer keys become after a JSON round trip?
5. Give two reasons not to use pickle for data you'll share with others.
6. A script opens "data/scores.csv". It works from the project folder but not from its parent. Why, and how do you fix it?
7. Your log.info() messages appear nowhere. Name two possible causes.