Adam Innes · Blog

Holding Structured Data in Python: Dicts, NamedTuple, TypedDict, Dataclasses and Pydantic

· 6 min · python, dataclasses, typing, pydantic

Every project hits the moment where a function stops returning one value and starts returning a small bundle of related things. An account, a parsed config, a row from an API. The first version is always a dict, because a dict takes zero thought. Six months later someone is reading payload["ballance"] in production and wondering why it’s always None.

Python gives you four ways out of that in the standard library, and one very popular third party option. They look similar from a distance and they are genuinely different in what they guarantee. Here’s how I think about picking one.

The plain dict, and how far it actually gets you

A dict is fine for data whose shape you don’t control and don’t want to pin down. JSON you’re passing straight through, a bag of options, anything genuinely dynamic. It’s cheap, it’s flexible, and every Python programmer already knows how it behaves.

The cost is that nothing knows what keys exist. A typo is a KeyError at best and a silent None at worst if you used .get(). Your editor can’t autocomplete, your type checker sees dict[str, Any] and shrugs, and the only documentation of the shape is whatever the code happens to do with it. That’s tolerable for a script and miserable for anything with more than one caller.

NamedTuple when the thing is really a value

If the bundle is a value, meaning two of them with the same contents are the same thing and nobody should be mutating it, typing.NamedTuple is a great fit and people underuse it.

from typing import NamedTuple

class Point(NamedTuple):
    x: float
    y: float
    label: str = ""

p = Point(1.0, 2.0)
p.x          # 1.0
p[0]         # 1.0, it really is a tuple
hash(p)      # works, so it can be a dict key or go in a set
p.x = 5      # AttributeError

It’s a genuine tuple subclass, so it unpacks, indexes, compares and hashes like one, and it’s immutable for free. Defaults are supported and have been since 3.6.1, with the usual rule that defaulted fields come last. When you need a modified copy you use _replace(), which returns a new instance.

The tuple-ness cuts both ways. Code can index into it positionally, which means adding a field in the middle is a breaking change nobody will notice until runtime. And if you ever need mutability you’re rewriting the class. The typing docs cover the details.

TypedDict when it has to stay a dict

Sometimes the data has to remain a dict, because it came from json.loads() or it’s going straight into a library that expects mapping access. PEP 589 added TypedDict in Python 3.8 exactly for that case: you describe which string keys exist and what type each value has, and your type checker enforces it while the runtime carries on using an ordinary dict.

from typing import TypedDict

class Movie(TypedDict):
    title: str
    year: int

m: Movie = {"title": "Arrival", "year": 2016}
type(m)                 # <class 'dict'>
isinstance(m, Movie)    # TypeError: TypedDict does not support instance and class checks

That last line is the thing to internalise. There is no Movie class sitting in memory doing anything. Keys are all required by default, total=False makes them all optional, and since 3.11 you can mark individual keys with Required and NotRequired instead of going all or nothing. A TypedDict is a promise to a type checker and nothing more, which is exactly why it’s so cheap to bolt onto existing dict-shaped code.

Dataclasses when you want an actual class

PEP 557 landed dataclasses in 3.7, and they’re the default answer for most of the cases in between. Eric Smith’s framing in the PEP was “mutable namedtuples with defaults”, and that’s close, except you also keep inheritance, methods, properties and everything else a normal class gives you.

from dataclasses import dataclass, field

@dataclass
class Account:
    id: str
    balance: float = 0.0
    tags: list[str] = field(default_factory=list)

You get __init__, __repr__ and __eq__ generated from the annotations, so Account("a1") == Account("a1") is True and printing one tells you what’s in it. Ordering methods are off by default and come back with order=True.

Note the field(default_factory=list). Writing tags: list[str] = [] raises ValueError at class creation time, because that list would be shared across every instance. Since 3.11 the decorator checks for unhashability rather than specifically looking for list, dict and set, so the protection covers your own mutable types too. The factory is called once per instance, which is what you wanted.

When a field is derived from the others, __post_init__ is the hook. It runs at the end of the generated __init__, after every field is set.

@dataclass
class Order:
    unit_price: float
    qty: int
    total: float = field(init=False)

    def __post_init__(self):
        self.total = round(self.unit_price * self.qty, 2)

Order(2.5, 4)   # Order(unit_price=2.5, qty=4, total=10.0)

One inheritance trap worth knowing: the generated __init__ does not call the base class __init__, so if you’re subclassing a regular class you have to do that yourself from __post_init__.

Mutability, hashability, and why slots matter

The hashing rules for dataclasses look fiddly and are actually quite principled. With the defaults (eq=True, frozen=False) the decorator sets __hash__ to None, so your instances are unhashable. That’s deliberate. The object is mutable and it compares by value, so if you dropped it in a set and then changed a field you’d have corrupted the set. With frozen=True you get __setattr__ and __delattr__ that raise FrozenInstanceError, and because the value can no longer change, the decorator generates a real __hash__ for you. If you turn eq off entirely, __hash__ is left alone and you inherit identity hashing from object.

So the practical rule is that a dataclass you want to use as a dict key or put in a set should be frozen. unsafe_hash=True exists to force the issue and the name is the warning.

The other flag worth reaching for is slots=True, added in 3.10. It regenerates the class with a __slots__ declaration, which means instances stop carrying a per-instance __dict__. The language reference is blunt about why you’d care: the space saved over using __dict__ can be significant, and attribute lookup gets faster too. If you’re holding a few hundred of these it doesn’t matter. If you’re holding a few million rows in memory it’s the difference between fitting and not.

@dataclass(slots=True)
class Point:
    x: float
    y: float

The catch is that you can’t assign attributes that aren’t declared, which is usually a feature. Weak references stop working unless you also pass weakref_slot=True (3.11 and up), and no-argument super() breaks inside a slotted dataclass because the decorator hands you back a new class object. Use the two-argument form there. Full details are in the dataclasses docs.

None of this validates anything

Here’s the part that catches people, and it’s stated plainly in both the PEP and the docs: with two narrow exceptions for ClassVar and InitVar, nothing in @dataclass examines the type in the annotation. The same goes for TypedDict and NamedTuple. Annotations are for type checkers, linters and editors. The interpreter does not look at them.

Which means this runs perfectly happily:

Account(id=42, balance="lots")
# Account(id=42, balance='lots', tags=[])

Your type checker will flag it if the call is in code it can see. If the values came out of json.loads(), a form post, a queue message or a third party API, the checker never saw them and has nothing to say. Those four standard library tools describe shape. They do not enforce it.

Pydantic, when the data comes from outside

For that boundary case the usual answer is Pydantic, which is third party and not in the standard library, so it’s a dependency decision rather than a free one. A model is a class inheriting from BaseModel with annotated attributes, and it validates on construction.

from pydantic import BaseModel, ValidationError

class Account(BaseModel):
    id: str
    balance: float = 0.0

Account(id="a1", balance="12.50")   # balance becomes 12.5, a float
Account.model_validate(payload)     # validate a dict you got from somewhere

Bad input raises ValidationError, and one exception carries every problem it found rather than one at a time, which makes it usable for API error responses. Note the coercion in that first line: by default Pydantic will cast input to make it fit the declared type, and the docs are upfront that this can lose information. If you don’t want that, strict mode turns coercion off, and model_config also gives you frozen=True if you want immutable models.

My rule of thumb is boring. Inside your program, where you control both ends, use the standard library: NamedTuple for immutable values, a dataclass for most things, TypedDict when it must stay a dict, and add frozen and slots when you need hashability or memory. At the edges, where bytes arrive from somewhere you don’t control, parse them once into a validated model and pass the validated object inward. Mixing those up is how you end up debugging a string that was supposed to be a float three call frames away from where it entered.

← all posts