MongoDB and the Art of Not Shooting Yourself in the Foot

Or: How I Learned to Stop Worrying and Actually Read the Documentation

There’s a special kind of developer. You’ve met them. They discover MongoDB, read the words “flexible schema” on the landing page, feel a warm rush of freedom in their chest, and immediately start designing a system that will haunt their team for the next three years.

Document databases are powerful. They are also consistently, creatively, and impressively misused. This article is about both sides of that coin: what MongoDB can do when you respect it, and what it will do to you when you don’t.

We’ll cover the architecture decisions that age well versus the ones that age like milk, indexing (the thing everyone gets wrong), connection pooling (the thing everyone ignores until production melts), schema philosophy, migrations, and finally the eternal question of Atlas vs self-managed vs Amazon’s “definitely-not-MongoDB” offering.

Buckle up.

Part 1: How People Actually Use Document Databases (And Why It Hurts)

The most common MongoDB architecture I’ve encountered in the wild looks like this: one collection. Everything in it. Users, orders, events, logs, audit trails, config values, and whatever else someone decided to persist on a Friday afternoon without thinking too hard about it.

This is not a flexible schema. This is a skip bin with an API.

The root cause is almost always the same: no upfront design. Someone reads “schemaless” and hears “no design needed.” They don’t. The flexibility MongoDB gives you is the freedom to choose your schema, not the permission to skip it. A relational database forces you to think at the start. MongoDB lets you defer that thinking indefinitely — right up until the moment you can’t query your own data without a PhD in aggregation pipelines.

Think of it like the Ghostbusters rule: just because you can cross the streams doesn’t mean you should. Dumping everything into one collection because it’s technically allowed is crossing the streams. You’ll know it was a mistake when it’s too late.

The second plague is indexes. Or rather, the complete absence of them. I’ve seen production systems doing full collection scans on tens of millions of documents because nobody ever bothered to run explain(). It works fine in development, of course — it always does. Three thousand documents load fast no matter what you do. Then you hit production, the collection grows, and suddenly a simple query takes four seconds and your on-call rotation becomes a recurring nightmare.

The worst part? When you tell people about this, some of them already knew. They were warned. They just didn’t act. Ignorance becomes negligence, and negligence becomes an incident at 2 AM.

The Auditing Trap

Auditing is a special category of misuse. The scenario plays out predictably: someone needs an audit trail, they already have MongoDB, they create a collection called audit_logs, and they start writing to it. So far, so reasonable.

Then no one sets a TTL index. The collection grows. Forever. Months pass. The disk fills. A frantic DBA (or an overworked senior engineer wearing a DBA hat that doesn’t quite fit) discovers that audit_logs now accounts for 80% of storage and contains events from eighteen months ago that nobody has ever queried.

The other variant is querying audit data in ways MongoDB wasn’t designed for — range scans across unbounded time windows, ad-hoc joins to the main data collection via application-layer loops, and reports that turn a simple “what changed last week” question into a five-minute full collection scan.

Part 2: What MongoDB Is Actually Good At (And You’re Probably Ignoring)

Now that we’ve established what not to do, let’s talk about what document databases do exceptionally well when you design around their strengths.

Time-Series Collections

MongoDB’s native time-series collections are underrated. If you’re storing metrics, events, sensor data, or any time-ordered sequence of measurements, they offer automatic bucketing, efficient compression, and query performance that would make your relational DBA uncomfortable.

The pattern is simple: declare your collection as a time-series type, specify your timeField and optionally a metaField for partitioning (think: sensor_id, service_name, user_id), and MongoDB handles the storage optimization for you. Queries against time ranges become dramatically cheaper because the data is physically organized around time.

If you’re currently storing event data in a regular collection with a created_at field and wondering why your time-range queries are slow, this is why.

Auditing Done Right: Change Streams

The right way to do auditing in MongoDB isn’t a collection you write to manually. It’s change streams.

Change streams give you a real-time feed of every insert, update, replace, and delete happening in your collection — including the full document delta. You capture what actually changed, not what your application thought should be logged. Your application code stays clean. Your audit trail is complete. And you can process the stream asynchronously, write it to a separate audit collection with a TTL index, and never worry about the main collection getting polluted with historical metadata.

This is the MongoDB equivalent of database triggers, except it works in a distributed system and you don’t have to cry in the shower afterwards.

Flexible Schema for Complex Real-World Data

Here’s where document databases shine and relational databases start crying: data that doesn’t fit neatly into rows.

Product catalogs where a t-shirt has size and color attributes, a laptop has RAM and CPU specs, and a book has ISBN and author — and adding a new product type shouldn’t require an ALTER TABLE. Multi-tenant configuration where each tenant has different fields, different feature flags, and a schema that evolves independently. Complex nested hierarchies where the alternative would be six JOINs and an ORM that’s slowly losing its mind.

The key insight is that the flexible schema isn’t for avoiding structure — it’s for accommodating structure that varies legitimately. If your data varies because of bad design, that’s a different problem and MongoDB won’t save you from it.

Part 3: Indexes — The Chapter People Skip (And Then Regret)

Let’s talk about the thing that separates a MongoDB deployment that hums from one that slowly kills your SLAs.

The Sins I’ve Had to Clean Up

Two patterns come up over and over. First: indexing high-cardinality fields that nobody queries. Someone adds db.collection.createIndex({ id: 1 }) on a UUID field, feels productive, and moves on. The index costs memory and write overhead on every insert. Nobody ever queries by that field directly — they query by user_id + status + created_at. The index is dead weight.

Second, and more insidious: indexing embedded array fields without understanding multikey behavior. When you index an array field, MongoDB creates a separate index entry for each element in the array. If your documents have arrays with hundreds of elements, your index size explodes. Write performance degrades. Index builds take forever. And nobody connects the dots until they look at the index stats.

It’s like that scene in Indiana Jones where he swaps the gold idol for a bag of sand and thinks he got away with it — right before the entire temple starts collapsing. The index looked fine. Production disagrees.

The Underrated Hero: Compound Indexes Designed Around Query Patterns

The most impactful indexing decision you can make isn’t picking the right index type — it’s designing compound indexes around your actual query patterns.

The ESR rule is your friend: Equality fields first, then Sort fields, then Range fields. If your most common query filters by tenant_id (equality), sorts by created_at (sort), and filters by status in [...] (range), your compound index should be { tenant_id: 1, created_at: 1, status: 1 } — in that order. Getting the order wrong doesn’t just make the index less efficient. It can make MongoDB ignore the index entirely.

Spend thirty minutes mapping your top ten query patterns before you create a single index. That thirty minutes will save you many future hours of incident investigation.

My Actual Diagnostic Workflow

When a query is slow in production, here’s the sequence I follow — every time, in this order:

Step 1: explain('executionStats') — non-negotiable. Look for COLLSCAN in the winning plan. If you see it, you don’t have an appropriate index for this query. Look at totalDocsExamined vs nReturned. If you’re examining 500,000 documents to return 12, you have a problem.

Step 2: Atlas Performance Advisor — if you’re on Atlas, use it. It analyzes your query patterns and recommends indexes with estimated impact. It’s not magic but it’s a very good starting point.

Step 3: Slow query logs — set slowOpThresholdMs to something reasonable (100ms is a good starting point) and watch for COLLSCAN entries. The logs will tell you exactly what queries are hurting and how often.

One important note on index creation: since MongoDB 4.2, all index builds are non-blocking by default. The old { background: true } option is silently ignored — you don’t need it and shouldn’t reference it. MongoDB’s WiredTiger engine handles the locking internally, taking an exclusive lock only at the very beginning and end of the build. The rest of the time, reads and writes proceed normally.

Part 4: SDKs, Connection Pooling, and the Mistakes That Will Find You

This section is dedicated to the code that runs fine for six months and then causes a 3 AM alert the week before a deadline.

The Classic Mistakes, In Order of How Often I’ve Seen Them

Creating a new MongoClient per request. This is the MongoDB equivalent of opening a new database connection for every HTTP request. Except MongoDB’s connection establishment is heavyweight enough that you’ll feel it at any reasonable load. Your MongoClient is a singleton. Create it once when your application starts. Share it everywhere. If your framework is making this hard, fight the framework.

This is the coding equivalent of the guy in Planes, Trains and Automobiles who books a new rental car for every mile of the journey. Technically functional. Catastrophically inefficient.

maxPoolSize too high. The default maxPoolSize in PyMongo is 100 connections. For a single application instance, this is often fine. For twenty application instances all connecting to the same MongoDB cluster, you’ve just configured 2,000 connections. MongoDB isn’t free with connections — each one consumes memory on the server. Set maxPoolSize deliberately based on your actual concurrency requirements and the number of instances you’re running. A number you calculated beats a default you inherited.

No timeouts. The default behavior of most MongoDB drivers when the server is unreachable or overloaded is to wait. And wait. And wait. Set connectTimeoutMS (how long to wait when establishing a connection) and serverSelectionTimeoutMS (how long the driver waits to find an available server before raising an error). If you don’t, you’ll discover the hard way that hanging threads are worse than fast failures.

A Minimal Sane PyMongo Configuration

from pymongo import MongoClient

client = MongoClient(
    host="mongodb://your-host:27017",
    maxPoolSize=50,                   # Deliberate, not default
    connectTimeoutMS=5000,            # Fail fast on connection
    serverSelectionTimeoutMS=5000,    # Don't hang waiting for a server
    socketTimeoutMS=30000,            # Per-operation timeout
)

For async workloads, Motor wraps PyMongo with the same configuration surface — the options translate directly.

Write Concerns: The Setting Nobody Reads Until Data Goes Missing

w:1 acknowledges a write once the primary has written it. Fast. Slightly risky if the primary fails before replication completes.

w:majority acknowledges only after a majority of replica set members have written. Slower. Much safer for anything you care about not losing.

The default in recent MongoDB versions is w:majority for most operations, which is sensible. The mistake is explicitly overriding it to w:1 everywhere in pursuit of performance, and then discovering that your data isn’t quite as durable as you assumed. Use w:majority for writes that matter. Use w:1 consciously and only when you’ve decided the speed tradeoff is worth it.

Bulk Writes: Stop Looping, Start Batching

If you’re inserting or updating documents in a loop — one operation per iteration — you’re paying network round-trip cost for every single document. For a hundred documents this is fine. For a hundred thousand documents this is a performance disaster.

bulk_write() lets you bundle operations into a single request. Your loop body builds a list of InsertOne, UpdateOne, or DeleteOne operations, and you send them all at once. The performance difference for large data loads is not subtle.

from pymongo import InsertOne

operations = [InsertOne(doc) for doc in documents]
collection.bulk_write(operations, ordered=False)

ordered=False allows MongoDB to execute operations in parallel and continue past individual failures. For bulk inserts where you don’t need strict ordering, it’s almost always the right choice.

Part 5: The Schema Philosophy — Freedom Isn’t Free

“Schemaless” is one of the most misunderstood words in the document database marketing dictionary. Your data always has a schema. The question is whether it’s enforced by the database, enforced by your application, enforced by convention and good intentions, or enforced by nothing at all and slowly drifting into chaos.

My position, after seeing this play out many times: free schema for early stages, enforce it once you know your data shape.

When you’re building something new, you don’t fully know what your documents will look like. The flexibility is useful — you can iterate quickly without migration ceremonies. But “early stage” has an end date. Once your data model stabilizes, you should be enforcing it — either through MongoDB’s own schema validation (JSON Schema, set at the collection level) or through a library like Pydantic in Python. Beanie (which wraps Motor with Pydantic models) is particularly good for this.

When Schema Drift Finds You

The nastiest schema drift scenarios I’ve dealt with personally:

The missing field. Early documents don’t have a field that was added later. Your application code checks if doc.get('status') in forty places instead of just trusting the field exists. You don’t know which documents have the field and which don’t without querying. You can’t add a database-level NOT NULL constraint after the fact because you’re not in a relational database.

The structural evolution. A field that started as a plain string ("address": "123 Main St") later became an embedded object ("address": {"street": "123 Main St", "city": "Athens", "country": "GR"}). Now you have two incompatible shapes in the same collection. Your application has a branch for each. Your queries are creative. Your colleagues are suspicious.

Both of these happen when “schemaless” is interpreted as permission to evolve the data model without a migration plan. It’s like the end of The Fly — you thought you were in control of the transformation, but by the time you notice something is wrong, it’s already too late to reverse it cleanly.

Migrations: The Practical Approach

When you need to reshape documents — add missing fields, change structure — the right approach combines two strategies:

Batch backfill for critical fields. Use bulk_write with UpdateOne to patch documents in controlled batches. Note that update_many doesn’t support a limit parameter — so if you want genuine batching, you need to drive it yourself with find + bulk_write:

from pymongo import UpdateOne

while True:
    # Find a batch of documents missing the field
    batch = list(
        collection.find({"status": {"$exists": False}}, {"_id": 1}).limit(1000)
    )
    if not batch:
        break

    operations = [
        UpdateOne({"_id": doc["_id"]}, {"$set": {"status": "legacy"}})
        for doc in batch
    ]
    collection.bulk_write(operations, ordered=False)

This gives you real batching, observable progress, and the ability to pause and resume without reprocessing documents you’ve already fixed.

Application-layer fallback for the rest. For fields that aren’t critical to correctness, handle the missing case in code during the transition period. doc.get('new_field', default_value) is not elegant, but it’s safe. Remove the fallback once the backfill is complete and you’re confident old documents are gone.

The important thing is treating schema changes as first-class migrations. They are migrations. They just look different.

Part 6: Self-Managed vs Atlas vs DocumentDB — The Honest Comparison

I’ve used all three in production. Here’s what I actually think, without the vendor marketing.

Self-Managed MongoDB

Running MongoDB yourself — on your own servers or on cloud VMs — gives you complete control and, at scale, significantly lower costs than managed offerings. You own the hardware economics.

The cost is operational overhead. Replica set management, backups, upgrades, monitoring, capacity planning — all yours. For a team with strong infrastructure capabilities and a workload large enough to justify the economics, self-managed makes sense. For everyone else, the ops burden usually isn’t worth it.

MongoDB Atlas

Atlas is the right default for most teams. You get automatic backups, point-in-time recovery, built-in monitoring with the Performance Advisor, global clusters, and features like Atlas Search and native time-series support that don’t exist on other platforms. The operational overhead is minimal.

The honest caveat: Atlas has vendor lock-in, and you should acknowledge this rather than pretend it doesn’t exist. Atlas-specific features — Search indexes, Data API, Charts, Triggers — are not portable. If you build on them heavily and later want to migrate, you’ll be rewriting. That might be a fine tradeoff. Just make it consciously.

Cost also scales aggressively on Atlas. Dedicated clusters get expensive at scale. If you’re running significant workloads and watching the bill, this is the forcing function that eventually sends teams back to self-managed.

AWS DocumentDB

DocumentDB is where the conversation gets uncomfortable.

AWS DocumentDB is wire-compatible with MongoDB. That means your MongoDB driver can connect to it and most basic operations will work. What it doesn’t mean is that DocumentDB is MongoDB.

The aggregation pipeline support lags behind MongoDB. Change streams — which I recommended earlier in this article as the right approach for auditing — behave differently and have significant limitations on DocumentDB. Some MongoDB operators simply don’t exist. Every time MongoDB releases a major version, DocumentDB takes months or years to catch up with the features that actually matter.

It’s the Total Recall situation: looks human, passes basic inspection, but if you look closely enough, something is slightly off — and in production, “slightly off” tends to become “completely on fire.”

If you’re an AWS-native shop that does simple CRUD on documents, never uses change streams, doesn’t need the aggregation pipeline beyond basic operations, and wants to stay entirely within the AWS ecosystem for compliance reasons, DocumentDB is fine. It’s a reasonable choice for that specific profile.

If you need the full MongoDB feature set — especially change streams and advanced aggregation — DocumentDB will eventually frustrate you. The documentation lists compatibility caveats in the fine print. Read the fine print before committing.

And like Atlas, DocumentDB is vendor lock-in. Just AWS-flavored instead of MongoDB-flavored. Neither vendor will remind you of this when you’re signing up.

The Verdict

There’s no universally right answer, which is the only honest conclusion:

• Small team or startup? Atlas. Spend your operational budget on shipping features, not babysitting database clusters.
• Large scale with infrastructure maturity? Self-managed, when the economics justify it.
• AWS-native, simple workloads, compliance requirements? DocumentDB can work. Verify feature compatibility for your specific use case before committing.

Closing Thoughts

MongoDB is a good database that gets used badly at an impressive rate. The flexible schema is a design tool, not a permission slip. The aggregation pipeline is powerful and worth learning. Time-series collections are underused. Change streams solve auditing better than anything you’ll build manually.

The indexes chapter is the one that will make the biggest difference in your day-to-day life. Learn explain(). Design compound indexes around your query patterns. Treat collection scans in production as incidents, because that’s what they are.

And if you’re still creating a new MongoClient per request — please stop. The database did nothing to deserve that.

Tags: mongodb, document-databases, database, architecture, python, performance, indexing, distributed-systems

Meta: MongoDB done right: indexing, connection pooling, schema strategy, auditing with change streams, and an honest take on Atlas vs self-managed vs DocumentDB.