Event-Driven Microservices with Kafka, TypeScript, and Outbox Pattern in Production
In distributed systems and microservice architectures, the most pernicious failure mode is the Dual-Write Problem. Whenever a service needs to mutate local state in an ACID-compliant database and broadcast an event to a message broker (such as Apache Kafka or AWS SNS/SQS), executing those two operations sequentially invites silent data corruption.
If the database commit succeeds but the network blips before the message broker acknowledges receipt, events are lost forever. Conversely, if the broker accepts the message but the database transaction rolls back, downstream services act on phantom state.
This guide provides a comprehensive, production-tested blueprint for solving the dual-write problem using the Transactional Outbox Pattern, strictly typed schema registries in TypeScript, and resilient idempotent Kafka consumers.
The Dual-Write Vulnerability Deconstructed
Consider this standard e-commerce order creation flow implemented in an application service:
// ANTI-PATTERN: The Naive Dual-Write Hazard
async function createOrder(orderData: OrderInput) {
// Step 1: Database Write
const order = await db.orders.create({ data: orderData });
// DANGER: What if the application process crashes or Kafka is temporarily unreachable?
// Step 2: Broker Publish
await kafkaProducer.send({
topic: "orders.v1.created",
messages: [{ key: order.id, value: JSON.stringify(order) }],
});
return order;
}Scenario A: Network Partition during Kafka Publish
[User Request] ──► [DB Transaction: COMMITTED]
│
▼ (Network Drop or Broker Timeout)
[Kafka Broker: Message Dropped]
Outcome: Database record exists, but Inventory and Payment services never learn of the order.
Scenario B: Database Deadlock after Broker Publish
[User Request] ──► [Kafka Broker: Message ACKNOWLEDGED]
│
▼ (Unique constraint or Deadlock Rollback)
[DB Transaction: ROLLED BACK]
Outcome: Downstream services charge the user's card for an order that does not exist.The Architectural Solution: The Transactional Outbox Pattern
The Transactional Outbox Pattern sidesteps distributed two-phase commits (2PC) by persisting outbound events into a dedicated outbox table inside the exact same local ACID database transaction as the business entity.
Because both writes occur inside a single relational database transaction, they are guaranteed to either commit together or roll back together atomically. An asynchronous relay worker (or CDC engine like Debezium) then tails the outbox table and delivers events reliably to Kafka with at-least-once delivery guarantees.
[Service Layer]
│
▼ (Single Database Transaction)
┌──────────────────────────────────────────────┐
│ PostgreSQL ACID Transaction Boundary │
│ │
│ INSERT INTO orders (...) │
│ INSERT INTO outbox_events (...) │
└──────────────────────┬───────────────────────┘
│ COMMITTED ATOMICALLY
▼
┌──────────────────────────────────────────────┐
│ Outbox Polling Relay / Debezium CDC Engine │
└──────────────────────┬───────────────────────┘
│ Stream to Kafka Broker
▼
┌──────────────────────────────────────────────┐
│ Apache Kafka: Topic: "orders.v1.created" │
└──────────────────────┬───────────────────────┘
│ Consume with Idempotency Key
▼
┌──────────────────────────────────────────────┐
│ Downstream Consumers (Inventory, Billing) │
└──────────────────────────────────────────────┘Implementation Part 1: Schema Definition & Atomic Transaction
First, define the PostgreSQL outbox schema:
CREATE TABLE outbox_events (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
aggregate_type VARCHAR(64) NOT NULL,
aggregate_id VARCHAR(128) NOT NULL,
event_type VARCHAR(128) NOT NULL,
payload JSONB NOT NULL,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
published_at TIMESTAMP WITH TIME ZONE NULL,
retry_count INT DEFAULT 0,
error_reason TEXT NULL
);
-- Index for high-performance sequential outbox polling
CREATE INDEX idx_outbox_unpublished
ON outbox_events (created_at ASC)
WHERE published_at IS NULL;Next, implement the transactional unit of work in TypeScript:
import { PrismaClient } from "@prisma/client";
const prisma = new PrismaClient();
export interface OrderCreatedEvent {
orderId: string;
customerId: string;
totalAmountCents: number;
currency: string;
timestamp: string;
}
export async function placeOrder(
customerId: string,
amount: number,
currency: string
) {
return await prisma.$transaction(async (tx) => {
// 1. Insert primary business record
const order = await tx.order.create({
data: {
customerId,
amount,
currency,
status: "PENDING_PAYMENT",
},
});
const eventPayload: OrderCreatedEvent = {
orderId: order.id,
customerId: order.customerId,
totalAmountCents: order.amount,
currency: order.currency,
timestamp: new Date().toISOString(),
};
// 2. Insert into outbox table in the SAME transaction
await tx.outboxEvent.create({
data: {
aggregateType: "ORDER",
aggregateId: order.id,
eventType: "OrderCreated",
payload: eventPayload as any,
},
});
return order;
});
}Implementation Part 2: High-Throughput Outbox Relay Worker
The relay worker continuously polls unpublished records in batches, uses optimistic row locking (SKIP LOCKED) to prevent worker collisions in scaled horizontal deployments, and writes them to Kafka:
import { Kafka, Producer } from "kafkajs";
import { PrismaClient } from "@prisma/client";
const prisma = new PrismaClient();
const kafka = new Kafka({
clientId: "order-outbox-relay",
brokers: [process.env.KAFKA_BROKER_URL || "localhost:9092"],
});
const producer: Producer = kafka.producer({
allowAutoTopicCreation: false,
transactionTimeout: 30000,
});
export async function startOutboxRelay() {
await producer.connect();
console.log("Outbox Relay Kafka Producer Connected");
const BATCH_SIZE = 100;
const POLL_INTERVAL_MS = 250;
setInterval(async () => {
try {
// 1. Fetch unpublished events safely with row locking
const unpublishedEvents = await prisma.$queryRaw<
Array<{
id: string;
aggregate_type: string;
aggregate_id: string;
event_type: string;
payload: any;
}>
>`
SELECT id, aggregate_type, aggregate_id, event_type, payload
FROM outbox_events
WHERE published_at IS NULL
ORDER BY created_at ASC
LIMIT ${BATCH_SIZE}
FOR UPDATE SKIP LOCKED;
`;
if (unpublishedEvents.length === 0) return;
// 2. Construct batch Kafka messages with strict partition keys
const kafkaMessages = unpublishedEvents.map((evt) => ({
key: evt.aggregate_id, // Preserves partition order for the same entity
value: JSON.stringify(evt.payload),
headers: {
"x-event-id": evt.id,
"x-event-type": evt.event_type,
"x-aggregate-type": evt.aggregate_type,
},
}));
// 3. Publish batch to Kafka
await producer.send({
topic: "domain.orders.events",
messages: kafkaMessages,
});
// 4. Mark events as published
const eventIds = unpublishedEvents.map((e) => e.id);
await prisma.outboxEvent.updateMany({
where: { id: { in: eventIds } },
data: { published_at: new Date() },
});
} catch (relayError) {
console.error("Outbox relay processing batch error:", relayError);
}
}, POLL_INTERVAL_MS);
}Implementation Part 3: Idempotent Consumer Design
Because the relay guarantees at-least-once delivery, network retries can result in duplicate Kafka messages. Downstream microservices must be strictly idempotent.
import { EachMessagePayload } from "kafkajs";
export async function handleOrderCreatedConsumer({
message,
}: EachMessagePayload) {
if (!message.value) return;
const eventId = message.headers?.["x-event-id"]?.toString();
if (!eventId) {
console.error("Missing idempotency header; reject message to DLQ");
return;
}
// Check if this event was already processed
const alreadyProcessed = await db.processedEvents.findUnique({
where: { eventId },
});
if (alreadyProcessed) {
console.log(`Skipping duplicate event ${eventId}`);
return;
}
const payload: OrderCreatedEvent = JSON.parse(message.value.toString());
// Execute downstream business operation atomically with idempotency marker
await db.$transaction(async (tx) => {
// Deduct stock
await tx.inventory.update({
where: { productId: payload.orderId },
data: { reservedCount: { increment: 1 } },
});
// Record processed event key
await tx.processedEvents.create({
data: {
eventId,
processedAt: new Date(),
},
});
});
}Production Reliability: Dead-Letter Queues (DLQ) & Failure Triaging
Even with strict schemas, unprocessable messages (poison pills) can stall a Kafka partition. Implement a dead-letter strategy:
[Kafka Consumer Group]
│
├──► Success ──► Commit Partition Offset
│
└──► Failure (Retried 3 times)
│
▼
[Forward to DLQ: "domain.orders.events.dlq"]
│
▼
[Alert on PagerDuty + Inspect in Admin Dashboard]Frequently Asked Questions (FAQ)
What is the performance overhead of polling PostgreSQL for outbox events?
With the index WHERE published_at IS NULL and FOR UPDATE SKIP LOCKED, queries execute in less than 1 millisecond on indexed B-trees. In extreme throughput environments (>10,000 events/sec), replacing polling with Debezium Change Data Capture (CDC) streaming the PostgreSQL Write-Ahead Log (WAL) directly is the recommended upgrade path.
Why is the partition key critical in Apache Kafka?
Kafka guarantees message ordering only within a single partition. By using aggregate_id (e.g., order_id or user_id) as the Kafka message key, all events pertaining to that specific aggregate hash to the same partition, guaranteeing they are processed in strict chronological order.
How do I prevent the outbox table from growing indefinitely?
Schedule a periodic retention job (e.g., pg_cron or a nightly serverless worker) to prune or archive rows where published_at < NOW() - INTERVAL '7 days'.
Conclusion
Building resilient distributed systems requires designing for network unreliability and hardware failure. By adopting the Transactional Outbox Pattern combined with idempotent consumers and Kafka partition affinity, you eliminate dual-write hazards and guarantee data integrity across your entire microservices architecture.