Implementing the Outbox Pattern in PHP and MySQL: Guaranteed Event Delivery in Microservices
Introduction: The Dual Write Problem and Event Loss
In a microservice architecture, services communicate through events. A typical scenario: after saving an order to the database, you need to publish an OrderCreated event to a message broker (RabbitMQ, Kafka, etc.). At first glance, the task seems trivial — save the record, send the event. But this is exactly where the dual write problem hides.
Consider the naive approach:
// Save the order
$db->exec("INSERT INTO orders (id, status) VALUES (1, 'created')");
// Publish the event to the broker
$broker->publish('order.created', ['order_id' => 1]);
There is no atomicity between these two operations. If the application crashes after the INSERT but before publishing — the event is lost. If the broker is unavailable — the event never arrives. As a result, the state of the database and the state of other services diverge, causing hard-to-debug data inconsistencies.
The Transactional Outbox pattern exists specifically to solve this problem.
What Is the Transactional Outbox Pattern and Why You Need It
The Outbox pattern (also known as Transactional Outbox) is an architectural pattern that guarantees atomic writing of business data and the events that need to be published. The idea is simple: instead of immediately publishing an event to the broker, we save it to a dedicated outbox table within the same MySQL transaction as the main data. A separate process (a polling worker) reads this table and publishes the events to the broker.
Key benefits of the pattern:
- Guaranteed event delivery — the event is saved atomically with the business data and cannot be lost due to an application failure.
- No broker dependency at request processing time — if the broker is unavailable, the request still completes successfully.
- Idempotency — publishing can be safely retried when the worker fails.
- Simplicity — no distributed transactions or two-phase commit required.
The pattern is especially relevant in PHP microservices where transactional reliability is critical and distributed transactions are undesirable.
Architectural Overview: Atomic Writes in MySQL
The architecture is built on three components:
- The main service — saves business data and inserts a record into the
outboxtable within a single transaction. - The outbox table — an intermediate event store inside the same MySQL database.
- The polling worker — a background process that periodically reads unprocessed records from
outbox, publishes them to the broker, and marks them as processed.
It is important to understand: the atomicity guarantee is achieved precisely because the write to outbox happens within the same MySQL transaction as the INSERT/UPDATE of the main data. Either both operations succeed, or both are rolled back.
Step-by-Step Implementation in PHP and Laravel
Outbox Table Structure
Let's create the outbox table in MySQL. It should store the event type, payload, processing status, and timestamps:
CREATE TABLE outbox (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
event_type VARCHAR(255) NOT NULL,
payload JSON NOT NULL,
status ENUM('pending', 'processing', 'processed', 'failed')
DEFAULT 'pending' NOT NULL,
attempts TINYINT UNSIGNED DEFAULT 0 NOT NULL,
created_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
processed_at DATETIME(3) NULL,
INDEX idx_status_created (status, created_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
The idx_status_created index is critical for polling query performance — the worker always selects rows by status = 'pending', ordered by creation time.
Writing Events in a Single Transaction (Plain PHP)
An example without a framework using PDO:
$pdo->beginTransaction();
try {
// 1. Main business operation
$stmt = $pdo->prepare(
"INSERT INTO orders (user_id, status, total) VALUES (?, 'created', ?)"
);
$stmt->execute([$userId, $total]);
$orderId = $pdo->lastInsertId();
// 2. Write the event to outbox within the same transaction
$payload = json_encode([
'order_id' => $orderId,
'user_id' => $userId,
'total' => $total,
], JSON_THROW_ON_ERROR);
$outbox = $pdo->prepare(
"INSERT INTO outbox (event_type, payload) VALUES ('OrderCreated', ?)"
);
$outbox->execute([$payload]);
$pdo->commit();
} catch (\Throwable $e) {
$pdo->rollBack();
throw $e;
}
Both INSERTs execute within a single transaction — atomicity is guaranteed.
Writing Events in Laravel
In Laravel, the implementation is even more concise thanks to Eloquent and the DB::transaction() method:
use Illuminate\Support\Facades\DB;
DB::transaction(function () use ($userId, $total) {
$order = Order::create([
'user_id' => $userId,
'status' => 'created',
'total' => $total,
]);
DB::table('outbox')->insert([
'event_type' => 'OrderCreated',
'payload' => json_encode([
'order_id' => $order->id,
'user_id' => $userId,
'total' => $total,
], JSON_THROW_ON_ERROR),
'created_at' => now(),
]);
});
Polling Worker in PHP
The worker runs as a separate long-lived process (e.g., managed by supervisord). Its job is to periodically fetch a batch of unprocessed events and publish them:
while (true) {
$pdo->beginTransaction();
try {
// Lock up to 10 rows
$stmt = $pdo->query(
"SELECT id, event_type, payload
FROM outbox
WHERE status = 'pending'
ORDER BY created_at ASC
LIMIT 10
FOR UPDATE SKIP LOCKED"
);
$events = $stmt->fetchAll(PDO::FETCH_ASSOC);
if (empty($events)) {
$pdo->rollBack();
sleep(1); // Pause when queue is empty
continue;
}
$ids = array_column($events, 'id');
// Mark as processing
$placeholders = implode(',', array_fill(0, count($ids), '?'));
$pdo->prepare(
"UPDATE outbox SET status = 'processing' WHERE id IN ($placeholders)"
)->execute($ids);
$pdo->commit();
// Publish events to the broker (outside the transaction)
foreach ($events as $event) {
try {
$broker->publish(
$event['event_type'],
json_decode($event['payload'], true, 512, JSON_THROW_ON_ERROR)
);
// Mark as processed
$pdo->prepare(
"UPDATE outbox
SET status = 'processed', processed_at = NOW(3)
WHERE id = ?"
)->execute([$event['id']]);
} catch (\Throwable $e) {
// On failure — increment the attempt counter
$pdo->prepare(
"UPDATE outbox
SET status = 'pending', attempts = attempts + 1
WHERE id = ?"
)->execute([$event['id']]);
error_log("Outbox publish failed for id={$event['id']}: " . $e->getMessage());
}
}
} catch (\Throwable $e) {
$pdo->rollBack();
error_log('Outbox worker error: ' . $e->getMessage());
sleep(5);
}
}
Optimizing the Polling Worker
SELECT FOR UPDATE SKIP LOCKED in MySQL 8+
The SELECT FOR UPDATE SKIP LOCKED construct was introduced in MySQL 8.0 and is a key tool for scaling the worker. It allows multiple worker instances to process the queue in parallel without blocking each other: each instance skips rows already locked by another process.
Without SKIP LOCKED, a second worker will wait for the lock to be released — creating a bottleneck. With SKIP LOCKED, each worker instantly gets its own batch of rows and operates independently.
Polling Intervals and Adaptive Polling
A static sleep(1) is not always optimal. When the queue is empty, you can apply exponential back-off — increasing the pause to 5–10 seconds to reduce load on MySQL. When events appear, immediately reset the interval to the minimum.
Handling Duplicates and Idempotency
The worker may deliver an event twice (for example, if it crashes after publishing to the broker but before updating the status). Therefore, event consumers must be idempotent. It is useful to pass the id of the outbox record as a unique event identifier — consumers can use it for deduplication.
You should also add a maximum retry limit: if attempts >= 5, move the record to failed status and alert the team.
Monitoring the Outbox Table
Without monitoring, the Outbox pattern loses its value — you won't know about a growing backlog or stuck events.
Key metrics to track:
- Number of rows with
pendingstatus — if it keeps growing, the worker is falling behind or has stopped. - Number of rows with
failedstatus — requires immediate attention. - Lag — the difference between the
created_atof the oldestpendingrecord and the current time. - Worker throughput — the number of events processed per second.
Query to get the current queue status:
SELECT
status,
COUNT(*) AS count,
MIN(created_at) AS oldest,
MAX(attempts) AS max_attempts
FROM outbox
WHERE status IN ('pending', 'processing', 'failed')
GROUP BY status;
These metrics can easily be exported to Prometheus via a custom exporter or integrated into Laravel Horizon / Telescope. Set up an alert: if the lag exceeds 60 seconds or the number of failed records is greater than zero — send a notification to Slack/PagerDuty.
When the queue grows, first check: is the worker running, are there any broker connectivity issues, has MySQL hit its connection limit.
Comparison with Alternatives
Change Data Capture (CDC) via Debezium
CDC is a more advanced approach: Debezium reads the MySQL binary log (binlog) and automatically publishes row changes to Kafka. It requires no changes to application code and no polling worker. But it comes at a cost: infrastructure complexity (Kafka Connect, Zookeeper/KRaft are required), the need to configure MySQL replication, and limitations on data types and schemas.
The Outbox pattern with a polling worker is simpler to implement and operate for most teams, especially if Kafka is not already part of the project.
Direct Event Publishing
Publishing an event directly from the service code (without an outbox) is the simplest approach, but it provides no delivery guarantees. It is only suitable for non-critical notifications where losing an event is acceptable.
Two-Phase Commit (2PC)
Theoretically solves the problem, but in practice it is extremely complex to implement, scales poorly, and creates single points of failure. It is virtually never used in PHP microservices.
Real-World Cases and Pitfalls
In practice, teams implementing the Outbox pattern in PHP and MySQL encounter a number of non-trivial challenges:
- Outbox table growth — processed records need to be cleaned up periodically. A simple cron job with
DELETE FROM outbox WHERE status = 'processed' AND processed_at < NOW() - INTERVAL 7 DAYsolves the issue, but don't forget about partitioning under high load. - Event ordering — the polling worker processes events in
created_atorder, but with parallel workers, the order within a single aggregate may be violated. Solution: route events by key (e.g.,order_id) to a single worker instance. - Large payloads — the
JSONcolumn in MySQL stores data efficiently, but avoid storing the entire object in the event. It is better to pass only the identifier and event type, letting consumers fetch the current data themselves (Event Notification vs. Event-Carried State Transfer). - Long-running transactions — if the main business transaction takes a long time, the row in
outboxremains invisible to the worker for an extended period. Monitor transaction duration. - Multiple outbox tables — under high load, it makes sense to separate events from different domains into dedicated tables to avoid lock contention.
The Outbox pattern is not a silver bullet — it is a deliberate trade-off between complexity and reliability. It adds latency to event delivery (equal to the polling interval), but in return provides atomicity and fault tolerance.
Conclusion and Recommendations
The Transactional Outbox pattern is one of the most practical ways to ensure guaranteed event delivery in PHP microservices without distributed transactions. Its implementation on MySQL requires minimal infrastructure and fits naturally into existing PHP projects, including those built on Laravel.
Key recommendations for successful adoption:
- Always write the event to
outboxin the same transaction as the main data — this is the foundation of atomicity. - Use
SELECT FOR UPDATE SKIP LOCKEDfor safe parallel polling in MySQL 8+. - Make event consumers idempotent — the worker may deliver an event more than once.
- Set up monitoring: queue lag and the number of
failedrecords should be visible in your dashboards. - Regularly clean up processed records from the
outboxtable. - Consider CDC via Debezium only if you already have Kafka infrastructure and your team is ready to operate it.
Start with a simple implementation, measure performance, and scale as load grows. The Outbox pattern in PHP and MySQL is a proven solution that works reliably even in high-throughput systems.
Technologies
Tags
Ruslan Ismailov
Senior Web / Backend Developer. Senior web/backend developer with 9 years of experience. Stack: PHP, Laravel, PostgreSQL, Redis, Docker, Kubernetes, REST, microservices, CI/CD. More about me →