Elasticsearch Percolator: Smart Notifications and Reverse Search in Microservices

Introduction: What Is Percolator and How Reverse Search Works

In the classic search model, you store documents and run queries against them. Elasticsearch Percolator flips this logic on its head: you store queries in an index, then check which of them match an incoming document. This is known as reverse search.

A practical example: a user sets up an alert — "notify me when the iPhone 16 Pro is in stock for less than $800." This filter is saved as a percolate query. When a new product document enters the system, Elasticsearch checks it against all stored queries and returns a list of matches — i.e., the users who need to be notified.

Key Percolator use cases in 2026:

Price alert systems in e-commerce
News feed monitoring and RSS aggregators
Metric alerts in observability platforms
Security event filtering (SIEM systems)
Smart content subscriptions on media platforms

The main difference from the classic approach: the number of stored queries can reach millions, while documents may arrive only a few per second. Percolator is specifically optimized for this inverted ratio of queries to data.

Architectural Overview: Percolator in a Microservice System

In a microservice architecture, Percolator typically serves as the core of a notification service or alert engine. A typical interaction schema involves several layers:

Subscription management service — accepts user-defined filters via REST API and stores them as percolate queries in Elasticsearch.
Event broker (Kafka, RabbitMQ, Pulsar) — receives incoming events from the product catalog, news feed, or monitoring system.
Percolate worker — a consumer microservice that pulls events from the broker, executes a percolate query against Elasticsearch, and forwards the list of matched subscriptions to the notification service.
Notification service — delivers notifications via email, push, Telegram, Slack, and other channels.

Asynchronous processing via an event broker is critical: it allows documents to be percolated independently of the main product stream without blocking writes. The percolate worker can scale horizontally — each instance handles its own topic partition.

An important architectural principle: the subscription index and the data index must be kept separate. Percolator operates on top of the target index mapping but stores queries in a dedicated field of type percolator.

Setting Up the Percolator Index: Mapping and Query Storage

Let's walk through the setup using a price alert index for a marketplace. First, we create the index with the correct mapping:

PUT /price-alerts
{
  "mappings": {
    "properties": {
      "query": {
        "type": "percolator"
      },
      "user_id": {
        "type": "keyword"
      },
      "notification_channel": {
        "type": "keyword"
      },
      "created_at": {
        "type": "date"
      },
      "category": {
        "type": "keyword"
      },
      "max_price": {
        "type": "double"
      },
      "product_name": {
        "type": "text",
        "analyzer": "standard"
      },
      "in_stock": {
        "type": "boolean"
      },
      "sku": {
        "type": "keyword"
      }
    }
  },
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "index.percolator.map_unmapped_fields_as_text": true
  }
}

The query field with type percolator is the key element. The remaining fields describe the structure of the documents that will be percolated. This is important: the mapping must match the structure of incoming data, not the subscriptions themselves.

Now let's save a user alert — a query for an iPhone under $800 that is in stock:

PUT /price-alerts/_doc/alert-user-42-iphone
{
  "user_id": "42",
  "notification_channel": "email",
  "created_at": "2026-01-15T10:00:00Z",
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "product_name": "iPhone 16 Pro"
          }
        },
        {
          "term": {
            "in_stock": true
          }
        }
      ],
      "filter": [
        {
          "range": {
            "max_price": {
              "lte": 80000
            }
          }
        }
      ]
    }
  }
}

Note that inside the query field you can use any standard Elasticsearch queries — bool, term, match, range, geo_distance, nested, and more. This provides enormous flexibility when building complex user-defined filters.

Implementing the Notification Service: Document Processing

When a new or updated product arrives in the catalog, the percolate worker executes a matching query:

POST /price-alerts/_search
{
  "query": {
    "percolate": {
      "field": "query",
      "document": {
        "product_name": "Apple iPhone 16 Pro 256GB",
        "sku": "APPL-IP16P-256",
        "max_price": 74990,
        "in_stock": true,
        "category": "smartphones"
      }
    }
  },
  "_source": ["user_id", "notification_channel"]
}

Elasticsearch returns all stored query documents that match the provided document. The response contains the _id of the alerts and user metadata. The worker then forwards the list to the notification service:

# Python worker pseudocode
def process_product_event(product: dict):
    response = es_client.search(
        index="price-alerts",
        body={
            "query": {
                "percolate": {
                    "field": "query",
                    "document": product
                }
            },
            "_source": ["user_id", "notification_channel"],
            "size": 1000  # max alerts per request
        }
    )

    matched_alerts = response["hits"]["hits"]

    for alert in matched_alerts:
        user_id = alert["_source"]["user_id"]
        channel = alert["_source"]["notification_channel"]
        notification_queue.publish({
            "user_id": user_id,
            "channel": channel,
            "product": product,
            "alert_id": alert["_id"]
        })

    return len(matched_alerts)

Important note: when there are many matches, use the size parameter and, if needed, pagination via search_after. By default, Elasticsearch returns only 10 matched queries.

Microservice Integration via REST API

The subscription management service exposes a REST API for the frontend and other microservices. A typical contract looks like this:

# Create an alert
POST /api/v1/alerts
Content-Type: application/json
Authorization: Bearer {token}

{
  "user_id": "42",
  "notification_channel": "push",
  "filters": {
    "product_name": "MacBook Pro",
    "max_price": 150000,
    "in_stock": true,
    "category": "laptops"
  }
}

# Response
{
  "alert_id": "alert-user-42-macbook-001",
  "status": "active",
  "created_at": "2026-03-10T12:00:00Z"
}

The subscription service translates user filters into an Elasticsearch DSL query and saves them in the percolator index. A key step is validating the query before saving. Elasticsearch provides the Validate API for this:

POST /price-alerts/_validate/query
{
  "query": {
    "bool": {
      "must": [
        { "match": { "product_name": "MacBook Pro" } },
        { "term": { "in_stock": true } }
      ],
      "filter": [
        { "range": { "max_price": { "lte": 150000 } } }
      ]
    }
  }
}

For asynchronous processing of incoming events, use Kafka with partitioning by product category. This allows each percolate worker instance to process its own category in parallel without competing for the same alerts.

When updating or deleting an alert, standard Elasticsearch Update/Delete operations are sufficient — the percolator index behaves like a regular index from a CRUD perspective.

Performance: Load Characteristics and Optimization

Percolator performance is primarily determined by the number of stored queries and their complexity. Key metrics and recommendations:

Shard count: Elasticsearch executes percolation in parallel across all shards. For one million queries, 5–10 shards is optimal. Excessive sharding creates coordination overhead.
Caching: Percolator makes aggressive use of the query cache. Term and filter queries (term, range) are cached efficiently. match queries are less cache-friendly — prefer term where possible.
Score sorting: if you don't need relevance scores, add "sort": ["_doc"] — this speeds up retrieval by 20–40%.
Document size: the smaller the document being percolated, the faster the processing. Avoid passing unnecessary fields.
Named queries: use _name in queries for debugging, but disable them in production — they add overhead.

Example of a production-optimized query:

POST /price-alerts/_search
{
  "query": {
    "percolate": {
      "field": "query",
      "document": {
        "product_name": "Samsung Galaxy S25",
        "max_price": 65000,
        "in_stock": true,
        "category": "smartphones"
      }
    }
  },
  "sort": ["_doc"],
  "_source": ["user_id", "notification_channel"],
  "size": 500,
  "track_total_hits": false
}

Benchmarks on a 3-node cluster with 16 CPU / 64 GB RAM show: with 500,000 stored percolate queries, the average execution time per percolate request is 15–50 ms depending on query complexity. With 5 million queries, it rises to 100–300 ms, requiring careful sharding optimization and hardware planning.

Real Use Case: End-to-End Price Alert System

Let's walk through the complete lifecycle of a price monitoring system for a marketplace with 2 million users:

User creates an alert via mobile app: "Notify me when the MacBook Pro drops below $1,500."
Subscription Service validates the request, translates it into Elasticsearch DSL, and saves it in the price-alerts index. In parallel, it stores metadata (email, push token) in PostgreSQL.
Supplier updates the price via Catalog Service → an event is published to the Kafka topic product.price.updated.
Percolate Worker (Go microservice) consumes the event, executes the percolate query, and receives the list of matched alerts.
Deduplication: results are checked against Redis — avoid notifying the same user for the same alert more than once per 24 hours. Key: notif:{alert_id}:{date}.
Notification Worker sends push notifications via Firebase and emails via SendGrid. Status is written to PostgreSQL.

// Go: Percolate Worker — simplified example
func (w *PercolateWorker) HandleProductEvent(ctx context.Context, product Product) error {
    res, err := w.esClient.Search(
        w.esClient.Search.WithIndex("price-alerts"),
        w.esClient.Search.WithBody(strings.NewReader(fmt.Sprintf(`{
            "query": {
                "percolate": {
                    "field": "query",
                    "document": {
                        "product_name": %q,
                        "max_price": %f,
                        "in_stock": %v,
                        "category": %q
                    }
                }
            },
            "sort": ["_doc"],
            "_source": ["user_id", "notification_channel"],
            "size": 1000,
            "track_total_hits": false
        }`, product.Name, product.Price, product.InStock, product.Category))),
    )
    if err != nil {
        return fmt.Errorf("percolate query failed: %w", err)
    }
    defer res.Body.Close()

    var result PercolateResult
    if err := json.NewDecoder(res.Body).Decode(&result); err != nil {
        return fmt.Errorf("decode response failed: %w", err)
    }

    for _, hit := range result.Hits.Hits {
        dedupKey := fmt.Sprintf("notif:%s:%s", hit.ID, time.Now().Format("2006-01-02"))
        if w.redis.SetNX(ctx, dedupKey, 1, 24*time.Hour).Val() {
            w.notifQueue.Publish(ctx, NotificationTask{
                AlertID:  hit.ID,
                UserID:   hit.Source.UserID,
                Channel:  hit.Source.NotificationChannel,
                Product:  product,
            })
        }
    }
    return nil
}

This architecture handles up to 10,000 price events per minute with 2 million active alerts, staying within a 500 ms end-to-end SLA from event to notification delivery.

Pitfalls and Limitations of Percolator

Percolator is a powerful tool, but it has significant limitations to account for when designing your system:

No support for join queries across documents. Percolator checks one document at a time. If your alert needs to account for related data from another index, you'll need to denormalize the data before percolating.
Query complexity has a nonlinear impact. Nested bool queries with wildcards or script filters can slow down processing by an order of magnitude. Profile them using "profile": true.
Mapping updates require reindexing. If the document structure changes, you'll need to recreate the alert index and migrate stored queries.
No native deduplication. Elasticsearch does not track which alerts have already fired — that's your responsibility (Redis, PostgreSQL).
Aggregations are not supported. A percolate query cannot contain aggregations inside a stored filter.
Not suitable for high-frequency events with simple rules. If you have 100,000 events per second with straightforward conditions, consider Kafka Streams or Flink, which offer lower latency.
Index size. Millions of complex queries require significant RAM for Lucene segments. Plan Elasticsearch heap at ~1–5 KB per query.

Percolator is ideal when the number of unique user filters vastly exceeds the frequency of incoming events, and the filters themselves are moderately complex. For simple rules with high event frequency, consider stream processing; for complex join conditions, consider an application-level rule engine.

Conclusion

Elasticsearch Percolator remains one of the most elegant tools for building smart notification systems and reverse search in microservice architectures. In 2026, it is more relevant than ever: the growth of personalized user filters in e-commerce, media, and observability creates exactly the load pattern that Percolator is optimized for.

Key takeaways for practical use:

Keep data indexes and percolator query indexes separate, and define the correct mapping from the start.
Build asynchronous processing through an event broker — this is a prerequisite for a production-ready system.
Implement notification deduplication at the Redis layer, outside of Elasticsearch.
Profile complex queries and prefer term/range filters over match and script queries.
Test performance with a realistic volume of stored queries — degradation is nonlinear.

A well-designed system built on Elasticsearch Percolator can serve millions of user alerts in real time with acceptable latency and horizontal scalability — without building a custom rule engine from scratch.

Elasticsearch Percolator: Smart Alerts and Reverse Search in Microservice Architecture