Backend development

Elasticsearch in 2026: Building Smart Search for High-Load REST APIs in Go

Ruslan Ismailov Published 14 min read
E

Introduction: Why Elasticsearch and Go in 2026

In 2026, full-text search remains one of the most demanding challenges in backend development. Users expect instant results with relevant rankings, typo tolerance, and complex filtering. Classic SQL queries with LIKE can't handle this at scale, while specialized solutions like Elasticsearch are purpose-built for exactly these scenarios.

Go has become the standard for high-load services: low overhead, a built-in goroutine scheduler, and a rich standard library for HTTP. The Go + Elasticsearch combination delivers a tool capable of processing thousands of search requests per second without performance degradation. In this article, we'll walk through everything from architectural design to Docker deployment with monitoring — using concrete code examples.

Architectural Overview

A typical architecture looks like this: the client calls the Go REST API, which translates the user's query into a DSL query for Elasticsearch and returns a normalized response. Data reaches the index in two ways: synchronously via the API (when creating or updating entities) and asynchronously via a message queue (Kafka, RabbitMQ) for bulk reindexing.

  • Go service — an HTTP server that accepts client requests and contains the search business logic.
  • elasticsearch-go client — the official library for interacting with the ES cluster.
  • Elasticsearch cluster — one or more nodes holding data indexes.
  • Kibana — for visualizing and debugging queries in the dev environment.
  • Prometheus + Grafana — monitoring metrics for both ES and the Go service.

The key principle: the Go service never gives clients direct access to Elasticsearch. All query-building logic is encapsulated at the service layer — this protects against injection attacks and allows DSL changes without altering the REST API contract.

Setting Up the elasticsearch-go Client

The official go-elasticsearch client supports all ES versions and provides type-safe API access. Installation:

go get github.com/elastic/go-elasticsearch/v8@latest

Initializing the client with connection pool configuration and retry logic:

package search

import (
    "crypto/tls"
    "net/http"
    "time"

    es8 "github.com/elastic/go-elasticsearch/v8"
)

func NewElasticsearchClient(addresses []string, username, password string) (*es8.Client, error) {
    transport := &http.Transport{
        MaxIdleConnsPerHost:   10,
        ResponseHeaderTimeout: 5 * time.Second,
        TLSClientConfig: &tls.Config{
            MinVersion: tls.VersionTLS12,
        },
    }

    cfg := es8.Config{
        Addresses: addresses,
        Username:  username,
        Password:  password,
        Transport: transport,
        // Retry on network errors and 5xx responses
        MaxRetries:            3,
        EnableRetryOnTimeout:  true,
        RetryBackoff: func(i int) time.Duration {
            return time.Duration(i*100) * time.Millisecond
        },
        // Cluster node discovery
        DiscoverNodesOnStart:  true,
        DiscoverNodesInterval: 5 * time.Minute,
    }

    client, err := es8.NewClient(cfg)
    if err != nil {
        return nil, fmt.Errorf("failed to create ES client: %w", err)
    }

    return client, nil
}

Important: always close the response body after processing. The client uses HTTP keep-alive, and unclosed bodies block connections in the pool. Wrap response handling in a helper:

func parseResponse(res *esapi.Response, target interface{}) error {
    defer res.Body.Close()

    if res.IsError() {
        var errBody map[string]interface{}
        if err := json.NewDecoder(res.Body).Decode(&errBody); err != nil {
            return fmt.Errorf("ES error [%s]", res.Status())
        }
        return fmt.Errorf("ES error [%s]: %v", res.Status(), errBody["error"])
    }

    return json.NewDecoder(res.Body).Decode(target)
}

Index Design: Mappings and Analyzers

A well-designed mapping is the foundation of performant search. For an e-commerce product catalog, an index might look like this:

{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "analysis": {
      "analyzer": {
        "russian_product": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "russian_stemmer", "synonym_filter"]
        }
      },
      "filter": {
        "russian_stemmer": {
          "type": "stemmer",
          "language": "russian"
        },
        "synonym_filter": {
          "type": "synonym",
          "synonyms_path": "analysis/synonyms.txt"
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "russian_product",
        "fields": {
          "keyword": { "type": "keyword" },
          "suggest": { "type": "search_as_you_type" }
        }
      },
      "description": {
        "type": "text",
        "analyzer": "russian_product"
      },
      "price": { "type": "scaled_float", "scaling_factor": 100 },
      "category_id": { "type": "keyword" },
      "tags": { "type": "keyword" },
      "in_stock": { "type": "boolean" },
      "created_at": { "type": "date" }
    }
  }
}

Key mapping decisions:

  • multi-field for title — text for full-text search, keyword for sorting, search_as_you_type for autocomplete.
  • scaled_float for price — precise calculations without precision loss in aggregations.
  • keyword for categories and tags — exact filtering without tokenization.

Creating the index from Go code:

func (r *ProductRepository) CreateIndex(ctx context.Context) error {
    mapping, err := os.ReadFile("mappings/products.json")
    if err != nil {
        return err
    }

    res, err := r.client.Indices.Create(
        "products",
        r.client.Indices.Create.WithBody(bytes.NewReader(mapping)),
        r.client.Indices.Create.WithContext(ctx),
    )
    if err != nil {
        return fmt.Errorf("failed to create index: %w", err)
    }
    defer res.Body.Close()

    if res.IsError() {
        return fmt.Errorf("index creation error: %s", res.Status())
    }

    return nil
}

Implementing Search: From Simple Queries to Aggregations

Full-Text Search with Fuzzy Matching

Let's implement a search method that supports fuzzy matching and field boosting by relevance:

type SearchRequest struct {
    Query      string   `json:"query"`
    Categories []string `json:"categories"`
    MinPrice   float64  `json:"min_price"`
    MaxPrice   float64  `json:"max_price"`
    InStock    *bool    `json:"in_stock"`
    Page       int      `json:"page"`
    PageSize   int      `json:"page_size"`
}

func (r *ProductRepository) Search(ctx context.Context, req SearchRequest) (*SearchResult, error) {
    from := (req.Page - 1) * req.PageSize

    // Build bool query
    query := map[string]interface{}{
        "query": map[string]interface{}{
            "bool": map[string]interface{}{
                "must": []map[string]interface{}{
                    {
                        "multi_match": map[string]interface{}{
                            "query":     req.Query,
                            "fields":    []string{"title^3", "description^1", "tags^2"},
                            "fuzziness": "AUTO",
                            "operator":  "and",
                        },
                    },
                },
                "filter": buildFilters(req),
            },
        },
        "from": from,
        "size": req.PageSize,
        "highlight": map[string]interface{}{
            "fields": map[string]interface{}{
                "title":       map[string]interface{}{},
                "description": map[string]interface{}{},
            },
        },
        "aggs": buildAggregations(),
    }

    body, _ := json.Marshal(query)

    res, err := r.client.Search(
        r.client.Search.WithContext(ctx),
        r.client.Search.WithIndex("products"),
        r.client.Search.WithBody(bytes.NewReader(body)),
        r.client.Search.WithTrackTotalHits(true),
    )
    if err != nil {
        return nil, fmt.Errorf("search request failed: %w", err)
    }

    var result SearchResult
    if err := parseResponse(res, &result); err != nil {
        return nil, err
    }

    return &result, nil
}

func buildFilters(req SearchRequest) []map[string]interface{} {
    filters := []map[string]interface{}{}

    if len(req.Categories) > 0 {
        filters = append(filters, map[string]interface{}{
            "terms": map[string]interface{}{"category_id": req.Categories},
        })
    }

    if req.MinPrice > 0 || req.MaxPrice > 0 {
        priceRange := map[string]interface{}{}
        if req.MinPrice > 0 {
            priceRange["gte"] = req.MinPrice
        }
        if req.MaxPrice > 0 {
            priceRange["lte"] = req.MaxPrice
        }
        filters = append(filters, map[string]interface{}{
            "range": map[string]interface{}{"price": priceRange},
        })
    }

    if req.InStock != nil {
        filters = append(filters, map[string]interface{}{
            "term": map[string]interface{}{"in_stock": *req.InStock},
        })
    }

    return filters
}

func buildAggregations() map[string]interface{} {
    return map[string]interface{}{
        "by_category": map[string]interface{}{
            "terms": map[string]interface{}{
                "field": "category_id",
                "size":  20,
            },
        },
        "price_stats": map[string]interface{}{
            "stats": map[string]interface{}{"field": "price"},
        },
        "price_ranges": map[string]interface{}{
            "range": map[string]interface{}{
                "field": "price",
                "ranges": []map[string]interface{}{
                    {"to": 1000},
                    {"from": 1000, "to": 5000},
                    {"from": 5000},
                },
            },
        },
    }
}

HTTP Handler in the REST API

func (h *SearchHandler) HandleSearch(w http.ResponseWriter, r *http.Request) {
    var req SearchRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        http.Error(w, "invalid request body", http.StatusBadRequest)
        return
    }

    if req.PageSize == 0 || req.PageSize > 100 {
        req.PageSize = 20
    }
    if req.Page == 0 {
        req.Page = 1
    }

    result, err := h.repo.Search(r.Context(), req)
    if err != nil {
        h.logger.Error("search failed", "error", err)
        http.Error(w, "search unavailable", http.StatusServiceUnavailable)
        return
    }

    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(result)
}

Performance Optimization

Bulk Indexing

When loading large amounts of data, use the Bulk API — it's orders of magnitude faster than indexing documents one by one:

func (r *ProductRepository) BulkIndex(ctx context.Context, products []Product) error {
    var buf bytes.Buffer

    for _, p := range products {
        meta := map[string]interface{}{
            "index": map[string]interface{}{
                "_index": "products",
                "_id":    strconv.Itoa(p.ID),
            },
        }
        metaLine, _ := json.Marshal(meta)
        buf.Write(metaLine)
        buf.WriteByte('\n')

        docLine, _ := json.Marshal(p)
        buf.Write(docLine)
        buf.WriteByte('\n')
    }

    res, err := r.client.Bulk(
        bytes.NewReader(buf.Bytes()),
        r.client.Bulk.WithContext(ctx),
        r.client.Bulk.WithIndex("products"),
        r.client.Bulk.WithRefresh("false"), // don't wait for refresh for speed
    )
    if err != nil {
        return fmt.Errorf("bulk index failed: %w", err)
    }
    defer res.Body.Close()

    var bulkResponse struct {
        Errors bool `json:"errors"`
        Items  []map[string]interface{} `json:"items"`
    }
    json.NewDecoder(res.Body).Decode(&bulkResponse)

    if bulkResponse.Errors {
        return fmt.Errorf("bulk index completed with errors")
    }

    return nil
}

Bulk operation recommendations: batches of 500–1000 documents, packet size no larger than 5–15 MB, parallel worker pools with goroutine limits via semaphore.

Caching and Shard Management

  • Request cache — ES automatically caches aggregations for static data. Enable "request_cache": true in requests that include aggregations.
  • Shard sizing — optimal shard size is 10–50 GB. Avoid over-sharding: 1–3 shards are sufficient for indexes up to 50 GB.
  • Go-level caching — for popular queries, add a Redis cache with a 30–60 second TTL before hitting ES. Use a hash of the query parameters as the cache key.
  • Index aliases — use aliases for zero-downtime reindexing: the service always references the alias, not a specific index.

Docker Deployment

Dockerfile for the Go service using a multi-stage build:

# Dockerfile
FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o search-api ./cmd/api

FROM alpine:3.19
RUN apk --no-cache add ca-certificates tzdata
WORKDIR /app
COPY --from=builder /app/search-api .
COPY --from=builder /app/mappings ./mappings
EXPOSE 8080
CMD ["./search-api"]

docker-compose.yml configuration:

version: '3.8'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
    environment:
      - node.name=es01
      - cluster.name=search-cluster
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    healthcheck:
      test: ["CMD-SHELL", "curl -sf http://localhost:9200/_cluster/health || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 5

  kibana:
    image: docker.elastic.co/kibana/kibana:8.13.0
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
    ports:
      - "5601:5601"
    depends_on:
      elasticsearch:
        condition: service_healthy

  search-api:
    build: .
    environment:
      - ES_ADDRESSES=http://elasticsearch:9200
      - ES_USERNAME=elastic
      - ES_PASSWORD=${ELASTIC_PASSWORD}
      - PORT=8080
    ports:
      - "8080:8080"
    depends_on:
      elasticsearch:
        condition: service_healthy

volumes:
  esdata:

Monitoring and Observability

In production, you need to track key Elasticsearch metrics: search request latency, JVM heap usage, the number of rejected requests in the thread pool, and cluster health.

Export metrics from the Go service via Prometheus:

var (
    searchDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "search_request_duration_seconds",
            Help:    "Duration of Elasticsearch search requests",
            Buckets: []float64{0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5},
        },
        []string{"status"},
    )
    searchErrors = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "search_errors_total",
            Help: "Total number of search errors",
        },
        []string{"type"},
    )
)

func init() {
    prometheus.MustRegister(searchDuration, searchErrors)
}

// Add measurement inside the Search method:
func (r *ProductRepository) Search(ctx context.Context, req SearchRequest) (*SearchResult, error) {
    start := time.Now()
    result, err := r.doSearch(ctx, req)

    status := "success"
    if err != nil {
        status = "error"
        searchErrors.WithLabelValues("elasticsearch").Inc()
    }
    searchDuration.WithLabelValues(status).Observe(time.Since(start).Seconds())

    return result, err
}

Additionally, set up elasticsearch-exporter for Prometheus — it collects metrics directly from the ES cluster and integrates easily with an existing Grafana dashboard.

Common Mistakes and How to Avoid Them

  • Dynamic mapping in production — always define explicit mappings. Dynamic mapping can create fields with unintended types and bloat the mapping to thousands of fields. Set "dynamic": "strict".
  • Using leading wildcard queries — *foo requires a full index scan. Instead, use search_as_you_type or an edge_ngram tokenizer for prefix search.
  • Ignoring circuit breakers — aggregations on large indexes can consume significant memory. Always set terminate_after and timeout in your queries.
  • No backpressure — the Go service must limit concurrent requests to ES. Use a semaphore or worker pool; otherwise, peak traffic will overwhelm the cluster.
  • Updating documents instead of using upsert — use _update with doc_as_upsert: true for idempotent operations to avoid errors on repeated requests.
  • Ignoring API versioning — different versions of go-elasticsearch are incompatible with each other. Pin the client version to match your ES cluster version.

Conclusion

Integrating Elasticsearch with a Go REST API in 2026 is a mature, well-understood problem with a rich toolset. The official elasticsearch-go client covers all use cases, from simple full-text search to complex aggregations. A well-designed mapping with analyzers, flexible DSL queries with fuzzy matching and filtering, bulk indexing, and Prometheus-based monitoring form the foundation of a reliable search service.

Start small: spin up the stack with Docker Compose, run load tests with real data, and set up alerts on latency and heap usage. Elasticsearch scales horizontally with ease — adding new nodes to the cluster resolves most performance issues as load grows. Investing in the right architecture from the start will pay dividends many times over as your system scales.

Technologies

Tags

Ruslan Ismailov

Senior Web / Backend Developer. Senior web/backend developer with 9 years of experience. Stack: PHP, Laravel, PostgreSQL, Redis, Docker, Kubernetes, REST, microservices, CI/CD. More about me →