Backend development

Go and gRPC in Kubernetes: Building High-Performance Inter-Service Communication with Load Balancing and Observability

Ruslan Ismailov Published 22 min read
G

1. Introduction: Why gRPC Outperforms REST for Inter-Service Communication Inside Kubernetes in 2026

By 2026, gRPC has firmly established itself as the de facto standard for inter-service communication inside Kubernetes clusters. While REST API remains a convenient choice for public APIs and browser interactions, gRPC offers fundamental advantages within microservice architectures.

First, gRPC uses HTTP/2, which means request multiplexing over a single connection, header compression, and binary serialization via Protocol Buffers. In practice, this delivers 20–40% lower latency and 3–10x reduction in data transfer volume compared to JSON-based REST APIs.

Second, strict typing through .proto files enforces contract-driven development: any API violation is caught at code generation time, not at runtime. This is critical when a team of 10+ developers is working across different microservices.

Third, gRPC natively supports four interaction patterns: unary RPC, server streaming, client streaming, and bidirectional streaming — capabilities that are extremely difficult to implement cleanly with REST API.

Finally, the Go ecosystem has first-class gRPC support via the official google.golang.org/grpc library, making the Go + gRPC + Kubernetes stack especially powerful for high-load systems.

2. Basic gRPC Service Setup in Go: Proto Files, Code Generation, Project Structure

Let's start by defining a service using Protocol Buffers. We'll create a typical order management service for a microservice architecture.

Project Structure

order-service/
├── api/
│   └── proto/
│       └── order/
│           └── v1/
│               └── order.proto
├── cmd/
│   └── server/
│       └── main.go
├── internal/
│   ├── server/
│   │   └── order_server.go
│   └── repository/
│       └── order_repo.go
├── gen/
│   └── go/
│       └── order/
│           └── v1/
├── Makefile
├── Dockerfile
└── go.mod

Proto File

// api/proto/order/v1/order.proto
syntax = "proto3";

package order.v1;

option go_package = "github.com/myorg/order-service/gen/go/order/v1;orderv1";

import "google/protobuf/timestamp.proto";

enum OrderStatus {
  ORDER_STATUS_UNSPECIFIED = 0;
  ORDER_STATUS_PENDING = 1;
  ORDER_STATUS_PROCESSING = 2;
  ORDER_STATUS_COMPLETED = 3;
  ORDER_STATUS_CANCELLED = 4;
}

message Order {
  string id = 1;
  string customer_id = 2;
  repeated OrderItem items = 3;
  OrderStatus status = 4;
  double total_amount = 5;
  google.protobuf.Timestamp created_at = 6;
}

message OrderItem {
  string product_id = 1;
  int32 quantity = 2;
  double price = 3;
}

message CreateOrderRequest {
  string customer_id = 1;
  repeated OrderItem items = 2;
}

message CreateOrderResponse {
  Order order = 1;
}

message GetOrderRequest {
  string order_id = 1;
}

message GetOrderResponse {
  Order order = 1;
}

message ListOrdersRequest {
  string customer_id = 1;
  int32 page_size = 2;
  string page_token = 3;
}

message ListOrdersResponse {
  repeated Order orders = 1;
  string next_page_token = 2;
}

service OrderService {
  rpc CreateOrder(CreateOrderRequest) returns (CreateOrderResponse);
  rpc GetOrder(GetOrderRequest) returns (GetOrderResponse);
  rpc ListOrders(ListOrdersRequest) returns (ListOrdersResponse);
  // Server streaming for tracking status changes
  rpc WatchOrderStatus(GetOrderRequest) returns (stream Order);
}

Code Generation via Makefile

# Makefile
PROTO_DIR := api/proto
GEN_DIR := gen/go

.PHONY: proto
proto:
	protoc \
		--proto_path=$(PROTO_DIR) \
		--go_out=$(GEN_DIR) \
		--go_opt=paths=source_relative \
		--go-grpc_out=$(GEN_DIR) \
		--go-grpc_opt=paths=source_relative \
		$(shell find $(PROTO_DIR) -name '*.proto')

.PHONY: run
run:
	go run ./cmd/server/...

.PHONY: test
test:
	go test -v -race ./...

gRPC Server Implementation in Go

// internal/server/order_server.go
package server

import (
	"context"
	"time"

	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"
	"google.golang.org/protobuf/types/known/timestamppb"

	orderv1 "github.com/myorg/order-service/gen/go/order/v1"
	"github.com/myorg/order-service/internal/repository"
)

type OrderServer struct {
	orderv1.UnimplementedOrderServiceServer
	repo repository.OrderRepository
}

func NewOrderServer(repo repository.OrderRepository) *OrderServer {
	return &OrderServer{repo: repo}
}

func (s *OrderServer) CreateOrder(
	ctx context.Context,
	req *orderv1.CreateOrderRequest,
) (*orderv1.CreateOrderResponse, error) {
	if req.CustomerId == "" {
		return nil, status.Error(codes.InvalidArgument, "customer_id is required")
	}
	if len(req.Items) == 0 {
		return nil, status.Error(codes.InvalidArgument, "at least one item is required")
	}

	order, err := s.repo.Create(ctx, req.CustomerId, req.Items)
	if err != nil {
		return nil, status.Errorf(codes.Internal, "failed to create order: %v", err)
	}

	return &orderv1.CreateOrderResponse{Order: order}, nil
}

func (s *OrderServer) WatchOrderStatus(
	req *orderv1.GetOrderRequest,
	stream orderv1.OrderService_WatchOrderStatusServer,
) error {
	ctx := stream.Context()
	ticker := time.NewTicker(2 * time.Second)
	defer ticker.Stop()

	for {
		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-ticker.C:
			order, err := s.repo.GetByID(ctx, req.OrderId)
			if err != nil {
				return status.Errorf(codes.Internal, "failed to get order: %v", err)
			}
			if err := stream.Send(order); err != nil {
				return err
			}
			if order.Status == orderv1.OrderStatus_ORDER_STATUS_COMPLETED ||
				order.Status == orderv1.OrderStatus_ORDER_STATUS_CANCELLED {
				return nil
			}
		}
	}
}

Server Entry Point

// cmd/server/main.go
package main

import (
	"fmt"
	"net"
	"os"
	"os/signal"
	"syscall"

	"google.golang.org/grpc"
	"google.golang.org/grpc/reflection"

	orderv1 "github.com/myorg/order-service/gen/go/order/v1"
	"github.com/myorg/order-service/internal/server"
)

func main() {
	port := os.Getenv("GRPC_PORT")
	if port == "" {
		port = "50051"
	}

	lis, err := net.Listen("tcp", fmt.Sprintf(":%s", port))
	if err != nil {
		panic(err)
	}

	grpcServer := grpc.NewServer(
		grpc.ChainUnaryInterceptor(
			loggingInterceptor,
			recoveryInterceptor,
		),
	)

	orderSrv := server.NewOrderServer(/* inject dependencies */)
	orderv1.RegisterOrderServiceServer(grpcServer, orderSrv)

	// gRPC reflection for grpcurl and other tools
	reflection.Register(grpcServer)

	quit := make(chan os.Signal, 1)
	signal.Notify(quit, syscall.SIGTERM, syscall.SIGINT)

	go func() {
		fmt.Printf("gRPC server listening on :%s\n", port)
		if err := grpcServer.Serve(lis); err != nil {
			panic(err)
		}
	}()

	<-quit
	fmt.Println("Shutting down gRPC server...")
	grpcServer.GracefulStop()
}

3. gRPC in Kubernetes: Why Standard kube-proxy Works Poorly with gRPC

This is one of the most common pitfalls when deploying gRPC services in Kubernetes. kube-proxy implements load balancing at the L4 (TCP) level using iptables or IPVS. For HTTP/1.1, this works fine: each request opens a new TCP connection, and the load balancer can distribute them evenly.

gRPC, built on top of HTTP/2, establishes a single long-lived TCP connection and multiplexes all requests through it. When a gRPC client connects to a ClusterIP Service, kube-proxy routes that single connection to a specific Pod and no longer balances subsequent RPC calls. As a result, one Pod receives 100% of the traffic while the others sit idle.

The problem is clearly illustrated by this scenario: 3 replicas of order-service, a ClusterIP Service, and all traffic going only to Pod #1.

There are three main approaches to solving this problem:

  • Client-side load balancing — the client is aware of all endpoints and balances requests itself.
  • Proxy/sidecar — Envoy or another L7 proxy intercepts traffic and balances at the HTTP/2 frame level.
  • Headless Service + DNS — DNS returns the IPs of all Pods, and the client connects to all of them.

4. Client-Side Load Balancing for gRPC in Go

The Go gRPC client has built-in load balancing support through the resolver.Resolver and balancer.Balancer interfaces. The key point is that the client must obtain the list of all Pod IP addresses, not the ClusterIP Service address.

Using a Headless Service and DNS Resolver

For client-side balancing, we create a headless Service (with clusterIP: None). In this case, DNS returns A records for each Pod rather than a single ClusterIP.

# kubernetes/order-service-headless.yaml
apiVersion: v1
kind: Service
metadata:
  name: order-service-headless
  namespace: production
spec:
  clusterIP: None  # This makes the Service headless
  selector:
    app: order-service
  ports:
    - name: grpc
      port: 50051
      targetPort: 50051
      protocol: TCP

gRPC Client with Round-Robin Load Balancing

// pkg/client/order_client.go
package client

import (
	"context"
	"fmt"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/balancer/roundrobin"
	"google.golang.org/grpc/credentials/insecure"
	"google.golang.org/grpc/keepalive"

	orderv1 "github.com/myorg/order-service/gen/go/order/v1"
)

type OrderClient struct {
	client orderv1.OrderServiceClient
	conn   *grpc.ClientConn
}

func NewOrderClient(ctx context.Context, target string) (*OrderClient, error) {
	// target for headless service: "dns:///order-service-headless.production.svc.cluster.local:50051"
	conn, err := grpc.DialContext(
		ctx,
		target,
		grpc.WithTransportCredentials(insecure.NewCredentials()),
		// Enable round-robin load balancing
		grpc.WithDefaultServiceConfig(`{"loadBalancingPolicy":"round_robin"}`),
		// Keepalive to maintain live connections to all endpoints
		grpc.WithKeepaliveParams(keepalive.ClientParameters{
			Time:                10 * time.Second,
			Timeout:             3 * time.Second,
			PermitWithoutStream: true,
		}),
		// Enable wait-for-ready for automatic reconnection
		grpc.WithDefaultCallOptions(grpc.WaitForReady(true)),
	)
	if err != nil {
		return nil, fmt.Errorf("failed to dial order service: %w", err)
	}

	return &OrderClient{
		client: orderv1.NewOrderServiceClient(conn),
		conn:   conn,
	}, nil
}

func (c *OrderClient) CreateOrder(
	ctx context.Context,
	customerID string,
	items []*orderv1.OrderItem,
) (*orderv1.Order, error) {
	resp, err := c.client.CreateOrder(ctx, &orderv1.CreateOrderRequest{
		CustomerId: customerID,
		Items:      items,
	})
	if err != nil {
		return nil, fmt.Errorf("CreateOrder RPC failed: %w", err)
	}
	return resp.Order, nil
}

func (c *OrderClient) Close() error {
	return c.conn.Close()
}

Custom Kubernetes Resolver

For more flexible management, you can implement a custom resolver that queries the Kubernetes Endpoints API directly via client-go. This allows instant reaction to Pod changes without DNS TTL delays.

// pkg/resolver/k8s_resolver.go
package resolver

import (
	"context"
	"fmt"

	v1 "k8s.io/api/core/v1"
	"k8s.io/client-go/informers"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/tools/cache"
	"google.golang.org/grpc/resolver"
)

const Scheme = "k8s"

type k8sResolverBuilder struct {
	clientset *kubernetes.Clientset
}

func NewBuilder(clientset *kubernetes.Clientset) resolver.Builder {
	return &k8sResolverBuilder{clientset: clientset}
}

func (b *k8sResolverBuilder) Build(
	target resolver.Target,
	cc resolver.ClientConn,
	opts resolver.BuildOptions,
) (resolver.Resolver, error) {
	r := &k8sResolver{
		cc:        cc,
		clientset: b.clientset,
		namespace: target.URL.Host,
		service:   target.URL.Path[1:], // strip leading /
		ctx:       context.Background(),
	}
	go r.watch()
	return r, nil
}

func (b *k8sResolverBuilder) Scheme() string { return Scheme }

type k8sResolver struct {
	cc        resolver.ClientConn
	clientset *kubernetes.Clientset
	namespace string
	service   string
	ctx       context.Context
}

func (r *k8sResolver) watch() {
	factory := informers.NewSharedInformerFactoryWithOptions(
		r.clientset,
		0,
		informers.WithNamespace(r.namespace),
	)
	endpointsInformer := factory.Core().V1().Endpoints().Informer()
	endpointsInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
		AddFunc:    func(obj interface{}) { r.updateAddresses(obj) },
		UpdateFunc: func(_, obj interface{}) { r.updateAddresses(obj) },
		DeleteFunc: func(obj interface{}) { r.updateAddresses(obj) },
	})
	factory.Start(r.ctx.Done())
}

func (r *k8sResolver) updateAddresses(obj interface{}) {
	endpoints, ok := obj.(*v1.Endpoints)
	if !ok || endpoints.Name != r.service {
		return
	}

	var addrs []resolver.Address
	for _, subset := range endpoints.Subsets {
		for _, addr := range subset.Addresses {
			for _, port := range subset.Ports {
				addrs = append(addrs, resolver.Address{
					Addr: fmt.Sprintf("%s:%d", addr.IP, port.Port),
				})
			}
		}
	}

	r.cc.UpdateState(resolver.State{Addresses: addrs})
}

func (r *k8sResolver) ResolveNow(opts resolver.ResolveNowOptions) {}
func (r *k8sResolver) Close()                                     {}

5. Server-Side Load Balancing: Envoy as a Sidecar

An alternative to client-side balancing is using Envoy proxy in sidecar mode. Envoy understands HTTP/2 and gRPC at the L7 level, enabling balancing of individual RPC calls rather than TCP connections.

Deployment with Envoy Sidecar

# kubernetes/order-service-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      containers:
      - name: order-service
        image: myorg/order-service:latest
        ports:
        - containerPort: 50051
          name: grpc
        env:
        - name: GRPC_PORT
          value: "50051"
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 256Mi
      - name: envoy
        image: envoyproxy/envoy:v1.28.0
        ports:
        - containerPort: 10000
          name: grpc-envoy
        - containerPort: 9901
          name: admin
        volumeMounts:
        - name: envoy-config
          mountPath: /etc/envoy
      volumes:
      - name: envoy-config
        configMap:
          name: envoy-config

Envoy Configuration for gRPC

# kubernetes/envoy-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-config
  namespace: production
data:
  envoy.yaml: |
    static_resources:
      listeners:
      - name: grpc_listener
        address:
          socket_address:
            address: 0.0.0.0
            port_value: 10000
        filter_chains:
        - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              codec_type: AUTO
              stat_prefix: grpc_ingress
              http2_protocol_options: {}
              route_config:
                name: local_route
                virtual_hosts:
                - name: backend
                  domains: ["*"]
                  routes:
                  - match:
                      prefix: "/"
                    route:
                      cluster: order_service_cluster
                      timeout: 30s
                      retry_policy:
                        retry_on: "reset,connect-failure,retriable-status-codes"
                        num_retries: 3
                        retriable_status_codes: [503]
              http_filters:
              - name: envoy.filters.http.grpc_stats
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.grpc_stats.v3.FilterConfig
                  stats_for_all_methods: true
              - name: envoy.filters.http.router
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

      clusters:
      - name: order_service_cluster
        connect_timeout: 5s
        type: STRICT_DNS
        lb_policy: ROUND_ROBIN
        typed_extension_protocol_options:
          envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
            "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
            explicit_http_config:
              http2_protocol_options: {}
        load_assignment:
          cluster_name: order_service_cluster
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: order-service-headless.production.svc.cluster.local
                    port_value: 50051

    admin:
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 9901

6. Health Checking and Graceful Shutdown

Kubernetes requires working health-check endpoints for proper Pod lifecycle management. The gRPC Health Checking Protocol is the standard way to implement this.

gRPC Health Check Implementation in Go

// cmd/server/main.go (extended version)
package main

import (
	"context"
	"fmt"
	"net"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/health"
	"google.golang.org/grpc/health/grpc_health_v1"
	"google.golang.org/grpc/reflection"

	orderv1 "github.com/myorg/order-service/gen/go/order/v1"
	"github.com/myorg/order-service/internal/server"
)

func main() {
	grpcPort := getEnv("GRPC_PORT", "50051")
	httpPort := getEnv("HTTP_PORT", "8080") // for HTTP liveness probe

	lis, err := net.Listen("tcp", fmt.Sprintf(":%s", grpcPort))
	if err != nil {
		panic(err)
	}

	grpcServer := grpc.NewServer()

	// Register health check service
	healthSrv := health.NewServer()
	grpc_health_v1.RegisterHealthServer(grpcServer, healthSrv)

	orderSrv := server.NewOrderServer()
	orderv1.RegisterOrderServiceServer(grpcServer, orderSrv)
	reflection.Register(grpcServer)

	// Mark service as SERVING
	healthSrv.SetServingStatus("order.v1.OrderService", grpc_health_v1.HealthCheckResponse_SERVING)
	healthSrv.SetServingStatus("", grpc_health_v1.HealthCheckResponse_SERVING)

	// HTTP server for Kubernetes liveness/readiness probes
	httpMux := http.NewServeMux()
	httpMux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
		w.Write([]byte("ok"))
	})
	httpMux.HandleFunc("/ready", func(w http.ResponseWriter, r *http.Request) {
		// Check dependencies (DB, cache, etc.)
		w.WriteHeader(http.StatusOK)
		w.Write([]byte("ready"))
	})
	httpServer := &http.Server{
		Addr:    fmt.Sprintf(":%s", httpPort),
		Handler: httpMux,
	}

	quit := make(chan os.Signal, 1)
	signal.Notify(quit, syscall.SIGTERM, syscall.SIGINT)

	go func() {
		fmt.Printf("gRPC server on :%s\n", grpcPort)
		if err := grpcServer.Serve(lis); err != nil {
			panic(err)
		}
	}()

	go func() {
		fmt.Printf("HTTP health server on :%s\n", httpPort)
		if err := httpServer.ListenAndServe(); err != nil && err != http.ErrServerClosed {
			panic(err)
		}
	}()

	<-quit

	// Graceful shutdown
	// 1. Mark service as NOT_SERVING so no new requests are accepted
	healthSrv.SetServingStatus("", grpc_health_v1.HealthCheckResponse_NOT_SERVING)
	healthSrv.SetServingStatus("order.v1.OrderService", grpc_health_v1.HealthCheckResponse_NOT_SERVING)

	// 2. Allow time for Kubernetes to update Endpoints (terminationGracePeriodSeconds)
	time.Sleep(5 * time.Second)

	// 3. Finish in-flight requests
	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()
	httpServer.Shutdown(ctx)
	grpcServer.GracefulStop()
}

func getEnv(key, fallback string) string {
	if v := os.Getenv(key); v != "" {
		return v
	}
	return fallback
}

Kubernetes Deployment with Probes

# kubernetes/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  namespace: production
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: order-service
        image: myorg/order-service:latest
        ports:
        - containerPort: 50051
          name: grpc
        - containerPort: 8080
          name: http
        livenessProbe:
          httpGet:
            path: /healthz
            port: http
          initialDelaySeconds: 10
          periodSeconds: 10
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /ready
            port: http
          initialDelaySeconds: 5
          periodSeconds: 5
          failureThreshold: 3
        lifecycle:
          preStop:
            exec:
              # Additional delay before SIGTERM for proper drain
              command: ["/bin/sh", "-c", "sleep 5"]

7. Observability: Metrics, Tracing, Logging

Observability is one of the key aspects of production-ready gRPC services in Kubernetes. Let's look at integration with Prometheus and OpenTelemetry.

gRPC Metrics with Prometheus

// cmd/server/main.go — adding Prometheus metrics
package main

import (
	"net/http"

	"github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/prometheus"
	prom "github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
	"google.golang.org/grpc"
)

func buildGRPCServer() *grpc.Server {
	reg := prom.NewRegistry()

	grpcMetrics := prometheus.NewServerMetrics(
		prometheus.WithServerHandlingTimeHistogram(
			prometheus.WithHistogramBuckets([]float64{.001, .005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10}),
		),
	)
	reg.MustRegister(grpcMetrics)

	grpcServer := grpc.NewServer(
		grpc.ChainUnaryInterceptor(
			grpcMetrics.UnaryServerInterceptor(),
		),
		grpc.ChainStreamInterceptor(
			grpcMetrics.StreamServerInterceptor(),
		),
	)

	// Prometheus HTTP endpoint
	http.Handle("/metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{}))

	return grpcServer
}

OpenTelemetry Tracing

// pkg/telemetry/tracer.go
package telemetry

import (
	"context"
	"fmt"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
)

func InitTracer(ctx context.Context, serviceName, otlpEndpoint string) (func(), error) {
	exporter, err := otlptracegrpc.New(ctx,
		otlptracegrpc.WithEndpoint(otlpEndpoint),
		otlptracegrpc.WithInsecure(),
	)
	if err != nil {
		return nil, fmt.Errorf("failed to create OTLP exporter: %w", err)
	}

	res, err := resource.New(ctx,
		resource.WithAttributes(
			semconv.ServiceName(serviceName),
			semconv.ServiceVersion("1.0.0"),
		),
	)
	if err != nil {
		return nil, err
	}

	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(res),
		sdktrace.WithSampler(sdktrace.AlwaysSample()),
	)

	otel.SetTracerProvider(tp)
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
		propagation.TraceContext{},
		propagation.Baggage{},
	))

	return func() { tp.Shutdown(ctx) }, nil
}

// OTel integration with gRPC
// Add the following interceptors in main.go:
// import "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
//
// grpc.NewServer(
//     grpc.StatsHandler(otelgrpc.NewServerHandler()),
// )

ServiceMonitor for Prometheus Operator

# kubernetes/service-monitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: order-service-monitor
  namespace: production
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: order-service
  endpoints:
  - port: http
    path: /metrics
    interval: 15s
    scrapeTimeout: 10s

Structured Logging

// internal/interceptors/logging.go
package interceptors

import (
	"context"
	"time"

	"go.uber.org/zap"
	"google.golang.org/grpc"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"
)

func LoggingUnaryInterceptor(logger *zap.Logger) grpc.UnaryServerInterceptor {
	return func(
		ctx context.Context,
		req interface{},
		info *grpc.UnaryServerInfo,
		handler grpc.UnaryHandler,
	) (interface{}, error) {
		start := time.Now()
		resp, err := handler(ctx, req)
		duration := time.Since(start)

		code := codes.OK
		if err != nil {
			code = status.Code(err)
		}

		logger.Info("gRPC request",
			zap.String("method", info.FullMethod),
			zap.String("code", code.String()),
			zap.Duration("duration", duration),
			zap.Error(err),
		)
		return resp, err
	}
}

8. Security: mTLS for gRPC in Kubernetes Without a Service Mesh

Mutual authentication via mTLS is the security standard for gRPC in production. Let's implement it without a service mesh, using cert-manager for certificate management.

Installing cert-manager and Creating a CA

# kubernetes/cert-manager-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca
spec:
  ca:
    secretName: internal-ca-secret
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: order-service-tls
  namespace: production
spec:
  secretName: order-service-tls
  duration: 2160h  # 90 days
  renewBefore: 360h  # renew 15 days before expiry
  issuerRef:
    name: internal-ca
    kind: ClusterIssuer
  subject:
    organizations:
      - myorg
  dnsNames:
    - order-service.production.svc.cluster.local
    - order-service-headless.production.svc.cluster.local
  usages:
    - server auth
    - client auth  # Required for mTLS

gRPC Server with mTLS

// pkg/tls/config.go
package tlsconfig

import (
	"crypto/tls"
	"crypto/x509"
	"fmt"
	"os"

	"google.golang.org/grpc/credentials"
)

func NewServerTLSCredentials(certFile, keyFile, caFile string) (credentials.TransportCredentials, error) {
	cert, err := tls.LoadX509KeyPair(certFile, keyFile)
	if err != nil {
		return nil, fmt.Errorf("load server keypair: %w", err)
	}

	caCert, err := os.ReadFile(caFile)
	if err != nil {
		return nil, fmt.Errorf("read CA cert: %w", err)
	}

	caPool := x509.NewCertPool()
	if !caPool.AppendCertsFromPEM(caCert) {
		return nil, fmt.Errorf("failed to add CA cert to pool")
	}

	tlsCfg := &tls.Config{
		Certificates: []tls.Certificate{cert},
		ClientCAs:    caPool,
		ClientAuth:   tls.RequireAndVerifyClientCert, // require client certificate
		MinVersion:   tls.VersionTLS13,
	}

	return credentials.NewTLS(tlsCfg), nil
}

func NewClientTLSCredentials(certFile, keyFile, caFile string) (credentials.TransportCredentials, error) {
	cert, err := tls.LoadX509KeyPair(certFile, keyFile)
	if err != nil {
		return nil, fmt.Errorf("load client keypair: %w", err)
	}

	caCert, err := os.ReadFile(caFile)
	if err != nil {
		return nil, fmt.Errorf("read CA cert: %w", err)
	}

	caPool := x509.NewCertPool()
	if !caPool.AppendCertsFromPEM(caCert) {
		return nil, fmt.Errorf("failed to add CA cert to pool")
	}

	tlsCfg := &tls.Config{
		Certificates: []tls.Certificate{cert},
		RootCAs:      caPool,
		MinVersion:   tls.VersionTLS13,
	}

	return credentials.NewTLS(tlsCfg), nil
}

Mounting TLS Secrets into a Pod

# Addition to deployment.yaml
spec:
  containers:
  - name: order-service
    env:
    - name: TLS_CERT_FILE
      value: /etc/tls/tls.crt
    - name: TLS_KEY_FILE
      value: /etc/tls/tls.key
    - name: TLS_CA_FILE
      value: /etc/tls/ca.crt
    volumeMounts:
    - name: tls-certs
      mountPath: /etc/tls
      readOnly: true
  volumes:
  - name: tls-certs
    secret:
      secretName: order-service-tls

9. Testing gRPC Services

Comprehensive gRPC testing in Go includes unit tests with mocks, integration tests with a real gRPC server, and contract tests.

Mocks via mockery and Server Testing

// internal/server/order_server_test.go
package server_test

import (
	"context"
	"net"
	"testing"

	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/mock"
	"github.com/stretchr/testify/require"
	"google.golang.org/grpc"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/credentials/insecure"
	"google.golang.org/grpc/status"

	orderv1 "github.com/myorg/order-service/gen/go/order/v1"
	"github.com/myorg/order-service/internal/mocks"
	"github.com/myorg/order-service/internal/server"
)

func setupTestServer(t *testing.T, mockRepo *mocks.OrderRepository) orderv1.OrderServiceClient {
	t.Helper()

	lis, err := net.Listen("tcp", "127.0.0.1:0")
	require.NoError(t, err)

	grpcSrv := grpc.NewServer()
	orderv1.RegisterOrderServiceServer(grpcSrv, server.NewOrderServer(mockRepo))

	go grpcSrv.Serve(lis)
	t.Cleanup(grpcSrv.Stop)

	conn, err := grpc.Dial(
		lis.Addr().String(),
		grpc.WithTransportCredentials(insecure.NewCredentials()),
	)
	require.NoError(t, err)
	t.Cleanup(func() { conn.Close() })

	return orderv1.NewOrderServiceClient(conn)
}

func TestCreateOrder_Success(t *testing.T) {
	mockRepo := mocks.NewOrderRepository(t)
	mockRepo.On("Create", mock.Anything, "customer-123", mock.Anything).
		Return(&orderv1.Order{
			Id:         "order-456",
			CustomerId: "customer-123",
			Status:     orderv1.OrderStatus_ORDER_STATUS_PENDING,
		}, nil)

	client := setupTestServer(t, mockRepo)

	resp, err := client.CreateOrder(context.Background(), &orderv1.CreateOrderRequest{
		CustomerId: "customer-123",
		Items: []*orderv1.OrderItem{
			{ProductId: "prod-1", Quantity: 2, Price: 99.99},
		},
	})

	require.NoError(t, err)
	assert.Equal(t, "order-456", resp.Order.Id)
	mockRepo.AssertExpectations(t)
}

func TestCreateOrder_ValidationError(t *testing.T) {
	mockRepo := mocks.NewOrderRepository(t)
	client := setupTestServer(t, mockRepo)

	_, err := client.CreateOrder(context.Background(), &orderv1.CreateOrderRequest{
		CustomerId: "", // empty customer_id
		Items:      []*orderv1.OrderItem{},
	})

	require.Error(t, err)
	st, ok := status.FromError(err)
	require.True(t, ok)
	assert.Equal(t, codes.InvalidArgument, st.Code())
	mockRepo.AssertNotCalled(t, "Create")
}

Tools for Manual Testing

The following tools are invaluable for manual testing of gRPC services:

  • grpcurl — curl for gRPC. Allows sending requests to a server with reflection enabled: grpcurl -plaintext localhost:50051 order.v1.OrderService/GetOrder
  • grpcui — a browser-based UI for gRPC, similar to Postman.
  • evans — an interactive REPL shell for gRPC with TLS and metadata support.
  • ghz — a load testing tool for gRPC: ghz --insecure --proto order.proto --call order.v1.OrderService.GetOrder -d '{"order_id":"123"}' localhost:50051

Contract Testing with protovalidate

// Using buf validate for proto contract validation
// Add to order.proto:
// import "buf/validate/validate.proto";
//
// message CreateOrderRequest {
//   string customer_id = 1 [(buf.validate.field).string.min_len = 1];
//   repeated OrderItem items = 2 [(buf.validate.field).repeated.min_items = 1];
// }

// Use an interceptor in the server:
func validationInterceptor() grpc.UnaryServerInterceptor {
	v, _ := protovalidate.New()
	return func(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
		if msg, ok := req.(proto.Message); ok {
			if err := v.Validate(msg); err != nil {
				return nil, status.Errorf(codes.InvalidArgument, "validation failed: %v", err)
			}
		}
		return handler(ctx, req)
	}
}

10. Conclusion: When gRPC Is the Right Choice and When REST API Is Better

After an in-depth exploration of gRPC for Go and Kubernetes, let's summarize: when should you invest in gRPC, and when is it simpler to stick with REST API?

gRPC is the right choice when:

  • Inter-service communication happens inside a Kubernetes cluster, where clients are other services rather than browsers.
  • Performance is critical: high-load systems with thousands of RPS per service will see a tangible benefit from binary serialization and HTTP/2.
  • A strict API contract is needed: .proto files as a single source of truth prevent breaking changes.
  • Streaming is required: real-time updates, event streaming, bidirectional communication.
  • A polyglot environment: .proto files generate clients for Go, Python, Java, Rust — without duplicating code.
  • Out-of-the-box observability matters: gRPC status codes, standard Prometheus metrics, native OpenTelemetry integration.

REST API remains the better choice when:

  • The API is publicly accessible and consumed by browsers or third-party clients.
  • The team is small and the overhead of proto files and code generation is not justified.
  • Simple debugging via curl without specialized tools is needed.
  • The service integrates with legacy systems or third-party APIs that expect JSON/HTTP.
  • Webhook support or simple CRUD operations without high load are required.

The optimal strategy for most teams in 2026: gRPC for internal inter-service communication (east-west traffic) and REST API or GraphQL for external public APIs (north-south traffic). For gRPC services in Kubernetes, be sure to configure client-side or proxy-based load balancing, full observability with Prometheus and OpenTelemetry, and mTLS via cert-manager — these three components transform a basic gRPC integration into a reliable, production-ready system.

CI/CD pipelines for Go gRPC services should also be adapted: add stages for linting proto files with buf, breaking change checks (buf breaking), code generation, and running integration tests with a real gRPC server — this ensures contract stability in a rapidly growing microservice architecture.

Technologies

Tags

Ruslan Ismailov

Senior Web / Backend Developer. Senior web/backend developer with 9 years of experience. Stack: PHP, Laravel, PostgreSQL, Redis, Docker, Kubernetes, REST, microservices, CI/CD. More about me →