Background: The Multi-Cluster DNS Challenge

Kubernetes clusters handle service name resolution internally via DNS (CoreDNS or kube-dns). However, Kubernetes does not natively support cross-cluster service discovery – a service name is only visible within its own cluster. In a hybrid or multi-cloud environment with multiple clusters, this becomes a challenge: how can services in different clusters find and communicate with each other by name? The Submariner project addresses multi-cluster networking, and its Lighthouse component specifically tackles cross-cluster DNS-based service discovery (Multicluster Service Discovery in OpenShift (Part 1)). Lighthouse provides a way for pods in any connected cluster to resolve DNS names of services from other clusters as if they were local, enabling seamless multi-cluster service connectivity (Multicluster Service Discovery in OpenShift (Part 1)) (Multicluster Service Discovery in OpenShift (Part 1)).

Lighthouse was created before a Kubernetes standard for multi-cluster services existed, but it aligns with the emerging Kubernetes Multi-Cluster Services (MCS) API. In fact, Lighthouse now implements the MCS API definitions (like ServiceExport and ServiceImport) defined by the Kubernetes community (Service Discovery :: Submariner k8s project documentation website). This means it uses standard resource types and a special DNS domain (clusterset.local) to make services discoverable across clusters (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita). When configured, a service exported from one cluster becomes accessible via a DNS name of the form ..svc.clusterset.local in other clusters (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita).

What is Submariner’s Lighthouse?

Submariner is an open-source project (a CNCF Sandbox project) that creates secure IP tunnels between Kubernetes clusters, essentially “flattening” the network so that pods and services in different clusters can communicate directly. Lighthouse is a component of Submariner that builds on this connectivity to provide multi-cluster service discovery via DNS (Multicluster Service Discovery in OpenShift (Part 1)). It works in conjunction with the network tunnels: Submariner ensures cross-cluster networking, and Lighthouse ensures that a service in cluster A can be reached by name from cluster B without manual configuration.

In simpler terms, Lighthouse lets you expose a Kubernetes Service from one cluster and access it by the same DNS name from other clusters. It accomplishes this by distributing service information between clusters and leveraging DNS resolution. Lighthouse’s solution is compatible with any Kubernetes CNI plugin or environment (cloud or on-prem), because it operates at the DNS level and is agnostic to the underlying network provider (GitHub - submariner-io/lighthouse: DNS service discovery across connected Kubernetes clusters.).

Some key features of Lighthouse include:

Lighthouse Architecture and Components

(Service Discovery :: Submariner k8s project documentation website) Lighthouse architecture: Lighthouse Agents in each cluster synchronize service metadata via a central broker, and each cluster runs a Lighthouse DNS (CoreDNS) server. The cluster DNS (CoreDNS) forwarder sends clusterset.local queries to the Lighthouse DNS server for resolution.

At a high level, Lighthouse’s architecture consists of a Broker, a Lighthouse Agent in each cluster, and a Lighthouse DNS server (based on CoreDNS) in each cluster (Service Discovery :: Submariner k8s project documentation website) (Multicluster Service Discovery in OpenShift (Part 1)). The Broker is a Kubernetes control plane (which can be a dedicated cluster or one of the participating clusters) that acts as a central exchange for service information. Each cluster that joins the Lighthouse cluster set does so by connecting to the Broker.

Let’s break down the components:

Lighthouse Agent (Controller)

The Lighthouse Agent runs in every member cluster. Its job is to watch for services being exported and share that information through the Broker to other clusters (Service Discovery :: Submariner k8s project documentation website). It also listens for services exported from other clusters and imports those into the local cluster’s records. In practice, Lighthouse leverages Kubernetes CRDs (Custom Resource Definitions) for this:

  • When you create a ServiceExport in a cluster, the Lighthouse Agent detects it. It then creates a corresponding ServiceImport resource (and related Endpoint data) in the Broker, representing that service in the multicluster context (Service Discovery :: Submariner k8s project documentation website). The ServiceImport contains information like the service’s cluster IP and which cluster it came from.
  • The Broker’s Kubernetes API acts as a hub. Agents from other clusters will see the new ServiceImport in the Broker and will import it into their local cluster (Service Discovery :: Submariner k8s project documentation website). Concretely, each cluster ends up with a copy of the ServiceImport for the service, which it can use to answer DNS queries.
  • This process happens for every exported service (and conversely, if a ServiceExport is removed, Lighthouse will withdraw that service’s info from the Broker and other clusters).

This architecture means service metadata is synchronized across clusters. The Lighthouse Agent essentially implements the controller logic of the multi-cluster service API. The result is that all clusters have a consistent view (via ServiceImport objects) of which services are available in the multi-cluster environment.

Lighthouse DNS Server (Multi-Cluster DNS)

The Lighthouse DNS server runs in each cluster as a CoreDNS instance with a custom plugin. It serves as the DNS authority for the special multi-cluster domain (by default, clusterset.local) (Multicluster Service Discovery in OpenShift (Part 1)). The cluster’s own DNS (typically CoreDNS) is configured to forward any queries for *.clusterset.local to this Lighthouse DNS service (Multicluster Service Discovery in OpenShift (Part 1)). In effect, Lighthouse inserts itself into the DNS resolution chain only for the multi-cluster service names.

Inside the Lighthouse DNS server, the custom lighthouse CoreDNS plugin uses the ServiceImports (and associated Endpoint data) that the Agent has distributed to build an in-memory cache of records (Service Discovery :: Submariner k8s project documentation website). When it receives a query, it will check this cache:

The workflow for a cross-cluster DNS query is as follows (Service Discovery :: Submariner k8s project documentation website):

  • A pod in Cluster A tries to resolve myservice.myns.svc.clusterset.local. This DNS request is intercepted by Cluster A’s CoreDNS.
  • CoreDNS sees that the query is for the clusterset.local domain, which it is not authoritative for. According to its configuration, it forwards the query to the Lighthouse DNS service running in Cluster A (Service Discovery :: Submariner k8s project documentation website).
  • The Lighthouse DNS server in Cluster A checks its cache of ServiceImports (which includes information about services exported from Cluster B, Cluster C, etc.). If myservice.myns has been exported by another cluster, Lighthouse finds the corresponding record (e.g., an IP from Cluster B) and returns it in a DNS response (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita). If it doesn’t find anything, it returns NXDOMAIN.
  • CoreDNS receives the answer from Lighthouse and sends the response back to the requesting pod, which can now connect to the service’s IP as if it were a local service.

This approach allows seamless cross-cluster service name resolution – pods continue to use DNS to connect to services, and whether the service is local or in another cluster is transparent to them. The entire lookup happens behind the scenes via Lighthouse’s coordination.

Lighthouse’s Embedded DNS vs. CoreDNS Plugin Approach

One notable design choice of Lighthouse is how it implements the DNS server. Instead of simply adding a plugin to the existing CoreDNS deployment in the cluster, Lighthouse runs its own CoreDNS-based DNS server as a separate component. In essence, Lighthouse embeds a CoreDNS process within its deployment (hence an “embedded DNS server”) rather than modifying the cluster’s CoreDNS directly (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita).

To clarify, the Lighthouse project actually includes a CoreDNS plugin named “lighthouse” which can handle multi-cluster DNS logic. Initially, one might expect to install this plugin into the cluster’s CoreDNS. However, Submariner’s Lighthouse opts for a different architecture: it packages CoreDNS (with the Lighthouse plugin) into a dedicated Deployment (submariner-lighthouse-coredns) and just configures the cluster’s DNS to forward queries to it (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita) (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita). This design has several advantages:

  • Isolation of Cross-Cluster DNS Logic: The multi-cluster domain resolution is kept separate from the cluster’s normal DNS service. This means any issues or bugs in Lighthouse’s DNS handling will not directly affect the regular cluster.local DNS resolution for in-cluster services (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita). The core Kubernetes DNS remains untouched except for one forwarding rule. This isolation improves reliability – if Lighthouse were to crash, your normal in-cluster DNS still works fine.
  • Ease of Installation & Compatibility: If Lighthouse had to be integrated as a plugin, one might need to custom-build a CoreDNS image with the plugin, or modify the CoreDNS ConfigMap extensively to load the external plugin. This can be challenging, especially on managed Kubernetes platforms (like OpenShift, GKE, or AKS) where the CoreDNS configuration is managed by the platform and custom plugins may not be supported. By running a separate DNS server, Lighthouse avoids incompatibilities – you simply add a stub-domain or forward entry to existing DNS. For example, on OpenShift, the Submariner operator adds a DNS forwarding rule for supercluster.local (or clusterset.local) via the DNS Operator (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita), and on AKS it updates the coredns-custom ConfigMap to achieve the same (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita). No custom CoreDNS binaries are required.
  • Future Flexibility (Standard API alignment): The multi-cluster services API is still evolving. Lighthouse initially implemented its own CRDs (like ServiceImport) before the Kubernetes SIG Multicluster standardized them (Multicluster Service Discovery in OpenShift (Part 1)). By keeping the Lighthouse DNS as a separate process under its control, the project can evolve its implementation easily – for instance, switching to upstream API versions or changing internal logic – without affecting the cluster’s core DNS. As long as the interface (forwarding queries to Lighthouse) remains the same, the internals can be updated to adapt to new versions of the MCS API (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita). If it were a built-in plugin, updating it could be more involved and tied to the cluster DNS version.
  • Works with Any Cluster DNS/CNI: This design is very flexible – it doesn’t even require that the cluster DNS be CoreDNS. Even if a cluster used another DNS implementation, as long as it can forward a domain to an external server, Lighthouse will work (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita). Similarly, it’s independent of the network plugin or service mesh; Lighthouse only assumes that clusters can route traffic to each other (which Submariner provides) and that DNS queries for the special domain can be forwarded. This aligns with the project’s goal to be CNI-agnostic and environment-agnostic (GitHub - submariner-io/lighthouse: DNS service discovery across connected Kubernetes clusters.).

In summary, the Lighthouse team embedded a CoreDNS server within Lighthouse to provide a drop-in DNS solution for multi-cluster services. This “batteries-included” DNS server approach means each cluster runs a Lighthouse DNS pod (CoreDNS with the lighthouse plugin) that cooperates with the cluster’s main DNS. The clusterset DNS zone is handled by Lighthouse’s pods, and the main CoreDNS just delegates those queries. This was a deliberate design choice to maximize compatibility and minimize disruption to existing clusters (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita) (Submariner Lighthouseにおける組み込み型CoreDNS設計の深掘り #kubernetes - Qiita).

Configuring and Using Lighthouse

Setting up Lighthouse is straightforward, especially with the Submariner toolkit. Typically, you would:

  1. Deploy a Broker and Join Clusters: One cluster (or a separate cluster) is designated as the broker. Using Submariner’s CLI subctl, you deploy the broker and then join each cluster to it. For example, subctl deploy-broker on the broker cluster, then on each member cluster subctl join --broker --clusterid --enable-discovery (in newer versions this flag might be --enable-service-discovery) to deploy Submariner components including Lighthouse (Multicluster Service Discovery in OpenShift (Part 2)). The Submariner operator on each cluster will install the Lighthouse Agent and Lighthouse CoreDNS pods automatically, as well as set up the CoreDNS forwarding for clusterset.local domain.
  2. Export a Service: By default, no service is exported (to avoid accidentally exposing everything). To export a service, you can use kubectl to create a ServiceExport object in the same namespace with the same name as the Service. For convenience, Submariner’s CLI offers subctl export service --namespace which creates the ServiceExport for you (Multicluster Service Discovery in OpenShift (Part 2)). For example, if you have a Deployment and Service called “nginx” in the “default” namespace of Cluster B, you would run subctl export service --namespace default nginx on Cluster B to export it.
  3. Discover from Other Clusters: Once exported, the service becomes discoverable to other clusters in the cluster set. From a pod in Cluster A (or even using kubectl run to create a temporary pod for testing), you should be able to resolve and reach nginx.default.svc.clusterset.local. For instance, you might exec into a test pod and run: 
$ kubectl exec -n default -ti  -- /bin/sh
   $ nslookup nginx.default.svc.clusterset.local
   Server:    10.96.0.10
   Address:   10.96.0.10#53

   Name:   nginx.default.svc.clusterset.local
   Address: 172.31.173.226

In this example, 172.31.173.226 is the Service’s cluster IP from the remote cluster that exported “nginx” (Multicluster Service Discovery in OpenShift (Part 2)). The DNS resolution is handled by Lighthouse behind the scenes. If you then attempt to actually curl nginx.default.svc.clusterset.local:8080 (assuming the service listens on port 8080), the request will be routed over the Submariner tunnels to the cluster that hosts the service, and you should get a response.

If the DNS lookup fails (you get an NXDOMAIN or no answer), it likely means either the service wasn’t exported or the local CoreDNS isn’t correctly forwarding the clusterset.local zone. Ensure that the ServiceExport exists and is in an Exported state (you can check by kubectl get serviceexport and seeing its status). Also verify that your cluster’s CoreDNS ConfigMap has an entry to forward clusterset.local to the Lighthouse DNS service IP (Troubleshooting :: Submariner k8s project documentation website) (Troubleshooting :: Submariner k8s project documentation website). This is usually done for you by Submariner, but in some cases (or custom setups) you might need to add a stanza like below to CoreDNS config:

clusterset.local:53 {
    forward . 
}

This tells CoreDNS to forward queries for the clusterset.local domain to Lighthouse’s CoreDNS. The IP to use is the ClusterIP of the submariner-lighthouse-coredns service (which you can find with kubectl -n submariner-operator get svc submariner-lighthouse-coredns (Multicluster Service Discovery in OpenShift (Part 2))).

Keep in mind that Lighthouse only answers for services that have been exported. For example, if you try to resolve a service that wasn’t exported, it will not be found. In a quick test, if you attempt to curl a service’s clusterset URL before exporting it, it will fail to resolve. After you create the ServiceExport (and give it a few seconds to propagate), the same curl should succeed (Multicluster Service Discovery in OpenShift (Part 2)) (Multicluster Service Discovery in OpenShift (Part 2)).

Verifying Multi-Cluster Service Discovery

You can verify that Lighthouse is working using standard DNS tools. As shown above, using nslookup or dig inside a cluster to query a ..svc.clusterset.local name will indicate whether the DNS is resolving to an IP. You should see an IP address in the ANSWER section which should match one of the service’s cluster IPs from a remote cluster. Here’s an example using a demo service called “nginx-hello” in namespace “demo” (from an AWS blog example):

$ kubectl exec -it client-hello -n demo -- nslookup nginx-hello.demo.svc.clusterset.local
Name:   nginx-hello.demo.svc.clusterset.local
Address: 172.20.80.119

As expected, we got an IP address for the multi-cluster service (Kubernetes Multi-Cluster Service Discovery using Open Source AWS Cloud Map MCS Controller | AWS Open Source Blog). If that service were available in multiple clusters, multiple IPs could be returned (and by default, Lighthouse would give the IP local to the querying cluster first, if the service exists locally).

Lighthouse also supports round-robin DNS responses when a service is backed by multiple clusters. For instance, if cluster1 and cluster2 both export a “nginx-demo” service, queries in cluster1 might sometimes get the local cluster1 IP and sometimes the remote cluster2 IP (if configured to do round-robin). This can be observed by making repeated queries – the returned address may cycle through the available endpoints in different clusters (Kubernetes Multi-Cluster Service Discovery using Open Source AWS Cloud Map MCS Controller | AWS Open Source Blog).

Recent Features: ClusterSet IPs and Headless Services

The Submariner community has been actively improving Lighthouse. Two interesting features in recent versions:

  • Cluster-Set Virtual IP: Lighthouse can allocate a virtual IP that is shared across all clusters for an exported service. This is sometimes called a ClusterSet IP. Normally, when you resolve a service via Lighthouse, you get the actual ClusterIP of one of the Kubernetes Services. With the ClusterSet IP feature enabled (by adding an annotation lighthouse.submariner.io/use-clusterset-ip on the ServiceExport, or enabling it for all services when deploying the broker), Lighthouse will instead assign a global virtual IP to represent that service (User Guide :: Submariner k8s project documentation website). All clusters will see the same virtual IP for the service, and Lighthouse DNS will return that IP for queries (User Guide :: Submariner k8s project documentation website). (This is useful in scenarios where you might want a consistent IP for a service across clusters, perhaps for external DNS or other integration.) Note that Submariner itself does not route traffic to this virtual IP – you’d need an external mechanism (e.g., routers or an external DNS) to handle it, so use this feature with that in mind (User Guide :: Submariner k8s project documentation website).
  • Headless Service & StatefulSet Support: Initially, Lighthouse focused on cluster IP services. Newer updates added support for Headless Services, particularly to enable stateful workloads across clusters. Kubernetes headless services (with no cluster IP) allow pods in a StatefulSet to have stable DNS names within a cluster (e.g., pod-0.myservice.myns.svc.cluster.local). Lighthouse extends this concept across clusters by incorporating the cluster ID into the DNS name. For example, if you have a StatefulSet “web” with a headless service “nginx-ss” and you export it from multiple clusters, you could address a specific pod in a specific cluster via a DNS name like web-0..nginx-ss.nginx-test.svc.clusterset.local (User Guide :: Submariner k8s project documentation website). This feature is advanced but demonstrates Lighthouse’s ability to handle even complex DNS scenarios in multi-cluster setups (ensuring uniqueness by cluster). It requires that the cluster IDs are DNS-compatible labels (User Guide :: Submariner k8s project documentation website) (since they appear in the DNS names).

For most users, the core functionality of Lighthouse – exporting services and resolving them across clusters – will be the main attraction. These additional features provide flexibility for more complex use cases.

Conclusion

Submariner’s Lighthouse component greatly simplifies multi-cluster Kubernetes deployments by providing a built-in, DNS-based service discovery mechanism. By exporting a service from one cluster, you can make it available to all connected clusters via the familiar Kubernetes DNS naming conventions (with a clusterset.local twist). Under the hood, Lighthouse handles distribution of service endpoints and integrates with CoreDNS, so that your applications don’t need any special logic – they can just use DNS and Kubernetes Services as they normally would.

We’ve seen how Lighthouse’s architecture (with Lighthouse Agents and an embedded CoreDNS server per cluster) makes it robust and easy to adopt without altering existing cluster DNS servers. We also discussed how to configure and use Lighthouse, from setting up the broker and agents to exporting services and verifying cross-cluster connectivity.

If you are implementing multi-cluster Kubernetes for high availability, disaster recovery, or hybrid cloud workloads, Lighthouse is a powerful ally. It works in tandem with Submariner’s network connectivity to truly blur the lines between clusters when it comes to service access. Developers and DevOps teams can deploy multi-cluster services with minimal changes to their workflows.

Further Resources: The Submariner project’s official documentation and GitHub have more details. The [Submariner.io Service Discovery docs】 (Service Discovery :: Submariner k8s project documentation website) provide a deep dive into the architecture, and the project is open source on GitHub (submariner-io/lighthouse). The Kubernetes MCS (Multi-Cluster Services) API is also worth reading up on, as Lighthouse’s design aligns with it. With Lighthouse in place, you can turn a collection of Kubernetes clusters into a unified, service-discovery-friendly ClusterSet – making multi-cluster Kubernetes feel a lot more like a single extended cluster. (Multicluster Service Discovery in OpenShift (Part 1)) (Multicluster Service Discovery in OpenShift (Part 1))