Skip to main content

Run several installations on one server

This page is for parties that host Linksight for several organizations. It shows how to run a separate Linksight installation per organization on one Docker server, so you don't need a virtual machine per organization.

An installation here means everything one docker-compose.yml starts: the Linksight Supervisor for Docker (the supervisor), plus the data station and Analysis Hub that it deploys. Each organization gets its own installation, registered under its own organization on the Governance Hub.

Installations on one server can't share the same host ports. You can solve that in one of two ways:

SetupHow partners reach each data stationPlatformsSupervisor version
Separate portsEach data station uses its own ports, for example 8443/9090 and 8444/9091.Linux and Windows ServerAny
Shared ports behind an SNI proxyAll data stations use 8443/9090. A proxy forwards each connection by the data station's DNS name.Linux with Docker Engine 25 or laterv1.1.10 or later

Separate ports is the simpler setup, but every installation adds two open data station ports to the server and to your firewall. Use the SNI proxy to keep the number of open data station ports to two, 8443 and 9090, however many installations the server runs. Partners then only need to allow those two ports as well.

This page does not cover Kubernetes deployments, or running more than one data station under a single supervisor.

Every hosted organization has to trust the server operator

The supervisor controls Docker through the Docker engine socket. Anyone with administrator access to the server or to Docker can read the data of every installation on it. Make sure each hosted organization agrees to this before you install it next to others.

Before you start​

Check the following before you add a second installation:

  • Server resources. Every installation needs the minimum specifications of a single installation, see Deploy using Docker or Deploy on Windows Server. A data station uses all CPU cores during a computation, so installations that compute at the same time slow each other down.
  • Registration rights. Each installation is registered under its own hosted organization, by an IT admin or Organization admin of that organization. If you don't have one of these roles in a hosted organization, that organization's administrator registers the installation, or invites you with one of those roles.
  • One working directory and one data folder per installation. Start every new installation in a new, empty working directory with its own FULLDATAPATH. Give every working directory a unique name, for example ~/linksight/org-a and ~/linksight/org-b. Docker Compose names the project after this directory, so two installations in directories with the same name replace each other's supervisor. Names that only differ in upper and lower case count as the same name.
  • Permissions on Linux. The commands on this page run docker without sudo. Add sudo if your user is not in the docker group, see Deploy using Docker.
Never share or copy a data folder

The supervisor stores its ID in the data folder and names its containers, network and volumes after that ID. If two installations use the same data folder, or one starts from a copy of another's folder, both supervisors take over the same containers.

If an installation already runs on the server​

Treat the installation that already runs as the first installation, and keep its working directory and data folder. Some of its settings change as well: the memory limit, and for Option 2 the settings in step 6. To apply a change:

  1. Edit the .env file in its working directory.
  2. In the same directory, run docker compose up -d (on Windows Server: docker-compose up -d).

The supervisor then restarts and recreates the data station and the Analysis Hub with the new settings, which interrupts that installation for a moment. If you change DATASTATION_HOST, for example from an IP address to a DNS name, the data station announces its new address to its partners when it starts.

Use Let's Encrypt for one installation at most​

Automatic certificate issuance needs public port 80 (for http-01) or 443 (for tls-alpn-01), and only one installation can use each port. Enable ANALYSISHUB_AUTOCERT_ENABLED in one installation at most. If a reverse proxy already uses ports 80 and 443, no installation can use it. Give the other installations their own certificates, or let the reverse proxy handle TLS.

Choose the settings for each installation​

The following settings must differ between installations. Keep the defaults for the first installation, and pick unused values for every next one:

SettingFirst installationNext installation
Working directory~/linksight/org-a~/linksight/org-b
FULLDATAPATH/home/john/linksight/org-a/data/home/john/linksight/org-b/data
ANALYSISHUB_FRONTEND_FQDNhttp://ah.example.org:8000http://ah.example.org:8001
ANALYSISHUB_PORT80008001
ANALYSISHUB_GRPC_PORT80808081
DATASTATION_PORT84438444
DATASTATION_GRPC_PORT90909091
NETWORK_SUBNETemptyempty

Keep the following in mind when you pick the values:

  • On Windows Server, use paths like C:\linksight\org-a and C:\linksight\org-a\data.
  • FULLDATAPATH is an absolute path. For a new installation, the folder is empty when the installation starts for the first time.
  • ANALYSISHUB_FRONTEND_FQDN must match ANALYSISHUB_PORT, or be the URL of the reverse proxy.
  • DATASTATION_PORT and DATASTATION_GRPC_PORT only differ in Option 1. In Option 2, every installation keeps 8443 and 9090.
  • Leave NETWORK_SUBNET empty, or give each installation a different subnet.
  • SUPERVISOR_GRPC_PORT does not need to differ: it is only used inside each installation's own Docker network.

Check which ports are already in use​

To see which ports the installations on the server already use, run:

docker ps --format "table {{.Names}}\t{{.Ports}}"

To see the ports that other programs use, run:

sudo ss -ltn

If an installation uses a port that is already taken, its containers don't start, and the supervisor logs that the port is already allocated.

Split the memory​

By default, each data station may use 75% of the server's memory. With several installations, decide how they share the server's memory, and set one of the following in every installation's .env:

  • SUPERVISOR_DATASTATION_MEMORY_HOSTFRACTION: a fraction of the server's memory, for example 0.35.
  • SUPERVISOR_DATASTATION_MEMORY_LIMIT: an absolute size, for example 6GiB. This setting takes precedence over the fraction.

These settings need supervisor v1.1.8 or later.

You can split the memory in one of two ways:

  • A fixed share per installation (recommended). Set the limits so that together they fit in the server's memory, and leave room for the Analysis Hubs and the operating system. For example 0.35 each for two installations. Installations can compute at the same time without affecting each other, but an installation can't use the memory that idle installations leave unused.
  • Overlapping limits. Set limits that together exceed the server's memory, for example 0.6 each for two installations. An installation can then use more memory while the others are idle. If several installations compute at the same time and the server runs out of memory, the operating system stops a data station: its computation fails, and the data station restarts. Choose this only if the installations rarely compute at the same time.

For an installation that already runs, apply the change as described in If an installation already runs on the server.

Option 1: Separate ports​

In this setup, every installation publishes its data station and Analysis Hub on its own host ports. The data station registers its host and ports with the Linksight platform, so partners automatically connect to the right ports. All installations can share the same DATASTATION_HOST.

The following diagram shows two installations on their own ports:

Two installations on one server: partners reach each data station on its own ports through the firewall, and each organization's users reach its Analysis Hub on its own ports

Install each installation​

For every new installation, do the following. The commands use org-b as an example.

  1. Create its working directory and go to it:

    mkdir -p ~/linksight/org-b
    cd ~/linksight/org-b

  2. Download the files and configure .env as described in Deploy using Docker or Deploy on Windows Server. Use the values you chose in Choose the settings for each installation.

  3. Start the installation, as described on the same page.

  4. Register the installation under its hosted organization:

    1. Open the installation's Analysis Hub at its ANALYSISHUB_FRONTEND_FQDN, and click Connect.
    2. On the Governance Hub, check that the hosted organization is selected, not your own organization, and click Accept.

    If you don't have the IT admin or Organization admin role in the hosted organization, send the Analysis Hub URL to that organization's administrator, who does these two steps. See Register nodes for the full procedure.

Open the firewall for each installation​

Each installation after the first needs its own ports opened. The following rules are needed per installation, next to the rules for a single installation. Server stands for the IP address of the server that runs the installations.

DirectionProtocolPortSourceDestinationPurpose
INGRESSTCPDATASTATION_PORT (e.g. 8444)Any (0.0.0.0/0)ServermTLS HTTP server of this data station
INGRESSTCPDATASTATION_GRPC_PORT (e.g. 9091)Any (0.0.0.0/0)ServermTLS gRPC server of this data station
INGRESSTCPANALYSISHUB_PORT (e.g. 8001)The organization's usersServerAnalysis Hub frontend
INGRESSTCPANALYSISHUB_GRPC_PORT (e.g. 8081)The organization's usersServerAnalysis Hub API, for the Linksight Python Interface
Restrict to partner data stations

As for a single installation, we recommend limiting the source of the data station ports to the data stations of partner organizations.

On Windows Server, also allow the ports in the server firewall, for example:

netsh advfirewall firewall add rule name="Allow Linksight mTLS org-b" dir=in action=allow protocol=TCP localport=8444
netsh advfirewall firewall add rule name="Allow Linksight gRPC org-b" dir=in action=allow protocol=TCP localport=9091
netsh advfirewall firewall add rule name="Allow Linksight Analysis Hub org-b" dir=in action=allow protocol=TCP localport=8001,8081

Keep the following in mind for the partner connections:

  • Outgoing traffic. Each data station connects to its partners on the ports that the partners registered. If your firewall only allows outgoing traffic to ports 8443 and 9090, partners that are hosted on other ports are unreachable. In the same way, partners whose firewall only allows outgoing 8443 and 9090 must also allow the ports of your other installations.
  • Organizations on the same server. If two hosted organizations take part in the same data collaboration, their data stations connect to each other through the server's public address. Your perimeter firewall must then support NAT loopback (also called hairpinning). Otherwise, use DNS names as DATASTATION_HOST. In each installation, set CONTAINERS_EXTRA_HOSTS to map the names of all data stations on the server to the server's internal address, for example ds-org-a.example.org:10.0.0.5,ds-org-b.example.org:10.0.0.5.
  • One IP address for all data stations. Partners see the same IP address for every data station on the server. A partner can't allow one hosted data station by IP address without also allowing the others.

Check that the installations work​

Check the following after you start and register an installation:

  • docker ps shows a set of containers per installation (supervisor, data station and Analysis Hub). The containers of each installation share a name suffix, and each set uses its own host ports.
  • Each Analysis Hub loads at its own ANALYSISHUB_FRONTEND_FQDN.
  • Once an installation is part of a data collaboration, its data station shows as online on the Network Status page of the Governance Hub.

Option 2: Shared ports behind an SNI proxy​

In this setup, every data station is reachable on the server's ports 8443 and 9090, under its own DNS name. A proxy in front of the data stations reads the DNS name that a partner connects to from the start of the TLS connection (the Server Name Indication, or SNI), and forwards the connection to that data station. The proxy does not decrypt anything: the mutual TLS connection still runs end to end between the two data stations.

This setup is available on Linux only, and needs Docker Engine 25 or later.

The setup works as follows:

  • Each data station has its own DNS name (its DATASTATION_HOST), and all names point to the server's public IP address. Partners send that name as SNI when they connect.
  • The data stations don't publish ports on the server (SUPERVISOR_DATASTATION_PUBLISH_PORTS=false). Only the proxy listens on 8443 and 9090.
  • The proxy and the data stations share an internal Docker network. This page names it linksight-edge, but you can choose another name. Each data station joins it under its DATASTATION_HOST (SUPERVISOR_DATASTATION_EXTRA_NETWORK=linksight-edge), so the proxy can forward to that name.
  • The Analysis Hubs are not behind this proxy. Each one keeps its own published ports, as in Option 1.

The following diagram shows the proxy in front of the data stations:

All data stations on ports 8443 and 9090 behind HAProxy, which routes each connection by SNI over the shared Docker network; each Analysis Hub keeps its own published ports for its organization's users

Proxy requirements​

You can use any proxy that meets the following requirements:

  • It runs as a container on the server, attached to the linksight-edge Docker network, and publishes ports 8443 and 9090 through a second, normal Docker network. linksight-edge is internal, and Docker doesn't publish ports from an internal network.
  • It routes each connection by the SNI to the data station with that DATASTATION_HOST, on the same port.
  • It passes the TLS connection through unchanged. No SSL offloading, no TLS termination and no PROXY protocol towards the data stations: they break the mutual TLS between data stations. See SSL Offloading, Deep Packet Inspection and Web Application Firewalls.
  • It looks up a data station's name again when the address changes, because a data station gets a new address on the Docker network every time it is recreated.
  • It starts even when a data station isn't running yet.
  • Its idle timeouts for client and server connections are 60 seconds or more. Data stations ping each other every 30 seconds on active connections.

The steps below use HAProxy as an example.

Example: Set up the proxy and the installations with HAProxy​

The example pulls the HAProxy image from Docker Hub, which is not part of the required connections: allow registry-1.docker.io, auth.docker.io and production.cloudflare.docker.com during installation, or use a mirror.

  1. Create a DNS A record for every data station, all pointing to the server's public IP address. For example ds-org-a.example.org and ds-org-b.example.org.

    caution

    Use DNS names, not IP addresses. Partners don't send an SNI when they connect to an IP address, so the proxy can't tell the data stations apart.

  2. Create the shared Docker network as an internal network:

    docker network create --internal linksight-edge

    The data stations also join this network. Docker picks a container's default route by network name, so a normal network named linksight-edge would take over the data stations' outgoing traffic from their own network. An internal network has no route out, so it can't. The supervisor logs a warning if the network is not internal.

  3. Create a working directory for the proxy:

    mkdir -p ~/linksight/sni-proxy
    cd ~/linksight/sni-proxy

  4. In that directory, create the following two files. In haproxy.cfg, replace ds-org-a.example.org and ds-org-b.example.org with the DNS names of your data stations. For more than two data stations, see Add or remove a data station.

    docker-compose.yml
    services:
      haproxy:
        image: haproxy:3.2
        restart: always
        ports:
          - "8443:8443"
          - "9090:9090"
        volumes:
          # Mount the directory, not the file: editors that save by replacing
          # haproxy.cfg would otherwise leave the container on the old version.
          - ./:/usr/local/etc/haproxy:ro
        networks:
          # default: for the published ports; linksight-edge: to the data stations
          - default
          - linksight-edge

    networks:
      linksight-edge:
        external: true
    haproxy.cfg
    global
      log stdout format raw local0

    defaults
      mode tcp
      log global
      log-format "%ci:%cp %ft %b/%s sni=%[capture.req.hdr(0)] %Tt %B %ts"
      timeout connect 5s
      timeout client 5m
      timeout server 5m

    # Docker's DNS, so a backend follows its data station when it is recreated.
    resolvers docker
      nameserver dns 127.0.0.11:53
      hold valid 10s

    frontend datastation_https
      bind :8443
      tcp-request inspect-delay 5s
      tcp-request content capture req.ssl_sni len 253
      tcp-request content accept if { req.ssl_hello_type 1 }
      use_backend ds1_https if { req.ssl_sni -i ds-org-a.example.org }
      use_backend ds2_https if { req.ssl_sni -i ds-org-b.example.org }

    frontend datastation_grpc
      bind :9090
      tcp-request inspect-delay 5s
      tcp-request content capture req.ssl_sni len 253
      tcp-request content accept if { req.ssl_hello_type 1 }
      use_backend ds1_grpc if { req.ssl_sni -i ds-org-a.example.org }
      use_backend ds2_grpc if { req.ssl_sni -i ds-org-b.example.org }

    # Connections whose SNI matches no data station are closed.

    backend ds1_https
      server ds1 ds-org-a.example.org:8443 resolvers docker init-addr none
    backend ds1_grpc
      server ds1 ds-org-a.example.org:9090 resolvers docker init-addr none

    backend ds2_https
      server ds2 ds-org-b.example.org:8443 resolvers docker init-addr none
    backend ds2_grpc
      server ds2 ds-org-b.example.org:9090 resolvers docker init-addr none

  5. Start the proxy:

    docker compose up -d

    If an installation on the server still publishes 8443 and 9090, the proxy can't start. Move that installation behind the proxy first, with the settings from step 6, as described in If an installation already runs on the server.

  6. Install and register each installation as in Option 1, with the following values in its .env:

    SettingValueNotes
    DATASTATION_HOSTds-org-a.example.orgThis installation's own DNS name.
    DATASTATION_PORT8443The same for every installation.
    DATASTATION_GRPC_PORT9090The same for every installation.
    SUPERVISOR_DATASTATION_PUBLISH_PORTSfalse
    SUPERVISOR_DATASTATION_EXTRA_NETWORKlinksight-edge

    The other settings in Choose the settings for each installation still differ per installation: the working directory, FULLDATAPATH, the Analysis Hub settings and the memory limits.

    Keep the SUPERVISOR_VERSION variable at its default value, so the supervisor is recent enough.

  7. In the firewall, open ports 8443 and 9090 to the server for partners, as for a single installation. No extra data station ports are needed per installation. Each installation still needs its Analysis Hub ports, as in the Option 1 firewall rules.

Check that the proxy forwards the connections​

To test the routing before partners connect, run the following from a machine outside the server, for every data station and for both ports 8443 and 9090:

openssl s_client -connect <server public IP>:8443 -servername ds-org-a.example.org </dev/null

The TLS handshake fails, because the test has no client certificate. That is expected: what matters is the proxy log. On the server, run:

cd ~/linksight/sni-proxy
docker compose logs -f haproxy

A line like datastation_https ds1_https/ds1 sni=ds-org-a.example.org means the connection reached that data station. A line with <NOSRV> means the SNI didn't match any data station in haproxy.cfg. <NOSRV> lines without an SNI come from scanners on the internet, and are harmless.

Once the installations are registered and part of a data collaboration, their data stations show as online on the Network Status page of the Governance Hub.

Add or remove a data station​

Each data station has four entries in haproxy.cfg: a use_backend line in the datastation_https and datastation_grpc frontends, and two backends.

To add a data station, for example ds-org-c.example.org:

  1. Create its DNS record, as in step 1.

  2. Add its four entries to haproxy.cfg:

    # in frontend datastation_https
      use_backend ds3_https if { req.ssl_sni -i ds-org-c.example.org }
    # in frontend datastation_grpc
      use_backend ds3_grpc if { req.ssl_sni -i ds-org-c.example.org }

    backend ds3_https
      server ds3 ds-org-c.example.org:8443 resolvers docker init-addr none
    backend ds3_grpc
      server ds3 ds-org-c.example.org:9090 resolvers docker init-addr none

  3. Check the configuration and reload the proxy, as described below.

  4. Install the installation with the values from step 6.

To remove a data station, delete its four entries from haproxy.cfg, then check the configuration and reload the proxy.

To check the configuration and reload the proxy, run:

cd ~/linksight/sni-proxy
docker compose exec haproxy haproxy -c -f /usr/local/etc/haproxy/haproxy.cfg
docker compose kill -s HUP haproxy

The reload keeps open connections. If the check reports an error, fix it before you reload: a reload with an invalid configuration keeps the old configuration running.

Things to know about this setup​

  • One IP address for all data stations. Partners see the same IP address for every data station on the server. If a partner allows data stations by IP address, one rule covers all of them.
  • Organizations on the same server. Data stations on the same server reach each other directly over linksight-edge, so no NAT loopback is needed.
  • Start order. Create the linksight-edge network before you start an installation. If the network is missing, the supervisor doesn't start the data station, logs an error, and tries again every 20 seconds.

Serve the Analysis Hubs behind one reverse proxy​

Instead of giving every Analysis Hub its own port, you can serve all of them on port 443 behind one reverse proxy, with one FQDN per organization. The reverse proxy terminates TLS for the Analysis Hubs, and forwards each FQDN to that installation's ANALYSISHUB_PORT on the server.

For each installation, configure the following:

  • Set ANALYSISHUB_FRONTEND_FQDN to the URL the organization's users open, for example https://analysis.org-a.example.org.
  • Keep a unique ANALYSISHUB_PORT per installation, and forward the FQDN to it on the server's address, for example https://analysis.org-a.example.org → http://<server address>:8000.
  • Leave ANALYSISHUB_FRONTEND_TLS_CERTFILE, ANALYSISHUB_FRONTEND_TLS_KEYFILE and ANALYSISHUB_AUTOCERT_ENABLED empty: the reverse proxy handles TLS.
  • The supervisor still publishes ANALYSISHUB_PORT on all of the server's addresses. Allow that port in the firewall only for the reverse proxy, so users can't bypass it over plain HTTP.
  • The Linksight Python Interface connects to ANALYSISHUB_GRPC_PORT directly. Keep that port reachable for the organization's users.

The data station ports must bypass this reverse proxy. Partners connect to the data stations directly (Option 1) or through the SNI proxy (Option 2), never through a proxy that terminates TLS.

Updates and restarts​

Each installation updates and restarts on its own. An update or restart started from an Analysis Hub only affects that installation's containers. The other installations keep running.

Troubleshooting​

SymptomCauseSolution
Containers don't start and the supervisor logs that the port is already allocatedAnother installation or program uses the same port. In Option 2: an installation still publishes 8443/9090, or its supervisor is older than v1.1.10.Pick an unused port, see Check which ports are already in use. In Option 2, check the installation's .env, and check its supervisor version with docker compose logs supervisor | grep "lsd - ".
Starting an installation replaces another installation's supervisorBoth working directories have the same name, so Docker Compose treats them as one project.Run docker compose down in one of the directories, rename that directory to a unique name, and start both installations again with docker compose up -d.
Two installations stop or restart each other's containersThey use the same data folder, or one started from a copy of the other's data folder.Give each installation its own, initially empty, FULLDATAPATH.
The second supervisor can't create its networkBoth installations set the same NETWORK_SUBNET.Leave NETWORK_SUBNET empty, or give each installation a different subnet.
Data stations are stopped during computationsThe memory limits of all installations together exceed the server's memory, and several installations computed at the same time.Lower the limits so that they fit in the server's memory, see Split the memory.
(Option 2) The data station isn't started, and the supervisor logs SUPERVISOR_DATASTATION_EXTRA_NETWORK 'linksight-edge' with not foundThe linksight-edge network doesn't exist.Run docker network create --internal linksight-edge. The data station starts within 20 seconds.
(Option 2) The data station isn't started, and the supervisor logs Container cannot be connected to network endpointsDocker Engine is older than version 25.Upgrade Docker Engine to version 25 or later, or use Option 1.
(Option 2) The data station isn't started, and the supervisor logs invalid SUPERVISOR_DATASTATION_PUBLISH_PORTSThe value is not true or false.Set exactly true or false, and apply the change.
(Option 2) Partners can't connect, and the proxy logs <NOSRV> with their data station's nameThe name is missing or misspelled in haproxy.cfg.Fix the name, then check the configuration and reload the proxy.
(Option 2) A change to haproxy.cfg has no effectThe configuration has an error, so the reload kept the old configuration.Run the configuration check, fix the reported error, and reload again.

To see the supervisor's log, run docker compose logs supervisor in the installation's working directory (on Windows Server: docker-compose logs supervisor).