Deploying a Scalable XpoLog Instance on Amazon ECS. NLB with static public IP

A Step-by-Step Guide to Using an NLB with a Static IP and EFS for External Configuration

This guide details how to build a robust, scalable XpoLog logging endpoint on AWS ECS. By using an NLB with a static Elastic IP, you provide a stable, unchanging address for your on-premise log shippers. By using EFS for configuration, you create a flexible and persistent setup that is easy to manage and update.

1. Create New VPC:

Step 1: Navigate to the VPC Console

Open the AWS Management Console.
In the search bar at the top, type VPC and select it from the services.
Ensure you are in your desired AWS Region (e.g., Stockholm (eu-north-1)) by checking the top-right corner.

Step 2: Launch the VPC Creation Wizard

From the VPC dashboard, click the orange "Create VPC" button.
On the creation screen, make sure to select the "VPC and more" option on the left. This option builds the entire network stack for you.

Step 3: Configure Your VPC Settings

Now, you'll configure the network layout. The wizard provides a helpful preview diagram on the right that updates as you make changes.

Name tag auto-generation: Give your VPC a descriptive name, like xpolog-vpc. This will automatically name all the related resources.
IPv4 CIDR block: You can leave the default 10.0.0.0/16 unless you have a specific networking requirement.
Number of Availability Zones (AZs): Choose 2 for high availability. This is the standard for production workloads.
Number of public subnets: Set this to 2 (one per AZ). These will be used for your NLB.
Number of private subnets: Set this to 2 (one per AZ). These will be used for your secure ECS tasks.
NAT gateways: Select "1 per AZ". This allows tasks in your private subnets to access the internet (e.g., to pull container images) while remaining secure.
VPC endpoints: Select "None" for now to keep things simple.
DNS options: Keep the defaults enabled.

Step 4: Create and Verify

Review the diagram and configuration details at the bottom.
Click the "Create VPC" button.
The process will take a few minutes to provision all the resources (VPC, subnets, an Internet Gateway, NAT Gateways with Elastic IPs, and Route Tables).

Once the creation is complete, you will have a new, production-ready VPC. You can now go to the "Your VPCs" and "Subnets" sections to find the VPC ID, Public Subnet IDs, and Private Subnet IDs needed for your deployment scripts.

2) Reserve Elastic IPs (for NLB)

Why: NLB supports static IPs — one Elastic IP per public subnet used by the NLB.

EC2 → Network & Security → Elastic IPs → Allocate Elastic IP address → Allocate twice (you need 2 EIPs, one per AZ).
Record: EIP-AZ1, EIP-AZ2.

You already also paid for a NAT EIP via the VPC wizard; that’s separate. The two above are only for the NLB.

3) Security Groups

Classic NLBs did not support SGs.
But since 2020, NLBs in VPCs with awsvpc mode can attach Security Groups.
sg-ecs-tasks-xpolog: attached to ECS tasks ENIs.
sg-efs-xpolog: attached to EFS mount targets.
Create a security group for the NLB. Allow inbound TCP 30303 from 0.0.0.0/0. Outbound: all.”
Mention that traffic filtering happens on both the NLB SG and the ECS Task SG.

3.1 ECS Tasks SG

EC2 → Network & Security → Security Groups → Create security group
- Name: sg-ecs-tasks-xpolog
- VPC: xpolog-vpc
- Inbound rules → Add rule:
- So what do you need to put in the inbound rules?
  - Go to VPC → Subnets in the AWS console.
  - Find the two public subnets where your NLB is placed.
    - Example:
      - PublicSubnet1: 10.0.0.0/24
      - PublicSubnet2: 10.0.1.0/24
  - Back in your Security Group creation screen, add inbound rules:
    - Type: Custom TCP
    - Port: 30303
    - Source: each public subnet CIDR (one rule per subnet)
  - Type: Custom TCP, Port: 30303, Source: the two public subnet CIDR blocks that host the
    Why: NLB has no SG; its traffic originates from its subnet IPs. Opening from those CIDRs allows only NLB→tasks on 30303.
    (If you don’t want to calculate CIDRs, you can temporarily open from 0.0.0.0/0 for quick test and tighten later.)
- Outbound: allow all (default).
  Create SG. Record its ID.

Inbound rule for ECS tasks SG:

Type: Custom TCP, Port: 30303
Source: NLB security group (recommended)
(You can still mention subnet CIDRs as an alternative, but SG reference is cleaner.)

3.2 EFS SG

EC2 → Network & Security → Security Groups → Create security group
- Name: sg-efs-xpolog
- VPC: xpolog-vpc
- Inbound:
  - Type: NFS, Port 2049, Source: sg-ecs-tasks-xpolog (reference by SG ID).
    Why: only ECS tasks may mount EFS.
- Outbound: allow all (default).
  Create SG. Record its ID.

4) EFS (file system + mount targets + access point)

Why: Your task definition mounts persistent storage from EFS and runs as UID:GID = 1000:1000.

Create File System
EFS → Create file system
- Name: xpolog-efs
- VPC: xpolog-vpc
  Click Customize:
- Network: for each of your two private subnets, create/confirm Mount targets and set the Security group to sg-efs-xpolog.
- Throughput/Performance: General Purpose / Bursting (default is fine to start).
  Create.

Create:

Create Access Point
EFS → your file system → Access points → Create access point
- Name: xpolog-ap
- Root directory path: / (works with your JSON’s rootDirectory: "/")
- POSIX user: blank
- Root directory creation permissions: 0770 (Owner UID 0, GID 0)
  Create.
  Record: File system ID (fs-...) and Access Point ID (fsap-...).

5) CloudWatch Logs group

Why: Your containers log to /ecs/xpolog.

CloudWatch → Logs → Log groups → Create log group
- Name: /ecs/xpolog
- Retention: e.g., 30 days. Create.

6) IAM Roles (Execution & Task)

6.1 Execution Role (pull images, write logs, read secret)

IAM → Roles → Create role
- Trusted entity type: AWS service → Elastic Container Service → Elastic Container Service Task
- Permissions policies: add AmazonECSTaskExecutionRolePolicy
- Name: ecsTaskExecutionRole → Create.

6.2 Task Role (app runtime)

IAM → Roles → Create role
- Trusted entity type: AWS service → Elastic Container Service → Elastic Container Service Task
- Permissions: AmazonElasticFileSystemClientFullAccess, AmazonElasticFileSystemFullAccess
- Name: XpologTaskRole → Create.

7) Target Group (for NLB)

Why: Fargate uses target type = IP.

EC2 → Load Balancing → Target groups → Create target group
- Target type: IP
- Name: xpolog-tg
- Protocol: TCP, Port: 30303
- VPC: xpolog-vpc
- Health checks:
  - Protocol: TCP (simple and works with your app’s / check too)
    Create. (Don’t register targets manually; ECS will do it.)

8) Network Load Balancer (assign static IPs)

EC2 → Load Balancers → Create load balancer → Network Load Balancer
- Name: xpolog-nlb
- Scheme: Internet-facing
- IP address type: IPv4
- Network mapping: select your two public subnets
  - For each selected subnet, choose Elastic IP address → Use an Elastic IP and pick the EIP you allocated in Step 2.
- Listeners: Add TCP : 30303 → Forward to (we will select TG after creation if the UI doesn’t allow now)
  Create NLB.
If the wizard didn’t attach the TG:
- Open the NLB → Listeners → select TCP:30303 → Edit → Forward to xpolog-tg → Save.

Record: NLB DNS name and note you now have static IPs (the EIPs) for the NLB.

Reminder: NLBs do NOT have Security Groups. We already allowed traffic on port 30303 to tasks from the NLB public subnet CIDRs in Step 3.1.

9) ECS: Task Definition (Fargate + EFS)

nano xpolog-task.json

{
    "family": "xpolog-task",
    "networkMode": "awsvpc",
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "1024",
    "memory": "8192",
    "executionRoleArn": "arn:aws:iam::655536767854:role/ecsTaskExecutionRole-xpolog",
    "taskRoleArn": "arn:aws:iam::655536767854:role/XpologTaskRole-xpolog",
    "volumes": [
        {
            "name": "xpolog-storage",
            "efsVolumeConfiguration": {
                "fileSystemId": "fs-025714e3014f58deb",
                "transitEncryption": "ENABLED",
                "authorizationConfig": {
                    "accessPointId": "fsap-04ed2423de0fa867e"
                }
            }
        }
    ],
    "containerDefinitions": [
        {
            "name": "xpolog",
            "image": "1200km/xplg:7.Release-9787",
            "essential": true,
            "cpu": 1024,
            "memory": 8192,
            "entryPoint": [
                "/bin/sh",
                "-lc"
            ],
            "command": [
                "./start.sh"
            ],
            "portMappings": [
                {
                    "containerPort": 30303,
                    "protocol": "tcp"
                }
            ],
            "environment": [
                {
                    "name": "ram",
                    "value": "-Xmx4g"
                },
                {
                    "name": "profile",
                    "value": "-Dxpolog.uid.structure=master"
                },
                {
                    "name": "sharedstorage",
                    "value": "xpolog.home.root.path=/home/andrey/extconf2"
                }
            ],
            "healthCheck": {
                "command": [
                    "CMD-SHELL",
                    "bash -lc 'exec 3<>/dev/tcp/127.0.0.1/30303 || exit 1'"
                ],
                "interval": 30,
                "timeout": 10,
                "retries": 10,
                "startPeriod": 180
            },
            "mountPoints": [
                {
                    "sourceVolume": "xpolog-storage",
                    "containerPath": "/home/andrey/extconf2"
                }
            ],
            "dependsOn": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "/ecs/xpolog",
                    "awslogs-region": "eu-north-1",
                    "awslogs-stream-prefix": "xpolog"
                }
            }
        }
    ]
}

Register the task definition

aws ecs register-task-definition \
  --cli-input-json file://xpolog-task.json \
  --region eu-north-1

Validate

10) ECS: Cluster & Service

10.1 Cluster

ECS → Clusters → Create cluster

Name: xpolog-cluster → Create.

10.2 Service (Fargate)

ECS → Clusters → xpolog-cluster → Create → Service

Compute options: Fargate
Task definition: xpolog (latest revision)
Service name: xpolog-svc
Desired tasks: 1 (scale later)
Deployment options: rolling (default)

Networking

VPC: xpolog-vpc
Subnets: select the two private subnets
Security group: sg-ecs-tasks-xpolog
Public IP: DISABLED

Load balancing

Load balancer type: Network Load Balancer (NLB)
Select existing NLB: xpolog-nlb
Listener: TCP 30303
Target group: xpolog-tg
Container to load balance: xpolog : 30303
Create service.

The service-linked role AWSServiceRoleForECS will be auto-created if it does not exist.

Verifications:

$ curl -v --connect-timeout 5 http://13.61.167.126:30303/
curl -v --connect-timeout 5 http://13.51.137.251:30303/
curl -v --connect-timeout 5 http://xpolog-nlbAndrey-f8ed873f2ddfdc1c.elb.eu-north-1.amazonaws.com:30303/