System Requirements

Owned by Omry Koschitzky
Last updated: Jun 30, 2022 by Noga Aviram (Unlicensed)
4 min read

Overview

Resources consumption really changes between different clusters and XpoLog. Nowadays, it is possible to define very large machines or multiple small machines for applications deployment – which is preferred?

From our experience, the “rule” is medium-large machines that will balance between performance results and not too many cluster nodes to be managed.
There is no fixed formula as XpoLog is used for a wide variety of tasks. The details below will some important key notes based on our experience.

Generally, for low daily volume of data (<5 GB/Day) a laptop or standard machine is sufficient, but when it comes production deployments, there are a several recommendations to consider.

Supported Operating Systems (standalone installation)

The following operating systems are supported for standalone installation:

  • Linux Kernel versions 2.6.x and later (x86 (only 64-bit is supported))

  • Windows Server 2008 R2, Windows Server 2012, Server 2012 R2, and Server 2016, Windows 8, 8.1 and 10 (x86 (only 64-bit is supported))

  • Mac OS X 10.11 and Mac OS 10.12 and 10.13

Supported Application Servers

The following are supported Application Servers for deployment of XpoLog as an application (requires JAVA 1.8):

  • Tomcat 6+

  • JBoss 4+

  • WebSphere 6.1+

  • WebLogic 9+

Note:  XpoLog is known to run on additional application servers. However, it was tested only on the above servers.

Memory and JAVA

Memory is an important resource - aggregations are memory consumers and many data structures are stored in it. Depends on your daily volume, consider a 32GB or 16 GB machines. Less than 8 GB is less effective and impacts performance (it is not recommended to use machines with more than 64 GB). Java 1.8 is mandatory , and while allocating memory to XpoLog (XpoLog lax file) ensure that the Min (Xms) and Max (Xmx) allocated memory are identical – this will prevent heap size from resizing in runtime.
Always leave memory to the OS that XpoLog runs on:

  • In Linux OS- allocate approximately 25% of the available memory.

  • In Windows OS- allocate approximately 50% of the available memory.

In any case, do not allocated more than 32GB to XpoLog as crossing it mainly reduces memory management efficiency, CPUs usage and GC handling of large heap size.

Finally, pay attention to Swapping – swapping memory to disk significantly reduces performance. Try to lower swappiness as much as possible - this value controls the OS swapping policy.

Supported Browsers

The following browsers are supported:

  • Google Chrome (Version 50+) - recommended

  • Mozilla Firefox (Version 50+)

  • MS Internet Explorer (Version 11+)

  • MS Edge is no longer supported

Required Ports

The following are the default ports which should be opened:

  • 30303 - HTTP client access to XpoLog web interface and communication between different XpoLog instances (can be modified if needed)

  • 30443 - HTTPS client access to XpoLog web interface and communication between different XpoLog instances (can be modified if needed)

  • Syslog - TCP and/or UDP any port that is required to be used.

  • 22 - In order to enable XpoLog to establish connections to remote machines over SSH (can be modified if needed)

  • 25 - In order to enable XpoLog to use a SMTP server to send emails (can be modified if needed)

  • Alerting:

    • 162 - In order to enable XpoLog to send SNMP traps (can be modified if needed)

    • 7676 - In order to enable XpoLog to send JMS messages (can be modified if needed)

  • 389 - In order to enable XpoLog to authenticate users against an Active Directory (can be modified if needed)

  • 636 - In order to enable XpoLog to authenticate users against an Active Directory through a secured connection 

  • JDBC - In case XpoLog is planned to connect to databases using JDBC drivers, it is mandatory to open the relevant ports:

    • Oracle - 1521 (can be modified if needed)

    • MSSQL - 1433 (can be modified if needed)

    • MySQL - 3306 (can be modified if needed)

    • DB2 - 50000 (can be modified if needed)

    • Postgres - 5432 (can be modified if needed)

  • Windows Only:

    • 135-139, 445 - Share and UNC access to remote servers 

Hardware Recommendations

CPUs

You should choose a modern processor with multiple cores. Standard machines running XpoLog utilize 4-8 cores.
If you need to choose between faster CPUs or more cores, choose more cores.

Disks

Disks are important for XpoLog - XpoLog processing engines mostly perform write operations but not only, while XpoLog UI engines mostly perform read operations but not only.
XpoLog does many bulk reads/writes and many disk seeks and as disks are usually the slowest subsystem in a server, they easily become the bottleneck of the XpoLog cluster.
If you can afford SSDs, they are by far superior to any spinning media. If SSD is not an option, try to obtain the fastest disks possible (high-performance server disks, 15K RPM).
Network Attached Storage (NAS) is optional but note that NAS is often slower and displays larger latency. RAID 0, 10, 01, 0+1 will give the best performance, while RAID 5 will offer the worst performance.

It is mandatory to perform tests on your set of data and configuration deployment to determine optimized performance since it may vary between different types of data and searches XpoLog performs.

Note: In Linux deployments, it is mandatory to allocate a high number of open files and processes to the user that runs XpoLog - for more information please see Post Installation Recommendations or contact support@xplg.com for more details.

Network

A fast and reliable network is obviously important to performance in a distributed system. Modern data-center networking (1 GbE, 10 GbE) is sufficient for most clusters.
From our experience it is recommended to avoid clusters that are distributed over multiple data centers / geographical locations

General Recommendations Standard

The following is hardware recommendation for up to 3 concurrent users, and < 10 GB of daily logs volume:

  • 4 CPU cores (2.5-3 GHz per core)

  • Standard Linux or Windows 64-bit

  • 8 GB RAM

The following is hardware recommendation for up to 5 concurrent users, and < 10 GB of daily logs volume:

  • 8 CPU cores (3 GHz per core)

  • Standard Linux or Windows 64-bit OS

  • 16 GB RAM

Cluster

The following is hardware recommendation for up to 25 concurrent users, and higher daily logs volume:
(contact support@xplg.com to determine if a cluster is needed. More information about clustering can be found here. See installations details at HA Cluster Installation (multi-machines))

  •   Processor node

    •  <25 GB/Day: 

      • 1 Node: 8 CPU cores, 16 GB memory, 64-bit OS

    • <100 GB/Day:

      • 2 Nodes: 8 CPU cores, 16 GB memory, 64-bit OS

    • <500 GB/Day:

      • 3 Nodes: 8 CPU cores, 32 GB memory, 64-bit OS

    • >500 GB/Day - contact us to discuss.

  • UI node*

    • <25 GB/Day:

      • 4 CPU cores, 16 GB memory, 64-bit OS

    • <100 GB/Day:

      • 8 CPU cores, 24 GB memory, 64-bit OS

    • <500 GB/Day: 

      • 8 CPU cores, 32 GB memory, 64-bit OS

    • >500 GB/Day - contact us to discuss.

    * Number of UI nodes should be determined based on the expected number of concurrent user and desired redundancy/balancing.