Resources consumption really changes between different clusters and XpoLog. Nowadays, it is possible to define very large machines or multiple small machines for applications deployment – which is preferred?
From our experience, the “rule” is medium-large machines that will balance between performance results and not too many cluster nodes to be managed. There is no fixed formula as XpoLog is used for a wide variety of tasks. The details below will some important key notes based on our experience.
Generally, for low daily volume of data (<5 GB/Day) a laptop or standard machine is sufficient, but when it comes production deployments, there are a several recommendations to consider.
Supported Operating Systems (standalone installation)
The following operating systems are supported for standalone installation:
Linux Kernel versions 2.6.x and later (x86 (only 64-bit is supported))
Windows Server 2008 R2, Windows Server 2012, Server 2012 R2, and Server 2016, Windows 8, 8.1 and 10 (x86 (only 64-bit is supported))
Mac OS X 10.11 and Mac OS 10.12 and 10.13
Supported Application Servers
The following are supported Application Servers for deployment of XpoLog as an application (requires JAVA 1.8):
Note: XpoLog is known to run on additional application servers. However, it was tested only on the above servers.
Memory and JAVA
Memory is an important resource - aggregations are memory consumers and many data structures are stored in it. Depends on your daily volume, consider a 32GB or 16 GB machines. Less than 8 GB is less effective and impacts performance (it is not recommended to use machines with more than 64 GB). Java 1.8 is mandatory , and while allocating memory to XpoLog (XpoLog lax file) ensure that the Min (Xms) and Max (Xmx) allocated memory are identical – this will prevent heap size from resizing in runtime. Always leave memory to the OS that XpoLog runs on:
In Linux OS- allocate approximately 25% of the available memory.
In Windows OS- allocate approximately 50% of the available memory.
In any case, do not allocated more than 32GB to XpoLog as crossing it mainly reduces memory management efficiency, CPUs usage and GC handling of large heap size.
Finally, pay attention to Swapping – swapping memory to disk significantly reduces performance. Try to lower swappiness as much as possible - this value controls the OS swapping policy.
The following browsers are supported:
Google Chrome (Version 50+) - recommended
Mozilla Firefox (Version 50+)
MS Internet Explorer (Version 11+)
MS Edge is no longer supported
The following are the default ports which should be opened:
30303 - HTTP client access to XpoLog web interface and communication between different XpoLog instances (can be modified if needed)
30443 - HTTPS client access to XpoLog web interface and communication between different XpoLog instances (can be modified if needed)
Syslog - TCP and/or UDP any port that is required to be used.
22 - In order to enable XpoLog to establish connections to remote machines over SSH (can be modified if needed)
25 - In order to enable XpoLog to use a SMTP server to send emails (can be modified if needed)
162 - In order to enable XpoLog to send SNMP traps (can be modified if needed)
7676 - In order to enable XpoLog to send JMS messages (can be modified if needed)
389 - In order to enable XpoLog to authenticate users against an Active Directory (can be modified if needed)
636 - In order to enable XpoLog to authenticate users against an Active Directory through a secured connection
JDBC - In case XpoLog is planned to connect to databases using JDBC drivers, it is mandatory to open the relevant ports:
Oracle - 1521 (can be modified if needed)
MSSQL - 1433 (can be modified if needed)
MySQL - 3306 (can be modified if needed)
DB2 - 50000 (can be modified if needed)
Postgres - 5432 (can be modified if needed)
135-139, 445 - Share and UNC access to remote servers
You should choose a modern processor with multiple cores. Standard machines running XpoLog utilize 4-8 cores. If you need to choose between faster CPUs or more cores, choose more cores.
Disks are important for XpoLog - XpoLog processing engines mostly perform write operations but not only, while XpoLog UI engines mostly perform read operations but not only. XpoLog does many bulk reads/writes and many disk seeks and as disks are usually the slowest subsystem in a server, they easily become the bottleneck of the XpoLog cluster. If you can afford SSDs, they are by far superior to any spinning media. If SSD is not an option, try to obtain the fastest disks possible (high-performance server disks, 15K RPM). Network Attached Storage (NAS) is optional but note that NAS is often slower and displays larger latency. RAID 0, 10, 01, 0+1 will give the best performance, while RAID 5 will offer the worst performance.
It is mandatory to perform tests on your set of data and configuration deployment to determine optimized performance since it may vary between different types of data and searches XpoLog performs.
A fast and reliable network is obviously important to performance in a distributed system. Modern data-center networking (1 GbE, 10 GbE) is sufficient for most clusters. From our experience it is recommended to avoid clusters that are distributed over multiple data centers / geographical locations
General Recommendations Standard
The following is hardware recommendation for up to 3 concurrent users, and < 10 GB of daily logs volume:
4 CPU cores (2.5-3 GHz per core)
Standard Linux or Windows 64-bit
8 GB RAM
The following is hardware recommendation for up to 5 concurrent users, and < 10 GB of daily logs volume: