Adding Data to XPLG

You can add log data (agent-less) to XPLG from any of the following sources:

• Local – The log is on the same machine as XpoLog Center or can be directly accessed from a remote server (in the case of Windows) or a mounted directory (in the case of UNIX).

  • Windows Network – The log is on a remote Windows machine.

• Over SSH – The log is on a remote UNIX machine (with SSH connecting protocol). 

• Hadoop HDFS – The log is in a Hadoop environment.

• AWS S3 Bucket - The log is in Amazon Web Services S3 storage.

• Database – Connection is made to a database to import a table from the database as a log.

• Windows Events  – The log is a Windows Events log.

• Remote XpoLog – Connection is made with http(s) between XpoLog instances.

• Local XpoLog – Collection of data from an already existing log in a local XpoLog instance (usually used with data filtering to collect a desired part of an existing log).

Note: Windows Network and Windows Events logs are only available when XpoLog Center is installed on a Windows machine.

A single log can include many files of the same type or rotated files. For example, a messages log can include the files messages.1, messages.2, and more. It is recommended to capture multiple files of the same log type as one log, using a generic path. This can be done by appending a name pattern to the log path. For example, if a log type contains multiple files that follow a similar name pattern such as mylog.log.20110101_1, mylog.log.20110101_2, mylog.log.20110101_3, …, mylog.log.20110101_n, you can enter the generic log path: mylog.log.{date,yyyy-MM-dd}_{string}.

Examples:

• /var/log/messages{string}

• log.1.log, log.2.log, and log.3.log can be represented by the name pattern log.{string}.log.

• myapp.25-8-2009-22:30:00, myapp.26-8-2009-22:30:00, and myapp.27-8-2009-22:30:00 can be represented by the name pattern myapp.{date,dd-MM-yyyy-HH:mm:ss}.

As logs are written in free format, XpoLog uses its built-in mechanism to guess the structure of the incoming log, also called a pattern. For example, Xpolog guesses which field is the Date, the Time, and more, parses the data, and then adds the log records in bulk to XpoLog. Users can further normalize or tune the results of parsing. For example, if the results of parsing data is a log with columns Date, Text2, and Text3, the user can open the log under Folders and Logs, and click the log to open it and tune the parsing results. For example, Text2 column heading can be changed to Host, and Text3 column heading to Server Name. Next time you add a log of a similar type to the one that you edited, it uses the last structure that you applied to the new log.  

Once a log is brought into XpoLog it is indexed and undergoes Analytics. You can also run searches on it, and perform all log actions on it.

To add a log to XpoLog:

1. Open the XpoLog tab, click Add Data
   OR
   Open the XpoLog tab, and in the XpoLog Manager > Left Navigation Panel > Data select Add Data
   OR
   In the  XpoLog Manager > Left Navigation Panel > Data > Manager select a folder and click Add Log
   OR
   In the  XpoLog Manager > Left Navigation Panel > Folders And Logs, click Add Log
   OR
   In the XpoLog homepage, in the right pane under Get Data, click Add Log
   OR 
   In the XpoLog homepage, in the right pane under Get Data, click Remote Servers
   OR 
   In the XpoLog homepage, in the left pane under Add Data by System, choose the relevant system

2. Select Data Source.

3. Fill in the connectivity information for the selected log type, as follows:

   1. For a Local log, complete the information as described in Adding a Local Log.

   2. For a Windows Network log, complete the information as described in Adding a Windows Network Log.

   3. For an Over SSH log, complete the information as described in Adding an Over SSH Log.

   4. For a Hadoop HDFS log, complete the information as described in Adding a Hadoop HDFS Log.

   5. For a AWS S3 Bucket log, complete the information as described in Adding an AWS S3 Bucket Log.

   6. For a Database log, complete the information as described in Adding a Database Log.

   7. For a Windows Events log, complete the information as described in Adding a Windows Events Log.

   8. For a Remote XpoLog log, complete the information as described in Adding a Remote XpoLog Log.

   9. For a Local XpoLog log, complete the information as described in Adding a Local XpoLog Log.   

4. Set the log path to the directory and press Add Log.

5. At this point it is possible to save the log, or pressing on Active Log Pattern List  to review the Data Pattern Configuration.

6. Set the basic information of the log being added to XpoLog, including log name, location, and tagging to applications  (see Setting Log General Information).

7. To add additional log, select Save & Add Another or select Save & Close to finish.

8. Once the log is added to XpoLog, validate the log configuration (see Verifying Added Log Configuration).