Advanced Task

Background

This article addresses two scenarios:

Sync Data Performance
- In cases where large data sets or high-paced data streams need to be synchronized, Flux supports additional options that use multi-threaded data synchronization to significantly improve performance.
Sync Data from Isolated Environments
- When the main Flux instance or cluster, which syncs data to the centralized repository, does not have direct access to all Flux agents, Flux supports a “grid structure.” In this structure, proxy Flux instances with access to remote environments can sync data over HTTPS connections from the Flux agents in these isolated environments.

Performance Parameters

The performance parameters are configured per <SyncLogsNode>.

Parameter	Description	Default

Parameter	Description	Default
`multiReaders`	Indicates whether to use parallel sync flow.	`false`
`numberOfReaders`	Defines how many parallel readers to use.	`-1`
`maxQueueSize`	Specifies how much data to allow in the queue at once.	`52428800`
`dataBlocksSize`	Defines the block size for data transfer.	`1048576`

Example:

<SyncLogsConfiguration>
  <SyncLogsRepository repositoryDirectory="/mnt/FluxRepo" timeToKeep="30" />
  <SyncLogsNode repositoryDirectory="data-server-1" maxQueueSize="5000000000" dataBlocksSize="40000000" readDataThroughput="80000000" multiReaders="true" numberOfReaders="10">
    <Remote>
      <Account classKey="xpolog.eye.media.remote.RemoteXplgAccount" name="data-server-1"/>
    </Remote>
    <SyncLogsDirectory syncDirectory="/opt/data/live" directoriesToInclude="" directoriesToExclude="" filesToInclude="*.log" filesToExclude="" subdirsScanLevel="">
    </SyncLogsDirectory>
  </SyncLogsNode>
</SyncLogsConfiguration>

Isolated Environments

In isolated environments, use Flux instance(s) as a proxy to synchronize logs from remote environments. The main Flux cluster should only have access to the proxy Flux instance(s), while these proxy instances should have access to the Flux agents within their respective environments.

The following example synchronizes logs from the directory /opt/data/ on the flux-agent-1 and flux-agent-2 machines. However, the main Flux instance/cluster only communicates with flux-proxy-1 and flux-proxy-2 and not directly with the source agents.

<SyncLogsConfiguration>
  <SyncLogsRepository repositoryDirectory="/mnt/FluxRepo/" timeToKeep="30" />
  <Checksum interval="30m" algorithm="MD5" mailRecipients="ops@xplg.com" enabled="true" retryAttempts="3" />
  <ProxyRemote>
    <Account classKey="xpolog.eye.media.remote.RemoteXplgAccount" name="flux-proxy-1"/>
    <Account classKey="xpolog.eye.media.remote.RemoteXplgAccount" name="flux-proxy-2"/>
  </ProxyRemote>
  <SyncLogsNode repositoryDirectory="flux-agent-1">
    <Remote>
      <Account classKey="xpolog.eye.media.remote.RemoteXplgAccount" name="flux-agent1"/>
    </Remote>
    <SyncLogsDirectory syncDirectory="/opt/data/" directoriesToInclude="" directoriesToExclude="" filesToInclude="" filesToExclude="" subdirsScanLevel="">
    </SyncLogsDirectory>
  </SyncLogsNode>
  <SyncLogsNode repositoryDirectory="flux-agent-2">
    <Remote>
      <Account classKey="xpolog.eye.media.remote.RemoteXplgAccount" name="flux-agent-1"/>
    </Remote>
    <SyncLogsDirectory syncDirectory="/opt/data/" directoriesToInclude="" directoriesToExclude="" filesToInclude="" filesToExclude="" subdirsScanLevel="">
    </SyncLogsDirectory>
  </SyncLogsNode>
</SyncLogsConfiguration>

If the <ProxyRemote> tag is defined outside of any specific <SyncLogsNode>, it will be applied globally to all nodes in the task. Alternatively, it can be defined within an individual <SyncLogsNode>.

The following example synchronizes logs from the directory /opt/data/ on the flux-agent-1 machine using flux-proxy-1 and flux-proxy-2. Meanwhile, it directly syncs data from the directory /opt/data/ on the flux-agent-2 machine because the <ProxyRemote> tag is defined only inside the node for flux-agent-1:

<SyncLogsConfiguration>
  <SyncLogsRepository repositoryDirectory="/mnt/FluxRepo/" timeToKeep="30" />
  <Checksum interval="30m" algorithm="MD5" mailRecipients="ops@xplg.com" enabled="true" retryAttempts="3" />

  <SyncLogsNode repositoryDirectory="flux-agent-1">
    <ProxyRemote>
      <Account classKey="xpolog.eye.media.remote.RemoteXplgAccount" name="flux-proxy-1"/>
      <Account classKey="xpolog.eye.media.remote.RemoteXplgAccount" name="flux-proxy-2"/>
    </ProxyRemote>
    <Remote>
      <Account classKey="xpolog.eye.media.remote.RemoteXplgAccount" name="flux-agent1"/>
    </Remote>
    <SyncLogsDirectory syncDirectory="/opt/data/" directoriesToInclude="" directoriesToExclude="" filesToInclude="" filesToExclude="" subdirsScanLevel="">
    </SyncLogsDirectory>
  </SyncLogsNode>

  <SyncLogsNode repositoryDirectory="flux-agent-2">
    <Remote>
      <Account classKey="xpolog.eye.media.remote.RemoteXplgAccount" name="flux-agent-1"/>
    </Remote>
    <SyncLogsDirectory syncDirectory="/opt/data/" directoriesToInclude="" directoriesToExclude="" filesToInclude="" filesToExclude="" subdirsScanLevel="">
    </SyncLogsDirectory>
  </SyncLogsNode>
</SyncLogsConfiguration>