Amazon ELB

Background

Built in Amazon Elastic load balancer dashboards and consoles to gain deep-level insights on your global content delivery network all across your cloud AMIs at once.

The application is aimed to run on ELB access logs stored in one of your S3 buckets. It is required to enable the logging of the Load Balancer to generate the logs and configure the path where log data will be written to). 

Steps

  1. Add Log Data In XpoLog, When adding a log to XpoLog you can now set a Log Type (logtype). For Amazon ELB set the following logtypes:

    1. AWS

    2. ELB

    3. access

  2. The ELB access logs are placed in a S3 bucket that was configured, usually with the name structure: {string}.log
    It is required to configure a S3 account for XpoLog to connect and read the required data from the S3 bucket.

  3. Once all required information is set click next and edit the log pattern, this step is crucial to the accuracy and deployment of the Amazon ELB App. Use the following conversion table to build the XpoLog pattern out of the access log format.

 

Example

The Amazon ELB access default log format is: 

timestamp elb client:port backend:port request_processing_time backend_processing_time response_processing_time elb_status_code backend_status_code received_bytes sent_bytes "request" "user_agent" ssl_cipher ssl_protocol

 

In XpoLog this pattern will be translated into:

{date:Date,yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'} {text:ELB-Name,ftype=elb} {geoip:Client IP,ftype=remoteip}:{number:Client Port,ftype=remoteport} {geoip:BackEnd IP,ftype=backendip}:{number:BackEnd Port,ftype=backendport} {number:Request Processing Time (Sec),ftype=processrequestsecs}{expression:Request Processing Time (Ms),ftype=processrequestmilli,Request Processing Time (Sec) * 1000} {number:BackEnd Processing Time (Sec),ftype=backendprocessrequestsecs}{expression:BackEnd Processing Time (Ms),ftype=backendprocessrequestmilli,BackEnd Processing Time (Sec) * 1000} {number:Response Processing Time (Sec),ftype=responseprocessingtimesecs}{expression:Response Processing Time (Ms),ftype=responseprocessingtimemilli,Response Processing Time (Sec) * 1000} {text:ELB Status Code,ftype=elbstatuscode} {text:Response Status,ftype=respstatus} {number:Received Bytes,ftype=bytesreceived} {number:Bytes Sent,ftype=bytesent} "{choice:Method,ftype=reqmethod;,GET;POST} {url:URL,paramsFtype=querystring;ftype=requrl;paramsName=Query;,} {string:reqprotocol,ftype=reqprotocol;,}" "{text:User-agent,ftype=useragent}" {text:SSL Cipher,ftype=sslcipher} {text:SSL Protocol,ftype=sslprotocol}

 

For more information see below:

Amazon ELB Access Log Format Conversion Table

 

Format String

Description

XpoLog Pattern

XpoLog ftype

Format String

Description

XpoLog Pattern

XpoLog ftype

type

The type of request or connection. The possible values are as follows

  • http — HTTP

  • https — HTTP over SSL/TLS

  • h2 — HTTP/2 over SSL/TLS

  • ws — WebSockets

  • wss — WebSockets over SSL/TLS

{choice:Type,ftype=type;,http;https;h2;ws;wss}

type

timestamp

The time when the load balancer generated a response to the client, in ISO 8601 format. For WebSockets, this is the time when the connection is closed.

{date:Date,yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'}

 

elb

The resource ID of the load balancer. If you are parsing access log entries, note that resources IDs can contain forward slashes (/).

{text:ELB-Name,ftype=elb}

elb

client:port

The IP address and port of the requesting client.

{geoip:Client IP,ftype=remoteip}:{number:Client Port,ftype=remoteport} 

remoteip remoteport

backend:port

The IP address and port of the target that processed this request.

If the client didn't send a full request, the load balancer can't dispatch the request to a target, and this value is set to -.

If the target is a Lambda function, this value is set to -.

If the request is blocked by AWS WAF, this value is set to - and the value of elb_status_code is set to 403.

{geoip:BackEnd IP,ftype=backendip}:{number:BackEnd Port,ftype=backendport}

backendip

backendport

request_processing_time

The total time elapsed (in seconds, with millisecond precision) from the time the load balancer received the request until the time it sent it to a target.

This value is set to -1 if the load balancer can't dispatch the request to a target. This can happen if the target closes the connection before the idle timeout or if the client sends a malformed request.

This value can also be set to -1 if the registered target does not respond before the idle timeout.

{number:Request Processing Time (Sec),ftype=processrequestsecs}{expression:Request Processing Time (Ms),ftype=processrequestmilli,Request Processing Time (Sec) * 1000}

processrequestsecs

processrequestmilli

backend_processing_time

[HTTP listener] The total time elapsed, in seconds, from the time the load balancer sent the request to a registered instance until the instance started to send the response headers.

[TCP listener] The total time elapsed, in seconds, for the load balancer to successfully establish a connection to a registered instance.

This value is set to -1 if the load balancer can't dispatch the request to a registered instance. This can happen if the registered instance closes the connection before the idle timeout or if the client sends a malformed request.

This value can also be set to -1 if the registered instance does not respond before the idle timeout.

{number:BackEnd Processing Time (Sec),ftype=backendprocessrequestsecs}{expression:BackEnd Processing Time (Ms),ftype=backendprocessrequestmilli,BackEnd Processing Time (Sec) * 1000} 

backendprocessrequestsecs

backendprocessrequestmilli

target_processing_time

The total time elapsed (in seconds, with millisecond precision) from the time the load balancer sent the request to a target until the target started to send the response headers.

This value is set to -1 if the load balancer can't dispatch the request to a target. This can happen if the target closes the connection before the idle timeout or if the client sends a malformed request.

This value can also be set to -1 if the registered target does not respond before the idle timeout.

{number:Target Processing Time (Sec),ftype=targetresponseprocessingtimesecs}{expression:Target Response Processing Time (Ms),ftype=targetresponseprocessingtimemilli,Target Response Processing Time (Sec) * 1000}

 targetresponseprocessingtimesecs

targetresponseprocessingtimemilli

response_processing_time

The total time elapsed (in seconds, with millisecond precision) from the time the load balancer received the response header from the target until it started to send the response to the client. This includes both the queuing time at the load balancer and the connection acquisition time from the load balancer to the client.

This value is set to -1 if the load balancer can't send the request to a target. This can happen if the target closes the connection before the idle timeout or if the client sends a malformed request.

{number:Response Processing Time (Sec),ftype=responseprocessingtimesecs}{expression:Response Processing Time (Ms),ftype=responseprocessingtimemilli,Response Processing Time (Sec) * 1000} 

responseprocessingtimesecs

responseprocessingtimemilli

elb_status_code

The status code of the response from the load balancer.

 {text:ELB Status Code,ftype=elbstatuscode}

elbstatuscode

backend_status_code

The status code of the response from the registered instance.

{text:Response Status,ftype=respstatus}

respstatus

target_status_code

The status code of the response from the target. This value is recorded only if a connection was established to the target and the target sent a response. Otherwise, it is set to -.

{text:Target Status Code,ftype=respstatus}

respstatus

received_bytes

The size of the request, in bytes, received from the client (requester). For HTTP requests, this includes the headers. For WebSockets, this is the total number of bytes received from the client on the connection.

{number:Received Bytes,ftype=bytesreceived} 

bytesreceived

sent_bytes

The size of the response, in bytes, sent to the client (requester). For HTTP requests, this includes the headers. For WebSockets, this is the total number of bytes sent to the client on the connection.

 {number:Bytes Sent,ftype=bytesent}

bytesent

"request"

The request line from the client, enclosed in double quotes and logged using the following format: HTTP method + protocol://host:port/uri + HTTP version.

"{choice:Method,ftype=reqmethod;,GET;POST;HEAD} {url:URL,paramsFtype=querystring;ftype=requrl;paramsName=Query;,} {text:Request Protocol,ftype=reqprotocol;,}" 

reqmethod querystring requrl reqprotocol

"user_agent"

A User-Agent string that identifies the client that originated the request, enclosed in double quotes. The string consists of one or more product identifiers, product[/version]. If the string is longer than 8 KB, it is truncated.

"{text:User-agent,ftype=useragent}" 

useragent

ssl_cipher

[HTTPS listener] The SSL cipher. This value is recorded only if the incoming connection was established after a successful negotiation. Otherwise, it is set to -.

{text:SSL Cipher,ftype=sslcipher}

sslcipher

ssl_protocol

[HTTPS listener] The SSL protocol. This value is recorded only if the incoming connection was established after a successful negotiation. Otherwise, it is set to -.

{text:SSL Protocol,ftype=sslprotocol}

ssl-protocol

target_group_arn

The Amazon Resource Name (ARN) of the target group.

{text:Target Group arn,ftype=targetgrouparn}

targetgrouparn

"trace_id"

The contents of the X-Amzn-Trace-Id header, enclosed in double quotes.

"{text:Trace ID,ftype=traceid}"

traceid

"domain_name"

[HTTPS listener] The SNI domain provided by the client during the TLS handshake, enclosed in double quotes. This value is set to - if the client doesn't support SNI or the domain doesn't match a certificate and the default certificate is presented to the client.

"{text:Domain,ftype=domain}" 

domain

"chosen_cert_arn"

[HTTPS listener] The ARN of the certificate presented to the client, enclosed in double quotes. This value is set to session-reused if the session is reused.

"{text:Chosen Cert arn,ftype=chosencertarn}"

chosencertarn

matched_rule_priority

The priority value of the rule that matched the request. If a rule matched, this is a value from 1 to 50,000. If no rule matched and the default action was taken, this value is set to 0. If an error occurs during rules evaluation, it is set to -1. For any other error, it is set to -.

{text:Matched Rule Priority,ftype=matchedrulepriority}

matchedrulepriority

request_creation_time

The time when the load balancer received the request from the client, in ISO 8601 format.

{date:Request Creation Time,yyyy-MM-dd'T'HH:mm:ss z}

 

"actions_executed"

The actions taken when processing the request, enclosed in double quotes. This value is a comma-separated list that can include the following possible values: wafwaf-failedauthenticateredirectfixed-response, and forward. If no action was taken, such as for a malformed request, this value is set to -.

"{text:Actions Executed,ftype=actionsexecuted}"

actionsexecuted

"redirect_url"

The URL of the redirect target for the location header of the HTTP response, enclosed in double quotes. If no redirect actions were taken, this value is set to -.

"{text:Redirect URL,ftype=redirecturl}"

redirecturl}

"error_reason"

The reason code, enclosed in double quotes. If the request to the Lambda function succeeded, this value is set to -. If the request failed, this is one of the error codes described in Error Reason Codes. If the target is not a Lambda function, this value is set to -.

"{text:Error Reason,ftype=errorreason}"

errorreason