Amazon ELB
Background
Built in Amazon Elastic load balancer dashboards and consoles to gain deep-level insights on your global content delivery network all across your cloud AMIs at once.
The application is aimed to run on ELB access logs stored in one of your S3 buckets. It is required to enable the logging of the Load Balancer to generate the logs and configure the path where log data will be written to).
Steps
- Add Log Data In XpoLog, When adding a log to XpoLog you can now set a Log Type (logtype). For Amazon ELB set the following logtypes:
- AWS
- ELB
- access
- The ELB access logs are placed in a S3 bucket that was configured, usually with the name structure: {string}.log
It is required to configure a S3 account for XpoLog to connect and read the required data from the S3 bucket. - Once all required information is set click next and edit the log pattern, this step is crucial to the accuracy and deployment of the Amazon ELB App. Use the following conversion table to build the XpoLog pattern out of the access log format.
Example
The Amazon ELB access default log format is:
timestamp elb client:port backend:port request_processing_time backend_processing_time response_processing_time elb_status_code backend_status_code received_bytes sent_bytes "request" "user_agent" ssl_cipher ssl_protocol
In XpoLog this pattern will be translated into:
{date:Date,yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'} {text:ELB-Name,ftype=elb} {geoip:Client IP,ftype=remoteip}:{number:Client Port,ftype=remoteport} {geoip:BackEnd IP,ftype=backendip}:{number:BackEnd Port,ftype=backendport} {number:Request Processing Time (Sec),ftype=processrequestsecs}{expression:Request Processing Time (Ms),ftype=processrequestmilli,Request Processing Time (Sec) * 1000} {number:BackEnd Processing Time (Sec),ftype=backendprocessrequestsecs}{expression:BackEnd Processing Time (Ms),ftype=backendprocessrequestmilli,BackEnd Processing Time (Sec) * 1000} {number:Response Processing Time (Sec),ftype=responseprocessingtimesecs}{expression:Response Processing Time (Ms),ftype=responseprocessingtimemilli,Response Processing Time (Sec) * 1000} {text:ELB Status Code,ftype=elbstatuscode} {text:Response Status,ftype=respstatus} {number:Received Bytes,ftype=bytesreceived} {number:Bytes Sent,ftype=bytesent} "{choice:Method,ftype=reqmethod;,GET;POST} {url:URL,paramsFtype=querystring;ftype=requrl;paramsName=Query;,} {string:reqprotocol,ftype=reqprotocol;,}" "{text:User-agent,ftype=useragent}" {text:SSL Cipher,ftype=sslcipher} {text:SSL Protocol,ftype=sslprotocol}
For more information see below:
Amazon ELB Access Log Format Conversion Table
Format String | Description | XpoLog Pattern | XpoLog ftype |
|---|---|---|---|
| type | The type of request or connection. The possible values are as follows
| {choice:Type,ftype=type;,http;https;h2;ws;wss} | type |
timestamp | The time when the load balancer generated a response to the client, in ISO 8601 format. For WebSockets, this is the time when the connection is closed. | {date:Date,yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'} | |
elb | The resource ID of the load balancer. If you are parsing access log entries, note that resources IDs can contain forward slashes (/). | {text:ELB-Name,ftype=elb} | elb |
client:port | The IP address and port of the requesting client. | {geoip:Client IP,ftype=remoteip}:{number:Client Port,ftype=remoteport} | remoteip remoteport |
backend:port | The IP address and port of the target that processed this request. If the client didn't send a full request, the load balancer can't dispatch the request to a target, and this value is set to -. If the target is a Lambda function, this value is set to -. If the request is blocked by AWS WAF, this value is set to - and the value of elb_status_code is set to 403. | {geoip:BackEnd IP,ftype=backendip}:{number:BackEnd Port,ftype=backendport} | backendip backendport |
request_processing_time | The total time elapsed (in seconds, with millisecond precision) from the time the load balancer received the request until the time it sent it to a target. This value is set to -1 if the load balancer can't dispatch the request to a target. This can happen if the target closes the connection before the idle timeout or if the client sends a malformed request. This value can also be set to -1 if the registered target does not respond before the idle timeout. | {number:Request Processing Time (Sec),ftype=processrequestsecs}{expression:Request Processing Time (Ms),ftype=processrequestmilli,Request Processing Time (Sec) * 1000} | processrequestsecs processrequestmilli |
backend_processing_time | [HTTP listener] The total time elapsed, in seconds, from the time the load balancer sent the request to a registered instance until the instance started to send the response headers. [TCP listener] The total time elapsed, in seconds, for the load balancer to successfully establish a connection to a registered instance. This value is set to This value can also be set to | {number:BackEnd Processing Time (Sec),ftype=backendprocessrequestsecs}{expression:BackEnd Processing Time (Ms),ftype=backendprocessrequestmilli,BackEnd Processing Time (Sec) * 1000} | backendprocessrequestsecs backendprocessrequestmilli |
| target_processing_time | The total time elapsed (in seconds, with millisecond precision) from the time the load balancer sent the request to a target until the target started to send the response headers. This value is set to -1 if the load balancer can't dispatch the request to a target. This can happen if the target closes the connection before the idle timeout or if the client sends a malformed request. This value can also be set to -1 if the registered target does not respond before the idle timeout. | {number:Target Processing Time (Sec),ftype=targetresponseprocessingtimesecs}{expression:Target Response Processing Time (Ms),ftype=targetresponseprocessingtimemilli,Target Response Processing Time (Sec) * 1000} | targetresponseprocessingtimesecs targetresponseprocessingtimemilli |
response_processing_time | The total time elapsed (in seconds, with millisecond precision) from the time the load balancer received the response header from the target until it started to send the response to the client. This includes both the queuing time at the load balancer and the connection acquisition time from the load balancer to the client. This value is set to -1 if the load balancer can't send the request to a target. This can happen if the target closes the connection before the idle timeout or if the client sends a malformed request. | {number:Response Processing Time (Sec),ftype=responseprocessingtimesecs}{expression:Response Processing Time (Ms),ftype=responseprocessingtimemilli,Response Processing Time (Sec) * 1000} | responseprocessingtimesecs responseprocessingtimemilli |
elb_status_code | The status code of the response from the load balancer. | {text:ELB Status Code,ftype=elbstatuscode} | elbstatuscode |
backend_status_code | The status code of the response from the registered instance. | {text:Response Status,ftype=respstatus} | respstatus |
| target_status_code | The status code of the response from the target. This value is recorded only if a connection was established to the target and the target sent a response. Otherwise, it is set to -. | {text:Target Status Code,ftype=respstatus} | respstatus |
received_bytes | The size of the request, in bytes, received from the client (requester). For HTTP requests, this includes the headers. For WebSockets, this is the total number of bytes received from the client on the connection. | {number:Received Bytes,ftype=bytesreceived} | bytesreceived |
sent_bytes | The size of the response, in bytes, sent to the client (requester). For HTTP requests, this includes the headers. For WebSockets, this is the total number of bytes sent to the client on the connection. | {number:Bytes Sent,ftype=bytesent} | bytesent |
"request" | The request line from the client, enclosed in double quotes and logged using the following format: HTTP method + protocol://host:port/uri + HTTP version. | "{choice:Method,ftype=reqmethod;,GET;POST;HEAD} {url:URL,paramsFtype=querystring;ftype=requrl;paramsName=Query;,} {text:Request Protocol,ftype=reqprotocol;,}" | reqmethod querystring requrl reqprotocol |
"user_agent" | A User-Agent string that identifies the client that originated the request, enclosed in double quotes. The string consists of one or more product identifiers, product[/version]. If the string is longer than 8 KB, it is truncated. | "{text:User-agent,ftype=useragent}" | useragent |
ssl_cipher | [HTTPS listener] The SSL cipher. This value is recorded only if the incoming connection was established after a successful negotiation. Otherwise, it is set to -. | {text:SSL Cipher,ftype=sslcipher} | sslcipher |
ssl_protocol | [HTTPS listener] The SSL protocol. This value is recorded only if the incoming connection was established after a successful negotiation. Otherwise, it is set to -. | {text:SSL Protocol,ftype=sslprotocol} | ssl-protocol |
| target_group_arn | The Amazon Resource Name (ARN) of the target group. | {text:Target Group arn,ftype=targetgrouparn} | targetgrouparn |
| "trace_id" | The contents of the X-Amzn-Trace-Id header, enclosed in double quotes. | "{text:Trace ID,ftype=traceid}" | traceid |
| "domain_name" | [HTTPS listener] The SNI domain provided by the client during the TLS handshake, enclosed in double quotes. This value is set to - if the client doesn't support SNI or the domain doesn't match a certificate and the default certificate is presented to the client. | "{text:Domain,ftype=domain}" | domain |
| "chosen_cert_arn" | [HTTPS listener] The ARN of the certificate presented to the client, enclosed in double quotes. This value is set to session-reused if the session is reused. | "{text:Chosen Cert arn,ftype=chosencertarn}" | chosencertarn |
| matched_rule_priority | The priority value of the rule that matched the request. If a rule matched, this is a value from 1 to 50,000. If no rule matched and the default action was taken, this value is set to 0. If an error occurs during rules evaluation, it is set to -1. For any other error, it is set to -. | {text:Matched Rule Priority,ftype=matchedrulepriority} | matchedrulepriority |
| request_creation_time | The time when the load balancer received the request from the client, in ISO 8601 format. | {date:Request Creation Time,yyyy-MM-dd'T'HH:mm:ss z} | |
| "actions_executed" | The actions taken when processing the request, enclosed in double quotes. This value is a comma-separated list that can include the following possible values: waf, waf-failed, authenticate, redirect, fixed-response, and forward. If no action was taken, such as for a malformed request, this value is set to -. | "{text:Actions Executed,ftype=actionsexecuted}" | actionsexecuted |
| "redirect_url" | The URL of the redirect target for the location header of the HTTP response, enclosed in double quotes. If no redirect actions were taken, this value is set to -. | "{text:Redirect URL,ftype=redirecturl}" | redirecturl} |
| "error_reason" | The reason code, enclosed in double quotes. If the request to the Lambda function succeeded, this value is set to -. If the request failed, this is one of the error codes described in Error Reason Codes. If the target is not a Lambda function, this value is set to -. | "{text:Error Reason,ftype=errorreason}" | errorreason |