Amazon S3

Background

Built in Amazon S3 dashboards and consoles to gain deep-level insights on your Elastic S3 buckets access logs. DBAs, IT Admins, Sys Admins and DevOps – with rich premium visualizations like dashboards, widgets and consoles XpoLog S3 features.

The application is aimed to run on access logs of the S3 bucket itself, if enabled (click on the logging of the bucket to enable it and see the path where log data will be written to):

Steps



  1. Add Log Data In XpoLog, When adding a log to XpoLog you can now set a Log Type (logtype). For AWS S3 set the following logtypes:

    1. AWS

    2. S3

    3. access

  2. The S3 access log usually is placed in a 'logs' directory within the bucket if the logging is enabled. The files name structure: <DATE>-<UNIQUE_ID> - in XpoLog it should be represented as {date,yyyy-MM-dd-HH-mm-ss}-{string}
    It is required to configure a S3 account for XpoLog to connect and read the required data from the S3 bucket.

  3. Once all required information is set click next and edit the log pattern, this step is crucial to the accuracy and deployment of the AWS S3 App. Use the following conversion table to build the XpoLog pattern out of the access log format.



Example

The AWS S3 access log format is: 

Bucket Owner Bucket Time Remote IP Requester Request ID Operation Key Request-URI HTTP status Error Code Bytes Sent Object Size Total Time Turn-Around Time Referrer User-Agent Version Id



In XpoLog this pattern will be translated into:

{text:Bucket Owner - Canonical ID,ftype=usercanonicalID} {text:Bucket,ftype=bucket} [{date:Date,dd/MMM/yyyy:HH:mm:ss Z}] {geoip:Client IP,ftype=remoteip} {text:Requester - Canonical ID,ftype=requestercanonicalid} {text:Request ID,ftype=requestid} {text:Operation,ftype=operation} {text:key,ftype=key} "{choice:Method,ftype=reqmethod;,GET;POST;HEAD} {url:URL,paramsFtype=querystring;ftype=requrl;paramsName=Query;,} {string:reqprotocol,ftype=reqprotocol;,}" {number:ResponseStatus,ftype=respstatus} {text:Error Code,ftype=errorcode} {number:Bytes Sent,ftype=bytesent} {text:Object Size,ftype=objectsize} {number:ResponseTimeMilliSecs,ftype=processrequestmilli} {text:Turn Around Time,ftype=turnaroundtime} "{string:RefererQuery,ftype=refererquery;,}{regexp:Referer,ftype=referer;refName=RefererQuery,^([\w-]+://[^?]+|/[^?]+)}" "{string:User-Agent,ftype=useragent;,}" {text:Version ID,ftype=versionid}

for more information see below:



AWS S3 Access Log Format Conversion Table



Field Name

Description

XpoLog Pattern

XpoLog ftype

Field Name

Description

XpoLog Pattern

XpoLog ftype

Bucket Owner

The canonical user ID of the owner of the source bucket

{text:Bucket Owner - Canonical ID,ftype=usercanonicalID}

usercanonicalID

Bucket

The name of the bucket that the request was processed against. If the system receives a malformed request and cannot determine the bucket, the request will not appear in any server access log

 {text:Bucket,ftype=bucket}

bucket

Time

The time at which the request was received

 [{date:Time,dd/MMM/yyyy:HH:mm:ss z}]



Remote IP

The apparent Internet address of the requester. Intermediate proxies and firewalls might obscure the actual address of the machine making the request

 {geoip:Client IP,ftype=remoteip} 

remoteip

Requester

The canonical user ID of the requester, or the string "Anonymous" for unauthenticated requests. If the requester was an IAM user, this field will return the requester's IAM user name along with the AWS root account that the IAM user belongs to. This identifier is the same one used for access control purposes

{text:Requester - Canonical ID,ftype=requestercanonicalid}

requestercanonicalid

Request ID

The request ID is a string generated by Amazon S3 to uniquely identify each request

{text:Request ID,ftype=requestid}

requestid

Operation

The operation listed here is declared as SOAP.operationREST.HTTP_method.resource_type,WEBSITE.HTTP_method.resource_type, or BATCH.DELETE.OBJECT.

{text:Operation,ftype=operation}

operation

Key

The "key" part of the request, URL encoded, or "-" if the operation does not take a key parameter.

{text:key,ftype=key}

key

Request-URI

The Request-URI part of the HTTP request message.

"{choice:Method,ftype=reqmethod;,GET;POST;HEAD} {url:URL,paramsFtype=querystring;ftype=requrl;paramsName=Query;,} {string:reqprotocol,ftype=reqprotocol;,}"

method

querystring

requrl

reqprotocol

HTTP Status

The numeric HTTP status code of the GET portion of the copy operation

{number:ResponseStatus,ftype=respstatus}



respstatus

Error Code

The Amazon S3 Error Code, of the GET portion of the copy operation or "-" if no error occurred

{text:Error Code,ftype=errorcode}

errorcode

Bytes Sent

The number of response bytes sent, excluding HTTP protocol overhead, or "-" if zero

{number:Bytes Sent,ftype=bytesent}

bytesent

Object Size

The total size of the object in question

{text:Object Size,ftype=objectsize}

objectsize

Total Time

The number of milliseconds the request was in flight from the server's perspective. This value is measured from the time your request is received to the time that the last byte of the response is sent. Measurements made from the client's perspective might be longer due to network latency

{number:ResponseTimeMilliSecs,ftype=processrequestmilli}

processrequestmilli

Turn-Around Time

The number of milliseconds that Amazon S3 spent processing your request. This value is measured from the time the last byte of your request was received until the time the first byte of the response was sent

{text:Turn Around Time,ftype=turnaroundtime}

turnaroundtime

Referrer

The value of the HTTP Referrer header, if present. HTTP user-agents (e.g. browsers) typically set this header to the URL of the linking or embedding page when making a request

"{string:RefererQuery,ftype=refererquery;,}{regexp:Referer,ftype=referer;refName=RefererQuery,^([\w-]+://[^?]+|/[^?]+)}"

refererquery

reerer

User-Agent

The value of the HTTP User-Agent header

"{string:User-Agent,ftype=useragent;,}"

useragent

Version ID

The version ID of the object being copied or "-" if the x-amz-copy-source header didn’t specify a versionId parameter as part of the copy source

{text:Version ID,ftype=versionid}

versionid