Amazon CloudFront
- Omry Koschitzky
Background
Built in Amazon CloudFront dashboards and consoles to gain deep-level insights on your global content delivery network all across your cloud AMIs at once.
The application is aimed to run on AWS CloudFront access logs stored in one of your S3 buckets. It is required to enable the logging of the CloudFront distributions to generate the logs and configure the path where log data will be written to).
Steps
- Add Log Data In XpoLog, When adding a log to XpoLog you can now set a Log Type (logtype). For Amazon CloudFront set the following logtypes:
- AWS
- CloudFront
- access
- The CloudFront access log usually is placed in a S3 bucket usually under the directory awscf... with the files name structure: <SITE_ID>.<DATE>-<UNIQUE_ID>.gz - in XpoLog it should be represented as {string}.{date,yyyy-MM-dd}-{string}.gz
It is required to configure a S3 account for XpoLog to connect and read the required data from the S3 bucket. - Once all required information is set click next and edit the log pattern, this step is crucial to the accuracy and deployment of the Amazon CloudFront App. Use the following conversion table to build the XpoLog pattern out of the access log format.
Example
The Amazon CloudFront access log format is:
date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version
In XpoLog this pattern will be translated into:
{date:Date,yyyy-MM-dd HH:mm:ss}{tab}{text:x-edge-location,ftype=x-edge-location}{tab}{text:sc-bytes,ftype=bytesent}{tab}{text:c-ip,ftype=remoteip}{tab}{text:cs-method,ftype=reqmethod}{tab}{text:cs(Host),ftype=hostname}{tab}{text:cs-uri-stem,ftype=requrl}{tab}{text:sc-status,ftype=respstatus}{tab}{text:cs(Referer),ftype=referer}{tab}{text:cs(User-Agent),ftype=useragent}{tab}{text:cs-uri-query,ftype=querystring}{tab}{text:cs(Cookie),ftype=cookie}{tab}{text:x-edge-result-type,ftype=x-edge-result-type}{tab}{text:x-edge-request-id,ftype=x-edge-request-id}{tab}{text:x-host-header,ftype=x-host-header}{tab}{text:cs-protocol,ftype=protocol}{tab}{text:cs-bytes,ftype=bytesreceived}{tab}{text:time-taken,ftype=time-taken}{tab}{text:x-forwarded-for,ftype=x-forwarded-for}{tab}{text:ssl-protocol,ftype=ssl-protocol}{tab}{text:ssl-cipher,ftype=ssl-cipher}{tab}{text:x-edge-response-result-type,ftype=x-edge-response-result-type}{tab}{text:cs-protocol-version,ftype=protocolversion}
For more information see below:
Amazon CloudFront Access Log Format Conversion Table
| Fields | Description | XpoLog Pattern | XpoLog ftype |
|---|---|---|---|
date time | Date and time of event | {date:Date,yyyy-MM-dd HH:mm:ss} | |
x-edge-location | The edge location that served the request | {text:x-edge-location,ftype=x-edge-location} | x-edge-location |
sc-bytes | The total number of bytes that CloudFront served to the viewer in response to the request, including headers | {text:sc-bytes,ftype=bytesent} | bytesent |
c-ip | The IP address of the viewer that made the request | {text:c-ip,ftype=remoteip} | remoteip |
cs-method | HTTP access method: DELETE, GET, HEAD, OPTIONS, PATCH, POST, or PUT. | {text:cs-method,ftype=reqmethod} | reqmethod |
cs(Host) | The domain name of the CloudFront distribution | {text:cs(Host),ftype=hostname} | hostname |
cs-uri-stem | The portion of the URI that identifies the path and object | {text:cs-uri-stem,ftype=requrl} | requrl |
sc-status | An HTTP status code or | {text:sc-status,ftype=respstatus} | respstatus |
cs(Referer) | The name of the domain that originated the request. Common referrers include search engines, other websites that link directly to your objects, and your own website | {text:cs(Referer),ftype=referer} | referer |
cs(User-Agent) | The value of the | {text:cs(User-Agent),ftype=useragent}
| useragent |
cs-uri-query | The query string portion of the URI, if any. When a URI doesn't contain a query string, the value of | {text:cs-uri-query,ftype=querystring}
| querystring |
cs(Cookie) | The cookie header in the request, including name-value pairs and the associated attributes. If you enable cookie logging, CloudFront logs the cookies in all requests regardless of which cookies you choose to forward to the origin: none, all, or a whitelist of cookie names. When a request doesn't include a cookie header, the value of cs(Cookie) is a hyphen (-) | {text:cs(Cookie),ftype=cookie} | cookie |
x-edge-result-type | How CloudFront classified the response after the last byte left the edge location | {text:x-edge-result-type,ftype=x-edge-result-type} | x-edge-result-type |
| x-edge-request-id | An encrypted string that uniquely identifies a request | {text:x-edge-request-id,ftype=x-edge-request-id}
| x-edge-request-id |
x-host-header | The value that the viewer included in the | {text:x-host-header,ftype=x-host-header} | x-host-header |
cs-protocol | The protocol that the viewer specified in the request, either | {text:cs-protocol,ftype=protocol} | protocol |
cs-bytes | The number of bytes of data that the viewer included in the request (client to server bytes), including headers | {text:cs-bytes,ftype=bytesreceived} | bytesreceived |
time-taken | The number of seconds (to the thousandth of a second, for example, | {text:time-taken,ftype=time-taken}
| time-taken |
x-forwarded-for | If the viewer used an HTTP proxy or a load balancer to send the request, the value of | {text:x-forwarded-for,ftype=x-forwarded-for}
| x-forwarded-for |
ssl-protocol | When | {text:ssl-protocol,ftype=ssl-protocol} | ssl-protocol |
ssl-cipher | When | {text:ssl-cipher,ftype=ssl-cipher} | ssl-cipher |
x-edge-response-result-type | How CloudFront classified the response just before returning the response to the viewer | {text:x-edge-response-result-type,ftype=x-edge-response-result-type} | x-edge-response-result-type |
cs-protocol-version | he HTTP version that the viewer specified in the request. Possible values include | {text:cs-protocol-version,ftype=protocolversion} | protocolversion |