Amazon CloudFront

Owned by Omry Koschitzky
Last updated: Dec 20, 2017
3 min read

Background

Built in Amazon CloudFront dashboards and consoles to gain deep-level insights on your global content delivery network all across your cloud AMIs at once.

The application is aimed to run on AWS CloudFront access logs stored in one of your S3 buckets. It is required to enable the logging of the CloudFront distributions to generate the logs and configure the path where log data will be written to).

Steps

 

  1. Add Log Data In XpoLog, When adding a log to XpoLog you can now set a Log Type (logtype). For Amazon CloudFront set the following logtypes:

    1. AWS

    2. CloudFront

    3. access

  2. The CloudFront access log usually is placed in a S3 bucket usually under the directory awscf... with the files name structure: <SITE_ID>.<DATE>-<UNIQUE_ID>.gz - in XpoLog it should be represented as {string}.{date,yyyy-MM-dd}-{string}.gz
    It is required to configure a S3 account for XpoLog to connect and read the required data from the S3 bucket.

  3. Once all required information is set click next and edit the log pattern, this step is crucial to the accuracy and deployment of the Amazon CloudFront App. Use the following conversion table to build the XpoLog pattern out of the access log format.

 

Example

The Amazon CloudFront access log format is: 

date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version

 

In XpoLog this pattern will be translated into:

{date:Date,yyyy-MM-dd     HH:mm:ss}{tab}{text:x-edge-location,ftype=x-edge-location}{tab}{text:sc-bytes,ftype=bytesent}{tab}{text:c-ip,ftype=remoteip}{tab}{text:cs-method,ftype=reqmethod}{tab}{text:cs(Host),ftype=hostname}{tab}{text:cs-uri-stem,ftype=requrl}{tab}{text:sc-status,ftype=respstatus}{tab}{text:cs(Referer),ftype=referer}{tab}{text:cs(User-Agent),ftype=useragent}{tab}{text:cs-uri-query,ftype=querystring}{tab}{text:cs(Cookie),ftype=cookie}{tab}{text:x-edge-result-type,ftype=x-edge-result-type}{tab}{text:x-edge-request-id,ftype=x-edge-request-id}{tab}{text:x-host-header,ftype=x-host-header}{tab}{text:cs-protocol,ftype=protocol}{tab}{text:cs-bytes,ftype=bytesreceived}{tab}{text:time-taken,ftype=time-taken}{tab}{text:x-forwarded-for,ftype=x-forwarded-for}{tab}{text:ssl-protocol,ftype=ssl-protocol}{tab}{text:ssl-cipher,ftype=ssl-cipher}{tab}{text:x-edge-response-result-type,ftype=x-edge-response-result-type}{tab}{text:cs-protocol-version,ftype=protocolversion}

 

For more information see below:

Amazon CloudFront Access Log Format Conversion Table

 

Fields

Description

XpoLog Pattern

XpoLog ftype

Fields

Description

XpoLog Pattern

XpoLog ftype

date time

Date and time of event
Note: between yyyy-MM-dd and HH:mm:ss there should be a tab and not a space

{date:Date,yyyy-MM-dd HH:mm:ss}

 

x-edge-location

The edge location that served the request

{text:x-edge-location,ftype=x-edge-location}

x-edge-location

sc-bytes

The total number of bytes that CloudFront served to the viewer in response to the request, including headers

{text:sc-bytes,ftype=bytesent}

bytesent

c-ip

The IP address of the viewer that made the request

{text:c-ip,ftype=remoteip}

remoteip

cs-method

 HTTP access method: DELETE, GET, HEAD, OPTIONS, PATCH, POST, or PUT.

{text:cs-method,ftype=reqmethod}

reqmethod

cs(Host)

The domain name of the CloudFront distribution

{text:cs(Host),ftype=hostname}

hostname

cs-uri-stem

The portion of the URI that identifies the path and object

{text:cs-uri-stem,ftype=requrl}

requrl

sc-status

An HTTP status code or 000, which indicates that the viewer closed the connection (for example, closed the browser tab) before CloudFront could respond to a request.

{text:sc-status,ftype=respstatus}

respstatus

cs(Referer)

The name of the domain that originated the request. Common referrers include search engines, other websites that link directly to your objects, and your own website

{text:cs(Referer),ftype=referer}

referer

cs(User-Agent)

The value of the User-Agent header in the request. The User-Agent header identifies the source of the request, such as the type of device and browser that submitted the request and, if the request came from a search engine, which search engine

{text:cs(User-Agent),ftype=useragent}

 

useragent

cs-uri-query

The query string portion of the URI, if any. When a URI doesn't contain a query string, the value of cs-uri-query is a hyphen (-)

{text:cs-uri-query,ftype=querystring}

 

querystring

cs(Cookie)

The cookie header in the request, including name-value pairs and the associated attributes. If you enable cookie logging, CloudFront logs the cookies in all requests regardless of which cookies you choose to forward to the origin: none, all, or a whitelist of cookie names. When a request doesn't include a cookie header, the value of cs(Cookie) is a hyphen (-)

{text:cs(Cookie),ftype=cookie}

cookie

x-edge-result-type

How CloudFront classified the response after the last byte left the edge location

{text:x-edge-result-type,ftype=x-edge-result-type}

x-edge-result-type

x-edge-request-id

An encrypted string that uniquely identifies a request

{text:x-edge-request-id,ftype=x-edge-request-id}

 

x-edge-request-id

x-host-header

The value that the viewer included in the Host header for this request

{text:x-host-header,ftype=x-host-header}

x-host-header

cs-protocol

The protocol that the viewer specified in the request, either http or https

{text:cs-protocol,ftype=protocol}

protocol

cs-bytes

The number of bytes of data that the viewer included in the request (client to server bytes), including headers

{text:cs-bytes,ftype=bytesreceived}

bytesreceived

time-taken

The number of seconds (to the thousandth of a second, for example, 0.002) between the time that a CloudFront edge server receives a viewer's request and the time that CloudFront writes the last byte of the response to the edge server's output queue as measured on the server

{text:time-taken,ftype=time-taken}

 

time-taken

x-forwarded-for

If the viewer used an HTTP proxy or a load balancer to send the request, the value of c-ipin field 5 is the IP address of the proxy or load balancer. In that case, x-forwarded-for is the IP address of the viewer that originated the request

{text:x-forwarded-for,ftype=x-forwarded-for}

 

x-forwarded-for

ssl-protocol

When cs-protocol in field 17 is https, the SSL protocol that the client and CloudFront negotiated for transmitting the request and response. When cs-protocol is http, the value for ssl-protocol is a hyphen (-)

{text:ssl-protocol,ftype=ssl-protocol}

ssl-protocol

ssl-cipher

When cs-protocol in field 17 is https, the SSL cipher that the client and CloudFront negotiated for encrypting the request and response. When cs-protocol is http, the value for ssl-cipher is a hyphen (-)

{text:ssl-cipher,ftype=ssl-cipher}

ssl-cipher

x-edge-response-result-type

How CloudFront classified the response just before returning the response to the viewer

{text:x-edge-response-result-type,ftype=x-edge-response-result-type}

x-edge-response-result-type

cs-protocol-version

he HTTP version that the viewer specified in the request. Possible values include HTTP/0.9, HTTP/1.0, HTTP/1.1, and HTTP/2.0

{text:cs-protocol-version,ftype=protocolversion}

protocolversion