Haproxy logs
Jump to navigation
Jump to search
sumo
https://www.haproxy.com/blog/haproxy-log-customization/
HAProxy Log Customization
Baptiste Assmann | Oct 29, 2012 | BASICS, TECH | 10 comments
One of HAProxy’s greatest strengths has got to be its logging system. The amount of information it provides can be invaluable when diagnosing unexpected behavior, studying your website’s traffic, or just getting a feel for how HAProxy works.
HAProxy comes with a few standard log formats that define which fields will be captured. Common wisdom says that you should add the option httplog configuration directive to your frontend or defaults section when using HAProxy as a Layer 7 proxy. Or, use option tcplog when using it as a Layer 4 proxy. Either one will give your logs an instant power-up, making them more comprehensive.
Sometimes, though, these premade settings are not exactly what you need. In that case, you can configure a custom log format. In this blog post, you’ll see how.
Find out more in our blog post “Introduction to HAProxy Logging” or by registering for our webinar: “Deep Dive Into HAProxy Logging“
Customizing the HAProxy Logs
There are plenty of reasons why you might want to customize the fields captured by the HAProxy logs. For example:
the default log format is giving you more information that you need
you’re missing an important piece of information with the default log format
you need to structure the fields in a way that an external tool can read them
you rely on a standard log format and HAProxy must also comply
Luckily, it’s easy to add a template that says which fields to log and in which order. You just need to use the log-format directive. Let’s dive into its structure.
Adding a New HAProxy Log Format
The log-format directive goes into your defaults, frontend or listen section. It specifies a string that contains variables referring to the fields that you’d like to capture.
defaults
log-format "<format-string>"
view raw
When you place this into your defaults section, it affects all of the proxy sections (frontend and listen) that follow. If you need a different format for a particular proxy, adding another log-format to that section will override the default.
In the next section, you’ll learn about the variables that are available.
Variables
A large part of your custom log format will likely be made up of variables that are pre-defined by HAProxy. Others you will define yourself.
In the broadest terms, all variables follows the rules below when added to a format string:
It is preceded by a percent character: %
It can take flags in braces {}
If multiple flags, then they are separated by commas within the braces
Flags may be added or removed by prefixing them with a + or a – sign
Spaces, other than those between variables, must be escaped by a backslash
Currently, there are three flags:
Q: quotes a string
X: hexadecimal representation
E: escapes characters (“, \, ]) with backslashes
Here are a few examples:
log-format "%{+Q}r"
# Outputs: "GET / HTTP/1.1"
log-format "%{+X}Ts"
# Outputs: 5C5342A0
log-format "%{+E}HQ"
# Outputs: ?myparam=[some_brackets\]
log-format "%{+E,+Q}HQ"
# Outputs: "?myparam=[some_brackets\]"
view raw
The pre-defined variables (also available in the documentation) are:
Variable Meaning
%B bytes read
%CC captured request cookie
%CS captured response cookie
%H hostname
%HM HTTP method (e.g. POST)
%HP HTTP request URI without query string
%HQ HTTP request URI query string
%HU HTTP request URI
%HV HTTP version (e.g. HTTP/1.1)
%ID unique ID
%ST status code
%T GMT datetime
%Ta active time of requests (from %TR to end)
%Tc time to establish TCP connection to server
%Td Tt – (Tq + Tw + Tc + Tr)
%Tl local datetime
%Th connection handshake time (SSL, PROXY protocol)
%Ti idle time before the HTTP request
%Tq time to get the client request
%TR time to receive the full request from first byte
%Tr response time
%Ts timestamp
%Tt total session duration time
%U bytes uploaded
%ac process concurrent connections
%b backend name
%bc backend concurrent connections
%bi backend source IP (HAProxy connects with)
%bp backend source port (HAProxy connects with)
%bq backend queue
%ci client IP
%cp client port
%f frontend name
%fc frontend concurrent connections
%fi frontend IP
%fp frontend port
%ft frontend name transport (‘~’ suffix for SSL)
%lc frontend log counter
%hr captured request headers
%hrl captured request headers (CLF style)
%hs captured response headers
%hsl captured response headers (CLF style)
%ms accept date (milliseconds, left-padded with 0)
%pid HAProxy process ID
%r HTTP request
%rc number of retries
%rt request counter
%s server name
%sc server concurrent connections
%si server IP
%sp server port
%sq server queue
%sslc SSL cipher used (e.g. AES-SHA)
%sslv SSL version (e.g. TSLv1)
%t datetime (with millisecond resolution)
%tr datetime of start of HTTP request
%trg GMT datetime of start of HTTP request
%trl local datetime of start of HTTP request
%ts termination state
%tsc termination state with cookie status
The default log formats use these variables in the following ways:
TCP log format:
log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"
view raw
HTTP log format:
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"
view raw
CLF log format:
log-format "%{+Q}o %{-Q}ci - - [%trg] %r %ST %B \"\" \"\" %cp %ms %ft %b %s %TR %Tw %Tc %Tr %Ta %tsc %ac %fc %bc %sc %rc %sq %bq %CC %CS %hrl %hsl"
view raw
You can create your own format by using any of the pre-defined variables in the order that you wish. The following logs the client’s IP address and port, the server’s IP address and port, the path, and the response status:
log-format "%ci:%cp %si:%sp %HU %ST"
# Outputs: 192.168.50.1:56428 127.0.0.1:80 / 304
view raw
If you prefer, you can add text to give the fields more context. Placing %{+Q}o at the beginning will apply the flag, in this case quotes, to all of the following strings.
log-format "%{+Q}o\ client_address = %ci, client_port = %cp, server_address = %si, server_port = %sp path, = %HU, status = %ST"
# Outputs: client_address = "192.168.50.1", client_port = 56479, server_address = "127.0.0.1", server_port = 8080, path = "/", status = 304
view raw
You can also capture specific pieces of data not represented by the pre-defined variables. For example, you can log request headers. You’ll need to capture them first, though, and then refer to them by number. The numbers are incremented in the order in which the field was captured. Here are a few examples:
http-request capture req.hdr(Host) len 10
http-request capture req.hdr(Cookie) len 20
log-format "%[capture.req.hdr(0)] %{+Q}[capture.req.hdr(1)]"
# Outputs: example.com "MyCookie=abc123"
view raw
Custom variables can be logged by using the var fetch method. Note that you must prefix your variable names with txn.
http-request set-var(txn.MyVar) str("My Value")
log-format "%{+Q}[var(txn.MyVar)]"
# Outputs: "My Value"
view raw
Fetch methods can be included by placing them inside square brackets. In this example, the http_first_req fetch shows a 1 if this is the first request from the client, or a 0 if not.
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %[http_first_req]"
# Outputs: 192.168.50.1:53547 [16/Oct/2019:13:52:09.226] fe_main be_servers/s1 0/0/1/2/3 200 465 - - ---- 2/1/0/0/0 0/0 "GET / HTTP/1.1" 1