Logging Remote IP in Rails

When your Rails logs co-exist with other logs (e.g., web server, load balancer, etc), the concept of a “Remote IP” can vary based on which part of the stack you are considering. This is especially true when all you have to diagnose an issue is the end-user’s IP address. Distributed tracing helps, but many hosted applications can still look like a single Rails app in an archipelago of various services.

I once found myself deploying a new Rails app where the wrong IP address showed up as the “remote_ip” in the logs. I realized it had been a long time since I had to fix this, so I felt a need to document how to fix it & why it is so for folks who have not encountered this issue, or wonder what is going on.

My goal here is not to be exhaustive in how Rails resolves a remote IP, but to save others time.

So if you are stuck wondering why the “remote_ip” in your Rails request logs is wrong, or what the difference is between “client_ip” and “remote_ip”, let’s get into it.¹

Tl;dr

OK, OK, you’re in a rush.

You need to tell Rails to treat internal IPs as ‘trusted proxy’ servers. For apps hosted on CloudRun, Google appends the load balancer’s external IP to the X-Forwarded-For header which is sent to your app. Letting Rails know to ignore this IP will allow it use the IP address immediately preceding it in the chain of IPs within this header, and that’s nearly always the one you want².

# Configuring 'trusted proxies' in Rails config
#
# config/production.rb
config.action_dispatch.trusted_proxies = [
  "11.22.33.44" # i.e. your CloudRun external IP
].map { |proxy| IPAddr.new(proxy) }

The example above specifies a single IP but it can also be an IP range, a set of individual IP addresses, or a combination of both. In my experience with CloudRun, it’s been a single IP address.

Note: Avoid using a custom header (e.g.,X-REAL-USER-IP-USE-THIS) to expose the remote IP. Using an existing header like X-Forwarded-For allows future applications or integrations to work well and avoid future headaches.

The following goes into more detail.

So let’s dig into it a little.

Why is Rails reporting the wrong remote_ip?

Proxy servers (e.g. HAProxy, Nginx, etc…) can present multiple IP addresses to your Rails app and these appear in several ways (there are other ways but let’s stick to these for now):

1. The client’s IP address³ (this is what we want)
2. The IP address of the proxy server itself
3. The IP address of any intermediate servers handling the request between #1 and #2.

From your app’s point of view, all of these IP addresses are “remote,” and that is often the initial source of confusion.

Ways to access remote IP in Rails

Adding to the first potential source of confusion, a request in Rails exposes several different ways to access the remote IP, with subtle differences between them (hint: ActionDispatch::Request#remote_ip is most likely) what you want.

In all the examples below, I show the output of stepping through a Rails app, printing the values of three different request methods to show how their behavior changes in response to setting different HTTP headers.

1. request.client_ip
2. request.ip
3. request.remote_ip

To understand the differences in behavior, you need to understand what HTTP headers your proxy server sets or forwards. This may be hard for you to know in an actual production environment, but the two we’ll cover here are HTTP_CLIENT_IP and HTTP_X_FORWARDED_FOR.

request.ip

# curl http://localhost:3000/
# [request.client_ip, request.ip, request.remote_ip]
[nil, "::1", "::1"]

Here, ip exposes the IP of the request. Pretty simple. We haven’t provided any HTTP headers. In a real-world production environment (like Google CloudRun), you will have HTTP headers set by the proxy server that provide more information, and ip will give you the IP address of the proxy server itself. Interestingly, remote_ip prints the same IP here (we’ll talk about that later and show how these two can differ).

request.client_ip

This method is an accessor to the Client IP HTTP header as you can see in the code. Rails will add a few accessors out of the box for HTTP headers it thinks you will care about most. HTTP_CLIENT_IP is just one of those, so there’s nothing too special about it.

# curl http://localhost:3000/ \
#   -H "Client-IP: 5.5.5.5"

# [request.client_ip, request.ip, request.remote_ip]
["5.5.5.5", "::1", "5.5.5.5"]

In the example above, you see that client_ip returns the value of the header, ip returns the localhost address (I’m running on my laptop), but remote_ip also returns the value of the Client IP header.

Jumping ahead a bit, here’s the result of setting two different headers (Client-IP and X-Forwarded-For), to show how they relate.

# curl http://localhost:3000/ \
#    -H "X-Forwarded-For: 5.5.5.5 128.174.5.59" \
#    -H "Client-IP: 5.5.5.5"

# [request.client_ip, request.ip, request.remote_ip]
["5.5.5.5", "128.174.5.59", "128.174.5.59"]

Here you see that ip returns the value from X-Forwarded-For, while client_ip returns the header (this makes sense, as, again, client_ip is a straight mapping of the header).

So what happens if the client IP header says one thing, but X-Forwarded-For says something else? Good question. Let’s try it:

# curl http://localhost:3000/ \
#   -H "X-Forwarded-For: 128.174.5.59" \
#   -H "Client-IP: 5.5.5.5"

# [request.client_ip, request.ip, request.remote_ip]
eval error: IP spoofing attack?! HTTP_CLIENT_IP="5.5.5.5" HTTP_X_FORWARDED_FOR="128.174.5.59"
#  .../gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/remote_ip.rb:136:in `calculate_ip'
#  .../gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/remote_ip.rb:156:in `to_s'
#  .../gems/actionpack-6.1.6.1/lib/action_dispatch/http/request.rb:293:in `remote_ip'

This is Rails’ RemoteIp middleware being helpful. You might have a misconfigured proxy server, or you might have someone trying to do something funny. With regards to CloudRun (and, I suspect, other modern PaaS products), you are unlikely to encounter this spoofing error. A well-behaved proxy server should mitigate common IP spoofing attacks.

request.remote_ip

The Rails Guides states that remote_ip provides “The IP address of the client”. This isn’t the most helpful nor detailed description. Oh well.

We can’t really talk about remote_ip without considering the X-Forwarded-For header and the ‘RemoteIp’ middleware. You can inspect a bit of what this looks like on the request itself:

request.env["action_dispatch.remote_ip"]

#<ActionDispatch::RemoteIp::GetIp:0x0000000107149300
@check_ip=true,
@ip="::1",
@proxies=[
  #<IPAddr: IPv4:127.0.0.0/255.0.0.0>,
  #<IPAddr: IPv6:0000:0000:0000:0000:0000:0000:0000:0001/ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff>,
  #<IPAddr: IPv6:fc00:0000:0000:0000:0000:0000:0000:0000/fe00:0000:0000:0000:0000:0000:0000:0000>,
  #<IPAddr: IPv4:10.0.0.0/255.0.0.0>,
  #<IPAddr: IPv4:172.16.0.0/255.240.0.0>,
  #<IPAddr: IPv4:192.168.0.0/255.255.0.0>
],
@req=#<ActionDispatch::Request GET "http://localhost:3000/" for ::1>>

The proxies array lists “trusted proxies” (i.e., IP address spaces that aren’t really ‘remote’ and should be ignored when determining the remote IP).

In our final example below, we send an X-Forwarded-For header with multiple IP addresses, one of which we configure as a trusted proxy.

Keep in mind X-Forwarded-For can (and in production, most certainly will) contain multiple IP addresses, only one of which is the actual Remote IP we are interested in.

# application.rb
config.action_dispatch.trusted_proxies = [
  "2.2.2.2"
].map { |proxy| IPAddr.new(proxy) }


# curl http://localhost:3000/
#   -H "X-Forwarded-For: 128.174.5.59 2.2.2.2"

# [request.client_ip, request.ip, request.remote_ip]
[nil, "2.2.2.2", "128.174.5.59"]

In this example, client_ip is nil (no Client-IP header present), ip is the last IP address in the X-Forwarded-For list, and remote_ip is the first IP address in the X-Forwarded-For list. We hit the trifecta!

Pulling it together

You should rely on remote_ip to report the IP address of the client. Full stop.

If you are interested in the proxy server that sent you the request, use ip. And only rely on client_ip if you need to read that header (there might be a reason, but I haven’t encountered one in years). You may encounter exceptions to these guidelines, but that’s why they’re guidelines: they will get you far enough.

Setting trusted proxies

So we can tell Rails that an IP address (say, “11.22.33.44”) is a trusted proxy. What this means in practice can be seen by setting this configuration option and inspecting the request again:

Here’s that configuration again:

# config/production.rb
Rails.application.configure do
  # ...
  config.action_dispatch.trusted_proxies = [
    "11.22.33.44"
  ].map { |proxy| IPAddr.new(proxy) }
end

Here’s a peek at the request itself:

request.env["action_dispatch.remote_ip"]
#<ActionDispatch::RemoteIp::GetIp:0x000000010e80a2c8
@check_ip=true,
@proxies=[
  #<IPAddr: IPv4:11.22.33.44/255.255.255.255>
],
@req=#<ActionDispatch::Request GET "http://localhost:3000/" for ::1>>

The array of IP ranges from our earlier example has been replaced by our single IP (or range). If you need to retain the default IP ranges for some reason, just append ActionDispatch::RemoteIp::TRUSTED_PROXIES to the list of proxies in your application config.

Hosted solutions like CloudRun can be hard to wrangle with given that you can’t SSH into a CloudRun instance to try things out live. But I hope this gives you enough background to solve the problem next time without resorting to other tactics.