Rails IP Spoofing Vulnerabilities and Protection

I’ve recently bumped into an interesting post about a stackoverflow vulnerability discovered by Anthony Ferrara. I didn’t think too much about it. I’ve come across similar issues before, where the application relies on a piece of information that might be easy to forge. Telephony systems are vulnerable to Caller ID spoofing, which becomes increasingly easier with Voice-Over-IP providers. Web based applications can also be fooled if they rely on header information, such as the X-Forwarded-For, typically used by Proxy servers.

I was experimenting with switching rails from Phusion Passenger to Unicorn, when I suddenly came across a strange error message:

ActionDispatch::RemoteIp::IpSpoofAttackError (IP spoofing attack?!HTTP_CLIENT_IP="192.168.0.131"HTTP_X_FORWARDED_FOR="192.168.0.131"): app/controllers/application_controller.rb:138:in `append_info_to_payload'

That looked quite impressive. Rails is trying to identify spoofing attacks and raise an exception when it happens? Nice.

However, after digging a little deeper, trying to figure out what’s actually happening, it seems that Rails might actually be vulnerable to spoofing attacks under certain setups. I will try to describe those scenarios and suggest a few workarounds to avoid any pitfalls.

What I observed applies to Rails latest stable (3.2.9 at the time of writing), previous versions and potentially future versions as well (including 4.0).

TL;DR

Your rails application might be vulnerable to IP spoofing. To test it, try to add a fake X-Forwarded-For header and check which IP address appears in your log files.

e.g.

curl -H "X-Forwarded-For: 5.5.5.5" http://your.website.com

You can try to implement one of the workarounds mentioned below.

Risk

I should probably start with talking about the potential risk from an IP Spoofing attack in general. I think it’s fair to say that out-of-the-box, there’s no real inherent risk to a rails app from receiving a request from a spoofed IP address. In many ways, the application shouldn’t care if the request is coming from Australia or Bulgaria, and perhaps even less so if the originating IP address is 43.25.6.2 or 56.22.9.17.

Not always though.

Many applications use the client ip information for different purposes. From tracking the location the user is coming from and displaying suitable content or language, to blocking or allowing access to certain parts of the application. This information is also typically logged to the application log, and may be used to track problems, or even investigate a security breach, fraud or misuse.

Therefore, whilst it might not necessarily be a serious problem if those IPs can be spoofed, it’s probably best to avoid this. Developers ideally shouldn’t need to worry that when accessing request.remote_ip they might get a spoofed value. So ideally rails will take care of it for us and will do the right thing. However, this is not always possible, because it depends on how rails is running, and there are other factors to consider… Primarily proxy servers that handle the request before it hits rails.

Rails app architecture

A rails app might include zero or more proxy servers along the path as part of its architecture. These proxies can be the web server (e.g. nginx, or Apache), caching or load balancing devices (e.g. HAProxy, Varnish) and others.
Each proxy along the path will most likely alter the source IP address. So how can the rails app know what was the real IP address of the client?

X-Forwarded-For

A proxy server that alters the source IP address will normally add a HTTP header called X-Forwarded-For to the request. This header should include the IP address of the client. When multiple proxy servers are used, there is a chain of IP addresses being added to the same header, or new X-Forwarded-For headers being added to the request (so that the request might contain several X-Forwarded-For headers).

X-Forwarded-For is not actually a standard, but (at least according to wikipedia) considered the de-facto standard. There is however a draft standard for an HTTP Forwarded header (a header which will try to encapuslate a few non-standard headers, one of which is X-Forwarded-For).

What makes things even more complicated with a non-official-standard is that it’s hard to figure out the order of the data in this header, or different headers. And the order matters.

According to the wikipedia page (and by the looks of it, most implementations), a proxy server that receives a request, should add the source IP address that it receives (the IP from the actual request hitting the proxy, not in any HTTP headers), and append it to the end of the X-Forwarded-For header.

So if there is no header from the client, and the IP address of the client is 1.2.3.4, the header will look like X-Forwarded-For: 1.2.3.4 after going through the proxy server.

If the request comes from 4.5.6.7 and already has a header of X-Forwarded-For: 2.3.4.5, the header will now be X-Forwarded-For: 2.3.4.5, 4.5.6.7 and so on.

RFC 2616, the standard for the entire HTTP protocol, also clarifies what should happen when multiple headers with the same name are added to a request.

Multiple message-header fields with the same field-name MAY be
present in a message if and only if the entire field-value for that
header field is defined as a comma-separated list [i.e., #(values)].
It MUST be possible to combine the multiple header fields into one
“field-name: field-value” pair, without changing the semantics of the
message, by appending each subsequent field-value to the first, each
separated by a comma. The order in which header fields with the same
field-name are received is therefore significant to the
interpretation of the combined field value, and thus a proxy MUST NOT
change the order of these field values when a message is forwarded.

So, it shouldn’t matter if an HTTP request contains several X-Forwarded-For headers, or one. As long as the order of the headers are maintained by proxies, several headers can be combined into one, comma-separated values (whose order is also maintained).

This way, the header describes the entire chain of ‘hops’ between the client and the server, with the leftmost element pointing to the original client IP address, and the last one of the last proxy along the path.

Note however, that some implementations appear to be broken. This does indeed complicate things even further. I will get back to those later.

For the sake of simplicity, I will ignore those broken implementations, and will refer to X-Forwarded-For as one header with a list of ordered IPs.

Who do you trust?

So if we consider the way most proxies operate, when the request actually hits our application (or rails in this case), it might contain an ordered-list of IP addresses. The leftmost should indicate the client’s IP address, and the rightmost IP will be the IP address of the hop before our closest proxy.

To illustrate: if our header looks like

X-Forwarded-For: A, B, C

this should tell us that:

• A was the original client (received by proxy B)
• B was an intermediate proxy (received by proxy C)
• C was another imetermediate proxy (received by proxy D – which delivered the request to our app).

Notice that we shouldn’t see D’s IP on the header, but the actual request should come from D’s IP.

There’s one problem however. Which proxy servers can we really trust to send us the correct information?

Whilst we can think that the most interesting information (the actual client IP address) is on the leftmost part. The most trustworthy information is on the rightmost part.

If we blindly pick the leftmost IP address, we make it very easy for a malicious user to forge. All it takes is adding a header to the request

curl -H "X-Forwarded-For: 6.66.6.66" http://your.site.

However many other proxies we have along the path, they will typically append to this header and won’t modify it or check it. So the leftmost IP is the easiest to forge. In fact, the leftmost N IP addresses are easiest to forge (I can add as many IPs as I want as an attacker).

So in order to figure out with highest accuracy the real IP address, we have to traverse the chain of proxies in reverse.

We know that proxy D can be trusted. In most cases this can be our own web server. We might also know that proxy C is also trustworthy. Maybe it’s our load-balancer, or a caching service we can trust. Lets say that’s as far as we can trust in our little example. So if we trust D, the last IP address (of C) is trustworthy, and also it’s definitely not the actual client. The same goes with C, and hence B’s IP address is reliable. Can we trust B to send us A’s IP address? If we can’t, then we must only take B’s IP address as the client. We should not accept A as a valid client IP.

How to interpret X-Forwarded-For headers

In order to pick the right IP address, lets look at the following scenarios:

No proxy (e.g. Nginx/Passenger)

If we use no proxy at all, we should Ignore X-Forwarded-For completely. This might sound obvious, but this is not what’s actually happening by default. If we take the X-Forwarded-For into account, it makes it very easy to spoof the client IP.

In Rails 3.2 Stable (update: as well as in future Rails 4.0), it’s trivial to spoof the source IP in setups without a proxy by simply adding a (fake) X-Forwarded-For header.

One Proxy (e.g. Nginx/Unicorn)

If we use only one proxy, this means only one proxy we can trust. Remember that this proxy’s IP won’t appear on the X-Forwarded-For header at all. Our proxy will however add, or append to this header. In this scenario, the rightmost IP address on the list it the (only) one that can be trusted. No previous addresses on this list we can trust.

In Rails 3.2 Stable this seems to be handled properly, i.e. the last IP in the chain is used. However, there are a few commits that seem to change this behaviour. Current master version picks the first, which will most likely allow spoofing. There’s a pull request trying to address this problem. In fact, it’s this issue that inspired me to write this post, and also provided very useful links to other resources!

However… this pull request does not seem to solve the first problem (when no proxy is involved). It also seems to introduce a reverse option, which allows picking the first IP address. I’m not entirely clear when this is necessary, but it has a reasonably high probability of making it possible/easy to spoof the remote IP address.

Several (trusted) proxies (e.g. HAProxy – Nginx – Unicorn)

In a scenario where we have more than one proxy we can trust, the rightmost IP address will be of one of those proxies. The leftmost IP however cannot be blindly trusted. The recommended procedure is therefore to follow the chain, from the rightmost-element to the left, ignoring IP addresses of the proxies we trust. Once we hit the first unknown IP addresses when traversing from right to left, we use this as our client’s IP.

This seems like the best-practice, and implemented both with Apache mod_remoteip and nginx realip module.

The Apache page above sums it nicely:

When multiple, comma delimited useragent IP addresses are listed in the header value, they are processed in Right-to-Left order. Processing halts when a given useragent IP address is not trusted to present the preceding IP address.

As far as I could tell, the pull request got this right too. It goes through the list of trusted proxies and removes them from the list of addresses on the X-Forwarded-For header. Then it picks the last (rightmost) IP address.

When right means left?

So why do people want to be able to reverse the order? i.e. take the leftmost IP? I believe it’s a simple misunderstanding. In a normal scenario, where the request really comes from a legitimate source. Even though we can’t trust it 100%, most requests are not forged after all. In such a case, the leftmost IP address will actually be valid. I’m not sure, and it depends on many factors, but I believe in this case, the leftmost and the rightmost IPs are actually the same. If we imagine a chain of trusted proxies, the request header could look like this:

X-Forwarded-For: 1.2.3.4, t1, t2, t3

Here t1, t2 and t3 are the addresses of our trusted proxies. The rightmost non-proxy address in this case is the same as the leftmost address. However, this does not mean we should always trust the leftmost IP. I believe we should simply specify all the proxies that we can trust, and we can get the most correct (non-spoofed) IP address.

I therefore believe allowing people to configure the order in rails can be dangerous and should be avoided. If I’m not mistaken, if you define the list of trusted proxies correctly, this should produce the leftmost IP that can be trusted, which in most cases is the leftmost IP.

What needs to be fixed

As I mentioned above, I would recommend removing the last_forwarded_ip option that reverses the list as it is potentially dangerous. However, given that it’s not the default, it’s fair to argue that it’s a user’s choice whether to set it explicitly, and take the risk. If left as an option. I hope the potential risks from using it should be clearly documented, to help users make the right choice.

update: The last_forwarded_ip option was later removed.

More importantly, I also think rails should by default protect against fake X-Forwarded-For headers in the (reasonably common?) scenario where no proxies are used at all. Therefore, perhaps the default should be to ignore X-Forwarded-For headers unless specified in the config? I know this can break backwards compatibility and can confuse a lot of people who do use a proxy already.

Workarounds

If you’re not using a proxy, for example for those running Rails with Passenger, you can (and should) protect yourself.

If you are using a proxy, and somehow still can fake your ip address by using a fake X-Forwarded-For header, then perhaps you’re using a wrong version of rails? Otherwise, I’d love to hear about it. Looking at the current stable implementation this should not happen.

Remove the RemoteIp middleware

(thanks to André Arko for his comment!)
This is the easiest and most effective way to avoid spoofed IPs if you are not using a proxy. Simply add this line to your config/application.rb

config.middleware.delete ActionDispatch::RemoteIp

Modify or Block X-Forwarded-For

You can probably use the headers-more 3rd party nginx module, or mod_headers for Apache.

Then simply remove any X-Forwarded-For headers from the request and you should be safe.

(I haven’t actually tried either of those myself.)

Switch to a proxy-based solution

Perhaps more drastic, but you can switch to a proxy-based solution, like unicorn behind nginx. Make sure the solution you choose is not one of the broken implementations!

Inside Rails

update: see above how to remove the RemoteIp middleware, which is far easier and more elegant.

My ruby/rails skills are still somewhat limited, but I’m pretty sure it’s also possible to create a miniature middleware for this purpose, and this will sanitize or remove X-Forwarded-For headers.

Otherwise, you can monkey-patch the action_dispatch middleware. The simplest place to do this is probably here. Change the function to return the ip, i.e.

@remote_ip ||= ip

I’d be reluctant to promote these workarounds as a long-term solution however. I believe rails should by default protect against this, and that the user should explicitly specify whether they want to use any X-Forwarded-For headers at all.

Broken implementations

I promised to get back to those. What if some implementation does not play nicely? For example, what if the web server ignores previous X-Forwarded-For headers and only uses the last one?

From my limited experimentation, this does not happen with nginx and unicorn. From what I read, this also seems to be ok with other web servers like Apache and proxies like HAProxy. If you are using a web server that fails to deal with those headers properly, I strongly urge you to contact the people developing it and ask them to fix it. I also believe that if rails sticks to the correct, de-facto standard, it will encourage other projects to do the same and will set a good example.

IP spoofing attack?! alert

Just a quick side-note. I might be wrong here, but raising this error seems to me very loosely related to any real IP Spoofing attack.

If rails implements the proxy checks like they appear to be doing (using the rightmost value which is not a proxy server), this alert should not be necessary at all. It looks like it might at best get raised when the web server or one of the proxies is misconfigured and adds a CLIENT_IP header as well as an X-Forwarded-For header, but the former does not match the latter.

I’m not saying it’s a bad thing to do this check, but I really wonder if it should simply say something like “proxy configuration mismatch” instead?

UPDATE

The post has been updated slightly after the pull request was merged, and following some discussions on github, over email and on the comments to this post. The current state of Rails unfortunately still leaves applications which do not use a proxy server vunlerable to spoofing. On the bright side however: more documentation was added to highlight this very clearly. The implementation now seems correct and avoids using the first IP address or reversing the order, which could expose many more users. If you are not using a proxy you can avoid IP spoofing by removing the middleware via a single configuration line, see the now updated workarounds section.