Using a Proxy With Curl

By | 2020-11-21
0 Comment

Using a proxy with Curl is straightforward and flexible. There are a few methods you can use. You can specify the proxy URL on the command line, or set the appropriate environment variable.

Using -x

This is the most straightforward way to use a proxy server with curl. Using the -x option guarantees that you are using the proxy you intend. To do this, follow the -x option with the proxy URL. Below is an example of using an HTTP proxy running on the local machine:

# pwd
/var/log/squid
# cat access.log 

# curl -I -x http://localhost:3128 http://tylersguides.com
HTTP/1.1 301 Moved Permanently
Date: Thu, 19 Nov 2020 14:45:55 GMT
Server: Apache
X-Redirect-By: WordPress
Location: https://tylersguides.com/
host-header: c2hhcmVkLmJsdWVob3N0LmNvbQ==
Content-Type: text/html; charset=UTF-8
X-Cache: MISS from haproxy.tylersguides.com
X-Cache-Lookup: MISS from haproxy.tylersguides.com:3128
Via: 1.1 haproxy.tylersguides.com (squid/4.4)
Connection: keep-alive

# cat access.log 

1605797153.918   1040 ::1 TCP_MISS/301 406 HEAD http://tylersguides.com/ - HIER_DIRECT/162.241.219.14 text/html

Notice the log entry in the proxy access log.

You aren’t limited to HTTP proxies. SOCKS proxies are supported as well:

# curl -I -x socks5://127.0.0.1:1080 http://tylersguides.com
HTTP/1.1 301 Moved Permanently
Date: Thu, 19 Nov 2020 16:44:44 GMT
Server: Apache
X-Redirect-By: WordPress
Upgrade: h2,h2c
Connection: Upgrade
Location: https://tylersguides.com/
host-header: c2hhcmVkLmJsdWVob3N0LmNvbQ==
Content-Type: text/html; charset=UTF-8

# curl -I -x socks4://127.0.0.1:1080 http://tylersguides.com
HTTP/1.1 301 Moved Permanently
Date: Thu, 19 Nov 2020 16:45:07 GMT
Server: Apache
X-Redirect-By: WordPress
Upgrade: h2,h2c
Connection: Upgrade
Location: https://tylersguides.com/
host-header: c2hhcmVkLmJsdWVob3N0LmNvbQ==
Content-Type: text/html; charset=UTF-8

Authentication

Some proxies require authentication. In this case, supply them with the URL. Place them between the protocol and server. Separate the username and password with a :. Separate the credentials and server with @ If the password has a : or @ in it, that is ok. If the username or password contain characters that will be parsed by your shell or curl, specify them using followed by their hex value. For example in ASCII, @ has a value of 40 and : has a value of 3A. If the password is p@ssword, specify it as p%40ssword. Here are a few examples of URLs that include credentials:

http://username:password@proxy.tylersguides.com:3128
socks5://username:p%40ssword@proxy.tylersguides.com:3128

Now that you have seen how to add credentials to a proxy URL, here are a few working examples on one of my test systems. 

# curl -I -x socks5://proxy:bad@127.0.0.1:1080 http://tylersguides.com
curl: (7) User was rejected by the SOCKS5 server (1 1).
# curl -I -x socks5://proxy:abc@127.0.0.1:1080 http://tylersguides.com
HTTP/1.1 301 Moved Permanently
Date: Thu, 19 Nov 2020 17:36:11 GMT
Server: Apache
X-Redirect-By: WordPress
Upgrade: h2,h2c
Connection: Upgrade
Location: https://tylersguides.com/
host-header: c2hhcmVkLmJsdWVob3N0LmNvbQ==
Content-Type: text/html; charset=UTF-8
# curl -s -x http://proxy:bad@localhost:3128 http://tylersguides.com > /dev/null
# tail -1 /var/log/squid/access.log 
1605863655.687      5 127.0.0.1 TCP_DENIED/407 4029 GET http://tylersguides.com/ proxy HIER_NONE/- text/html
# curl -s -x http://proxy:abc@localhost:3128 http://tylersguides.com > /dev/null
# tail -1 /var/log/squid/access.log 
1605863706.098   1555 127.0.0.1 TCP_MISS/301 369 GET http://tylersguides.com/ proxy HIER_DIRECT/162.241.219.14 text/html

Notice how in some of the requests I use a bad password? This demonstrates that curl is actually attempting to authenticate. The -s causes curl to suppress output related to the transfer. I redirected the output when demonstrating bad credentials with the HTTP proxy for brevity. HTTP proxies typically respond with HTML status pages if they can’t serve the request.

If you send credentials, and the proxy server doesn’t require it, it won’t hurt anything. It should still function as expected, regardless of whether it is SOCKS or HTTP. Example:

# curl -I -x socks://proxy:abc@localhost:1080 http://tylersguides.com 
HTTP/1.1 301 Moved Permanently
Date: Fri, 20 Nov 2020 14:35:17 GMT
Server: Apache
X-Redirect-By: WordPress
Upgrade: h2,h2c
Connection: Upgrade
Location: https://tylersguides.com/
host-header: c2hhcmVkLmJsdWVob3N0LmNvbQ==
Content-Type: text/html; charset=UTF-8

# curl -I -x socks://localhost:1080 http://tylersguides.com 
HTTP/1.1 301 Moved Permanently
Date: Fri, 20 Nov 2020 14:35:23 GMT
Server: Apache
X-Redirect-By: WordPress
Upgrade: h2,h2c
Connection: Upgrade
Location: https://tylersguides.com/
host-header: c2hhcmVkLmJsdWVob3N0LmNvbQ==
Content-Type: text/html; charset=UTF-8

Environment Variables

You can also use a proxy server with curl by defining the relevant environment variable. There are several variables that you can define, I am only going to demonstrate a few. The documentation covering this is pretty good, so after I demonstrate a few examples you should be able to use the documentation to help you set the relevant variable.

I am using bash for the examples, so you may need to adjust the commands to set the variables if you are using a different shell.

Below I run a simple SOCKS proxy in the background and connect to it using curl. The proxy will output all connections made to it to the terminal. This makes it easy to demonstrate how setting the various proxy variables controls the behavior of curl.

# nylon -vvv -f &  
[1] 26696
# nylon: Listening on localhost:socks

If you are proxying HTTP or HTTPS, set one or both of http_proxy or HTTPS_PROXY. 

# export HTTP_PROXY='socks5://localhost:1080'
# curl -s https://tylersguides.com > /dev/null
# unset HTTP_PROXY
# export HTTPS_PROXY='socks5://localhost:1080'
# curl -s https://tylersguides.com > /dev/null
nylon: Connecting localhost:30772 <=> box5605.bluehost.com:https
nylon: (localhost:30772 <=> box5605.bluehost.com:https) Terminated connection
# curl -s http://tylersguides.com > /dev/null
# unset HTTPS_PROXY
# export http_proxy='socks5://localhost:1080'
# curl -s https://tylersguides.com > /dev/null
# curl -s http://tylersguides.com > /dev/null
nylon: Connecting localhost:42771 <=> box5605.bluehost.com:http
nylon: (localhost:42771 <=> box5605.bluehost.com:http) Terminated connection

Notice how curl only connected when the relevant variable for the protocol was set. Most protocols curl supports can be proxied with a protocol specific variable. Simply prepend the protocol to _PROXY when you set the variable value. Below are few examples:

# curl -s ftp://tylersguides.com > /dev/null
# export FTP_PROXY='socks5://localhost:1080'
# curl -s ftp://tylersguides.com > /dev/null
nylon: Connecting localhost:15728 <=> box5605.bluehost.com:ftp
nylon: (localhost:15728 <=> box5605.bluehost.com:ftp) Terminated connection
# curl -s ldap://tylersguides.com > /dev/null
# export LDAP_PROXY='socks5://localhost:1080'
# curl -s ldap://tylersguides.com > /dev/null
nylon: connect(): Connection refused
nylon: Negotiation failed

If you want curl to use a proxy server regardless of protocol, use the ALL_PROXY variable:

# echo $HTTPS_PROXY

# echo $FTP_PROXY

# echo $http_proxy

# export ALL_PROXY='socks5://localhost:1080'
# curl -s http://tylersguides.com > /dev/null
nylon: Connecting localhost:32011 <=> box5605.bluehost.com:http
nylon: (localhost:32011 <=> box5605.bluehost.com:http) Terminated connection
# curl -s https://tylersguides.com > /dev/null
nylon: Connecting localhost:64338 <=> box5605.bluehost.com:https
nylon: (localhost:64338 <=> box5605.bluehost.com:https) Terminated connection
# curl -s ftp://tylersguides.com > /dev/null
nylon: Connecting localhost:15415 <=> box5605.bluehost.com:ftp
nylon: (localhost:15415 <=> box5605.bluehost.com:ftp) Terminated connection

References

See Also