hurl: HTTP, HTTPS and Gopher file grabber

hurl is a relatively simple HTTP, HTTPS and Gopher client/file grabber.

Why?

Sometimes (or most of the time?) you just want to fetch a file via the HTTP, HTTPS or Gopher protocol.

The focus of this tool is only this.

Features

• Uses OpenBSD pledge(2) and unveil(2). Allow no filesystem access (writes to stdout).
• Impose time-out and maximum size limits.
• Use well-defined exitcodes for reliable scripting (curl sucks at this).
• Send as little information as possible (no User-Agent etc by default).

Anti-features

• No HTTP byte range support.
• No HTTP User-Agent.
• No HTTP If-Modified-Since/If-* support.
• No HTTP auth support.
• No HTTP/2+ support.
• No HTTP keep-alive.
• No HTTP chunked-encoding support.
• No HTTP redirect support.
• No (GZIP) compression support.
• No cookie-jar or cookie parsing support.
• No Gopher text handling (".\r\n").
• ... etc...

Dependencies

• C compiler (C99).
• libc + some BSD functions like err() and strlcat().
• LibreSSL(-portable)
• libtls (part of LibreSSL).

Optional dependencies

• POSIX make(1) (for Makefile).
• mandoc for documentation: https://mdocml.bsd.lv/

Clone

git clone git://git.codemadness.org/hurl

Browse

You can browse the source-code at:

• https://git.codemadness.org/hurl/
• gopher://codemadness.org/1/git/hurl

Download releases

Releases are available at:

• https://codemadness.org/releases/hurl/
• gopher://codemadness.org/1/releases/hurl

Build and install

$ make
# make install

Examples

Fetch the Atom feed from this site using a maximum filesize limit of 1MB and a time-out limit of 15 seconds:

hurl -m 1048576 -t 15 "https://codemadness.org/atom.xml"

There is an -H option to add custom headers. This way some of the anti-features listed above are supported. For example some CDNs like Cloudflare are known to block empty or certain User-Agents.

User-Agent:

hurl -H 'User-Agent: some browser' 'https://codemadness.org/atom.xml'

HTTP Basic Auth (base64-encoded username:password):

hurl -H 'Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=' \
	'https://codemadness.org/atom.xml'

GZIP (this assumes the served response Content-Type is gzip):

hurl -H 'Accept-Encoding: gzip' 'https://somesite/' | gzip -d