hurl: HTTP, HTTPS and Gopher file grabber

Last modification on

hurl is a relatively simple HTTP, HTTPS and Gopher client/file grabber.

Why?

Sometimes (or most of the time?) you just want to fetch a file via the HTTP, HTTPS or Gopher protocol.

The focus of this tool is only this.

Dependencies

  • C compiler (C99).
  • libc + some BSD functions like err() and strlcat().
  • LibreSSL(-portable)
  • libtls (part of LibreSSL).

Optional dependencies

  • POSIX make(1) (for Makefile).
  • mandoc for documentation: https://mdocml.bsd.lv/

Clone

git clone git://git.codemadness.org/hurl

Download releases

Releases are available at: https://codemadness.org/releases/hurl/.

Build and install

$ make
# make install

Features

  • Uses OpenBSD pledge(2) and unveil(2). Allow no filesystem access (writes to stdout).
  • Impose time-out and maximum size limits.
  • Use well-defined exitcodes for reliable scripting (curl sucks at this).
  • Send as little information as possible (no User-Agent etc by default).

Anti-features

  • No HTTP byte range support.
  • No HTTP User-Agent.
  • No HTTP If-Modified-Since/If-* support.
  • No HTTP auth support.
  • No HTTP/2+ support.
  • No HTTP keep-alive.
  • No HTTP chunked-encoding support.
  • No HTTP redirect support.
  • No (GZIP) compression support.
  • No cookie-jar or cookie parsing support.
  • No Gopher text handling (".\r\n").
  • ... etc...

Examples

Fetch the Atom feed from this site using a maximum filesize limit of 1MB and a time-out limit of 15 seconds:

hurl -m 1048576 -t 15 "https://codemadness.org/atom.xml"