Making a web client in the terminal

A browser window, a dark terminal window and a smartphone each send an envelope to the same server, and pages come back to each of them. — A , and a phone app play the same role: they ask first, and the answers.

A client is whatever asks first

and server are roles, not machines. Whichever side opens the conversation and asks for something is the client; the side that waits and answers is the server. Your browser is a client, so is curl, so is a phone app, so is another server calling an .

That is why you can write a web client in one line. It does not need a window, a , or a mouse — it needs to connect, send a , and read what comes back.

Question	Client	Server
Who speaks first	Opens the connection and sends the request.	Waits on a , silent until someone connects.
Who needs an address	Needs to know where the server is — a or an , and a port.	Needs a stable address, which is why servers get domain names and clients usually do not.
How many at once	Usually one conversation at a time, though a browser opens several to load a page faster.	Many at once, which is most of what makes real server complicated.

The same can be both

A web server that fetches a map or charges a card is acting as a client for that request. Roles swap by the second, so ask which direction a particular request travels rather than what kind of machine is at each end.

The quickest way to believe that is only text is to type a yourself. Start the server from the previous topic, then open a second and talk to it by hand.

Typing a request by hand with nc

connects to a and passes your keystrokes straight down the wire. Type the two lines, press Enter twice, and the server answers — you have just done a browser's job manually.

terminal

# connect, then type the two lines below and press Enter once more
nc localhost 8084

GET / HTTP/1.1
Host: localhost

# or send it all at once (printf's \r\n is HTTP's line break;
# sleep keeps the line open a moment so the shell server can answer)
{ printf 'GET / HTTP/1.1\r\nHost: localhost\r\n\r\n'; sleep 1; } | nc localhost 8084

The at the end matters. It tells the server the request is finished, which is why you press Enter one extra time.
HTTP requires carriage-return-and-newline at the end of each line. Typing works because most servers are forgiving, but the sends exactly what the standard asks for.
The line is required by . Leave it out against a real site and you will usually get an error back rather than a page.

Wrapping it into a script

Once the request is a single printf, a reusable is a few lines. This one takes a host, a port and a , sends the request, and splits the from the body so you can see the two parts.

client.sh

#!/bin/bash
# usage: bash client.sh localhost 8084 /

HOST=${1:-localhost}
PORT=${2:-8084}
PATH_=${3:-/}

{ printf 'GET %s HTTP/1.1\r\nHost: %s\r\nConnection: close\r\n\r\n' "$PATH_" "$HOST"; sleep 1; } \
  | nc "$HOST" "$PORT" \
  | tr -d '\r' \
  | awk 'BODY { print; next } /^$/ { BODY = 1; next } { print "[header] " $0 }'

The strips the that HTTP leaves at the end of every line, and marks each line as a header until it meets the empty line — after that, everything is the body. This is exactly the split a browser does before it starts drawing.

Run both halves together

Start the shell server in one terminal and this client in another, and you own both ends of a conversation you wrote yourself. When talking to the shell server, keep the connection open for a moment with , as in the above, or the answer never arrives. Change the in the server and the client sees it immediately — there is nothing else in between.

Back to the shell web server

curl: the client you will use every day

curl does properly what our script does roughly: , , , uploads, , . It is on almost every machine, and it is the standard way to test a server without a browser in the way.

terminal

# body only
curl http://localhost:8084

# body plus the response headers
curl -i http://localhost:8084

# the whole exchange, request headers included
curl -v http://localhost:8084

The options that matter

Option	What it does
`-i`	Show the headers as well as the body. The first thing to reach for when a page is not what you expected.
`-v`	Verbose: show the you sent and the response you got, line by line, including the connection itself.
`-L`	Follow redirects. Without it, a gives you an almost empty response instead of the page you wanted.
`-o FILE`	Save the response to a file instead of printing it. This is curl as a download tool.
`-H "Name: value"`	Add a request header — a content type, an authorisation , a language preference.
`-d "data"`	Send instead of just asking. -d alone makes it a (pick another method with -X), and this is how you test a form or an from the .
`-s`	Silent: no progress meter. Use it whenever the output feeds into another command or a script.
`-A "name"`	Pretend to be a particular browser. Some sites answer differently depending on who is asking.

Calling an API

Most of what a modern app talks to is not a page but an API returning . curl fetches it, and jq makes it readable — together they are how you check an before writing a single line of application .

terminal

# fetch JSON and pick one field out of it with jq
curl -s https://api.github.com/repos/sveltejs/svelte | jq '.stargazers_count'

# send JSON with POST
curl -X POST https://httpbin.org/post \
  -H 'Content-Type: application/json' \
  -d '{"name":"SONUB","level":1}'

# download to a file
curl -L -o page.html https://sonub.com

Get into the habit of testing with curl first. If curl gets the right answer, the server is fine and the is in your . If curl also gets a strange answer, the problem is on the server side — and you have not wasted an afternoon looking in the wrong place.

Chrome is the same client, at a different scale

sends the your command sent. Then it does the enormous amount of extra work that turns the answer into something a person can look at — and that extra work is the only real difference.

It is worth knowing the steps, because every one of them is a place where a page can go wrong. When something does not appear, the question is always which step failed.

From address bar to picture

Resolve the nameThe is turned into an by . If this fails you never reach the server at all, and the error mentions the name rather than the page.
Connect and secureIt opens a connection and, for , checks the before anything else is sent. A warning here means the could not be trusted, not that the page is missing.
Send the requestThe same handful of lines you typed by hand, with rather more : , accepted formats, preferred languages.
and fetch the restIt reads the and builds a from your . Then it requests everything the page points to. Stylesheets, images, fonts and scripts usually come several at once.
Style, lay out, paintIt applies the , works out where every box goes and draws it. runs while the HTML is still being read, and can change the page again after it is drawn.

Terminal client or browser

curl and nc Send a request and show you the answer, unchanged. Perfect for checking what a server really returns, and for scripts. They do not run JavaScript, so a page built in the browser looks empty to them.

Chrome Does all of the above and then renders, runs scripts, stores cookies, files and enforces . What you see is the result of the answer, not the answer itself.

Most browsers are the same engine

Chrome, , and are all built on , so a page that works in one usually works in the others. and use different engines, which is exactly why a site should be opened in at least two of them before you call it finished.

DevTools: watching the messages go by

ships with web tools, and they are already installed. Everything you sent by hand with is listed here, by request, with the beside it.

Opening it

Press , or Command-Option-I on a and Control-Shift-I elsewhere. Right-clicking any part of a page and choosing Inspect opens it with that already selected — the fastest way in.

The panels

Elements The live and the styles applied to it. Click an element to see its and change them on the spot — nothing is saved, so experiment freely.

Console Messages and errors from , and a prompt where you can run your own. A blank page with a red line here has told you exactly what broke.

Network Every request the page made, with method, status, size and timing. This is the panel that answers whether a problem is in the server or in the page.

Sources The scripts themselves, where you can set a and step through the line by line while the page waits.

Application What the site has stored on your machine: , , . The place to look when a site behaves oddly only for you.

Three things to try right now

Find your own requestOpen the , reload this page, and click the very first entry. The request and response there carry the same content as the lines you typed into nc. On a site connected over , the first line and show up as , :path and :authority.
Copy it as curlRight-click any request and choose Copy as cURL. Paste it into a and you have reproduced exactly what the browser did — headers, cookies and all.
Break it on purposeIn the , delete a heading and watch it vanish; in , throttle the speed to slow and reload. Both are safe, and both teach more than reading about them.

Now go back and read your own page

Open the HTML file you wrote in the first topic, inspect it, and you will recognise every tag in the tree. That loop — write it, serve it, request it, inspect it — is the whole of web development in miniature.

Back to HTML

You mark this yourself. Nothing is graded here.

Mark this step as understood?

This cannot be undone yet, so mark it only when you really understand.

Marked as understood

You mark this yourself. Nothing is graded here.

Notifications

How to enable notifications

Web Client

A client is whatever asks first