3 Tips for Long-Running Commands on a Remote Server

When needing to perform some long-running command, potentially with lots of output, on a remote server, there are a few things you can do to make your life easier. Given the tooling is available (which it should), this works with any kind of server or connection.

Use screen for Long-Running Commands

You might have come across the screen utility before. For any long-running command on a remote server, where you are prone to time-outs or other ways of having your connection closed (e.g., due to a network disconnect), I would strongly advise to use a screen session.

If you want to read more about that, I can recommend this nice little tutorial, as well as a much more complete blog post on using screen. For your everyday usage, you only really need two/three commands, though.

Start a screen Session

Having screen available on your remote server, you can start a new session by just typing:

screen

If you want to execute multiple commands in dedicated sessions, or just as a general best practice, you can name the session when creating it. For this, you need to pass the -S option, followed by the unique name.

For example:

screen -S media

List screen Sessions

If you have been disconnected or closed your terminal deliberately, you can always connect back to the server. Depending on how things are set up and what has happened since when you started the screen session, you may or may not automatically “land” in your screen session.

To see all running sessions, you can use:

screen -ls

In the output, you will see a unique ID, as well as the session name, if provided.

Reattach to a screen Session

If you do not connect back to the context of the session, or in case you explicitly detached from your session, you can use the following to reattach to it:

screen -r media

Please note that, in the above example, media is the session name we provided before. If you did not provide a session name, you can reattach using the unique ID (see screen -ls).

Occasionally, I found that reattaching with -r simply doesn’t work. I’m not sure why this is, maybe because the setup at hand was not a native linux server that I connected to and then executed a command, but instead a connection across multiple servers, using AWS tooling…?

Anyway, the way to definitely reattach to a screen session is to explicitly detach from anything else—using -d—while reattaching:

screen -dr media

Write the Output to a File

Depending on the command you are running, there may be lots and lots of output. Even when using screen, you might run into issues when there is more output than what the buffer can hold.

Redirect Output

While there are ways to configure the output buffer and/or screen directly, another much more straightforward (one-off) way is to write the output to a file. You can do so by delegating it via >, or if you want to append, use >>.

[COMMAND] > [FILE]

For example:

wp media regenerate > media.txt

Also Redirect Error Output

If you also want to write any errors printed to the screen to your file, you have to reroute stderr. You can do this like so:

[COMMAND] > [FILE] 2>&1

Essentially, this means that you want all errors to be written to the standard primary output. Actual example:

wp media regenerate > media.txt 2>&1

Write to File and Screen

Sometimes you want to write all outpt to the file, but also be able to just connect to the server and see what’s going on. Using > or >>, all output is redirected, so you don’t see anything in your terminal.

As always, there are multiple ways to have a command write to both terminal and file, but a really easy one is to use the tee command.

Pipe all output to tee and write to a file:

[COMMAND] | tee [FILE]

Real example:

wp media regenerate | tee media.txt

Again, if you are interested in errors as well, you can do like so:

[COMMAND] 2>&1 | tee [FILE]

Real example:

wp media regenerate 2>&1 | tee media.txt

Download the File from the Server

If you are wondering how you can get the file from the server, you should be able to use something like scp, or rsync. For example, like so:

scp [SERVER]:[FILE] .

Profile the Command

Sometimes you are interested in how long something took to execute. A simple-enough way to do this is to use the time utility.

Granted, this is not specific to a remotely-executed command, but nonetheless useful.

time [COMMAND]

Putting It All Together

Where that makes sense, you can also do all of the above. When working on migrating a large site from its dedicated stack ot another existing one, which also involved migrating all the uploads, and regenerating all the media files due to changed image sizes in the new theme, I did that quite a few times.

Kick it off:

screen -S media

time wp media regenerate 2>&1 | tee media.txt

Re-attach, if necessary:

screen -dr media

Finally, pull down the file:

scp [SERVER]:media.txt .

Done.