Running external commands from Python

Table of Contents

Motivation

Although Python has a ton of built-in features and the ecosystem is full of useful pip-installable packages sometimes you need to fire up some external tools

os.system

The simplest form of launching external tools is the os.system function

import os

os.system("echo Hello from the other side!")

subprocess module

The subprocess module is Python’s recommended way to executing shell commands

run function

The run() function was added in Python 3.5 and should be used whenever possible

import subprocess

cmd_line = "ls -al > folder_content.txt"
subprocess.run(cmd_line, shell=True, check=True)

if shell is set to True you can use other shell features such as shell pipes

If check is true, and the process exits with a non-zero exit code, a CalledProcessError exception will be raised.

If you encounter issues with encoding on Windows you can try setting the encoding:

subprocess_env = {'PYTHONUTF8': '1'}
subprocess.run(cmd_line, shell=True, check=True, env=subprocess_env)

shlex

shlex.split() can be useful when determining the correct tokenization for args, especially in complex cases:

cmd_line = "ls -al > folder_content.txt"
cmd_args = shlex.split(cmd_line)
print(cmd_args)

gives the output:

['ls', '-al', '>', 'folder_content.txt']

Popen

Popen is the most generic approach

call

is the old version and shall be avoided