Editing 1654: Universal Install Script

Jump to: navigation, search

Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.
Latest revision Your text
Line 8: Line 8:
  
 
==Explanation==
 
==Explanation==
Most users of computers today are used to simple, easy installation of programs. You just download a {{w|.exe}} or a {{w|Installer_(OS_X)#Installer_package|.pkg}}, double click it, and do what it says. Sometimes you don't even have to install anything at all, and it runs without any installation.
 
  
However, when things are more "homebrew", for example downloading source code, things are more complicated.  Under {{w|Unix-like}} systems, which this universal install script is designed for, you may have to work with "build environments" and "{{w|makefiles}}", and command line tools. To make this process simpler, there exist repositories of programs which host either packages of source code and the things needed to build it or the pre-built programs. When you download the package, it automatically does most of the work of building the code into something executable if necessary and then installing it. However, there are many such repositories, such as "{{w|pip (package manager)|pip}}" and "brew", among others listed in the comic. If you only know the name of a program or package, you may not know in which repository(ies) it resides.
+
Most users of computers today are used to simple, easy installation of programs. You just download a .exe or a .pkg, double click it, and do what it says. Sometimes you don't even have to install anything at all, and it runs without any installation.
  
The <code>install.sh</code> file provided in the comic is a {{w|shell script}}, which attempts to fix this problem by acting as a "universal install script" that contains a lot of common install commands used in various Unix-like systems. This script in particular is interpreted by the {{w|Bourne Again Shell}} (Bash), which is denoted by the <code>#!/bin/bash</code> in the first line. In between each of the install commands in the script is the & character, which in {{w|POSIX}}-compatible {{w|Unix shell|shells}} (including {{w|Bash (Unix shell)|Bash}}, a popular shell scripting language) means it should continue to run the next command without waiting for the first command to finish, also known as "running in the background". This has the effect of running all the install commands simultaneously; all output and error text provided by them will be mixed together as they are all displaying on the screen around the same time.
+
However, when things are more "homebrew", for example downloading source code, things are more complicated. Under {{w|Unix-like}} systems, which this universal install script is designed for, you may have to work with "build environments" and "makefiles", and command line tools. To make this process simpler, there exist repositories of programs which host either packages of source code and the things needed to build it or the pre-built programs. When you download the package, it automatically does most of the work of building the code into something executable if necessary and then installing it. However, there are many such repositories, such as "pip" and "brew", among others listed in the comic. If you only know the name of a program or package, you may not know in which repository(ies) it resides.
  
The script accepts the name of a program or package as an argument when you run it. This value is then referenced as "$1" (argument number 1). Everywhere the script says "$1", it substitutes in the name of the package you gave it. The end result is the name being tried against a large number of software repositories and package managers, and (hopefully) at least one of them will be appropriate and the program will be successfully installed. Near the end, it even tries copying the source code from an online source and then runs several commands which compile/build the program.
+
The script provided in the comic attempts to fix this problem, by giving a "universal install script", which contains a lot of common install commands used in various Unix-like systems. In between each of the install commands in the script is the & character, which in POSIX-compatible shells (including {{w|Bash (Unix shell)|Bash}}, a popular shell scripting language) means it should continue to run the next command without waiting for the first command to finish, and not print any output of the command other than errors.  This has the effect of running all the install commands simultaneously; whatever errors each commands would have because of a package not existing in that repository will be mixed together as they are all displaying on the screen around the same time.  More about the & below.
 +
 
 +
The script accepts the name of a program when you run it as an argument. This value is then referenced as "$1" (argument number 1). Everywhere the script says "$1", it substitutes in the name of the package you gave it. The end result is the name being tried against a large number of software repositories and package managers, and hopefully, at least one of them will be appropriate and the program will be successfully installed. Near the end, it even tries changing the current working directory to that which is assumed to hold the package to be installed, and then runs several commands which build the program from source code.
  
 
All in all, this script would probably work; it runs many standard popular repository programs and package managers, and runs the nearly-universal commands needed to build a program.  Most of the commands would simply give an error and exit, but hopefully the correct one will proceed with the install.
 
All in all, this script would probably work; it runs many standard popular repository programs and package managers, and runs the nearly-universal commands needed to build a program.  Most of the commands would simply give an error and exit, but hopefully the correct one will proceed with the install.
  
One of the more subtle jokes in the comic is the inclusion of <code>apt-get</code> and <code>sudo apt-get</code> in the same script. Good unix practice dictates never logging in as root; instead you stay logged in as your normal user, and run system admin accounts via <code>sudo program name</code>. This prevents accidental errors and enables logging of all sensitive commands. A side effect of this, however, is that an administrator may forget to prefix their command with <code>sudo</code>, and re-running it properly the second time. This is a common joke in the Linux community, an example of which can be found at this [https://web.archive.org/web/20220304210306/https://twitter.com/liamosaur/status/506975850596536320 viral tweet] which shows a humorous workaround for the issue.
+
One of the more subtle jokes in the comic is the inclusion of <code>apt-get</code> and <code>sudo apt-get</code> in the same script. In most cases this would be redundant as the <code>sudo</code> command is just to add admin permissions. This could be an allusion to a joke in the Linux community about forgetting to include the sudo command. An example of this joke being used elsewhere was a  [https://twitter.com/liamosaur/status/506975850596536320 viral tweet] that showed a workaround for the issue. Sudo has also been used both by [[Randall]] in [[149: Sandwich]] and by Jason Fox to force Randall to let him appear on xkcd with [[824: Guest Week: Bill Amend (FoxTrot)]].
 
 
Since Randall's script does not use sudo for any but the <code>apt-get</code> command, there are two possibilities: the script itself was run via the root user or via sudo, in which case the <code>sudo apt-get</code> is not needed, or the script was run as a normal user, in this case the commands may install a local (as opposed to system-wide) version depending on local conditions. For instance npm will install a copy of the package under $HOME/.npm and pip would work as long as the user is working in a [https://iamzed.com/2009/05/07/a-primer-on-virtualenv/ virtualenv] (which is standard practice for Python developers).
 
 
 
Sudo has also been used both by [[Randall]] in [[149: Sandwich]] and by Jason Fox to force Randall to let him appear on xkcd with [[824: Guest Week: Bill Amend (FoxTrot)]].
 
  
The tool <code>curl</code> downloads files from the network (e.g., the Internet). For example, <code>curl http://xkcd.com/</code> downloads and displays the xkcd HTML source. The pipe <code>|</code> in the script attaches the output of the command before the pipe to the input of the command after the pipe, thus running whatever commands exist in the web content. Although this "curl|sh" pattern is a common practice for conveniently installing software, it is considered extremely unwise; you are running untrusted code without validation, there may be a MITM who modifies the code you receive, or the remote system could have been hijacked and the code made malicious. Most local package managers (e.g. <code>apt</code>, <code>yum</code>) offer digitally-signed packages that thwart this problem. You can find many examples of software providers suggesting a <code>curl|sh</code> solution at [https://curlpipesh.tumblr.com/ curlpipesh]
+
Another explanation for this could be that plain "apt-get" is for Debian, while Ubuntu etc. use sudo.
  
There appears to be a bug with the & at the end of the "git clone" line; since a git repository typically contains program source code, not executables, it may have been intended to retrieve the source code with git and then compile and install the program in the next line. In this case, the single & should be replaced with &&, an operator that will run the second command only if the first one has completed successfully. This plays into a second bug on the "configure" line, where the placement of the & means that only the "make install" command will be run asynchronously after the "configure" and "make" steps have finished in sequence (though this would likely fail due to a lack for write permissions unless it was run with sudo). To make success as likely as possible, the two lines should be like this or script should be executed twice:
+
The tool <code>curl</code> downloads files from the network (e.g., the Internet). Used like <code>curl http://xkcd.com/</code> it downloads the xkcd main page and displays the HTML source code. The pipe <code>|</code> in the script attaches the output of the command before the pipe to the input of the command after the pipe. Both commands are executed concurrently. Bash is a popular shell for Unix-like operating systems. The line <code>curl "$1" | bash</code> tries to download a file from the network and to execute the download directly. <!--- Although this is a common practice for conveniently installing software, it is considered extremely insecure and should never be done. [ed. note: there's no reason this cannot be secure, especially if HTTPS with validated certificates is used, from a trusted domain which utilizes DNSSEC] -->
  
git clone <nowiki>https://github.com/</nowiki>"$1"/"$1" && (cd "$1"; ./configure; make; sudo make install) &
+
The use of & at the end of each line causes the shell interpreter to execute the commands in parallel (asynchronously) instead of sequentially. Even if single commands fail, the rest of them will be executed. Note this is even the case for the final commands which attempt to change to the installed package, probably the only reason why this may not work completely for packages that do need compiling after being downloaded. (However, just running this script again would probably do the trick.)
  
Since all commands are running in the background, any command that requires user input will stop and wait until brought to the foreground. A common request would be for a database password, or if it is allowed to restart services for the installation. This could lead to packages being only partly installed or configured. (See more about using "yes" below.)
+
There appears to be a bug with the & at the end of the "git clone" line; since a git repository typically contains program source code, not executables, it may have been intended to retrieve the source code with git and then compile and install the program in the next line. In this case, the single & should be replaced with &&, an operator that will run the second command only if the first one has completed successfully. This plays into a second bug on the "configure" line, where the placement of the & means that only the "make install" command will be run asynchronously after the "configure" and "make" steps have finished in sequence. To make success as likely as possible, the two lines should be like this:
  
===Title text===
+
git clone <nowiki>https://github.com/</nowiki>"$1"/"$1" && (cd "$1"; ./configure; make; make install) &
The title text mentions the possibility that the same program may be in multiple repositories. In this case, the script may download and install several different versions of the same software. This would likely only create a confusing install-base, as an operating system would tend to prefer to use one version over another regardless of which one functions. It is unlikely that different repositories include the same software, with the exception of pip/easy_install and the two forms of apt-get, which would each likely see the software is already installed and abort. A way to fix this would be to use if statements. An example of this is [https://github.com/TheOddCell/Universal-Install-Script/blob/main/install.sh here].
 
  
The title text also mentions that adding a way of automatically saying "yes" to questions asked during the different repository-fetching programs' running, by making them read input from another program that writes a (nearly) endless stream of "y"s, could simplify things further. This would not work for any {{w|curses (programming library)|curses}}-based menus, or to answer any more complicated questions. Adding <code>2>/dev/null</code> to a command redirects the second output stream (the "error stream") to the null device driver, which discards all writes to it, meaning errors (the package not existing) will not be sent to the screen.
+
The title text mentions the possibility that the same program may be in multiple repositories, so in this case, the script will download and install several versions, or it may fail on a number of repositories, in which case usually nothing bad happens. Since all the commands come from different operating systems, versions, or distributions, it is not very likely that more than one will work (with the exception of pip/easy_install and the two forms of apt-get) or even exist on the same system. It mentions that adding a way of automatically saying "yes" to questions asked during the different repository-fetching programs' running, by making them read input from another program that writes a (nearly) endless stream of "y"s, could simplify things further. <code>2>/dev/null</code> redirects the second output stream (the "error stream") to the null device driver, which discards all writes to it, meaning errors (the package not existing) will be ignored.
  
 
==Transcript==
 
==Transcript==
Line 66: Line 62:
 
*pkg is the package management tool on {{w|Berkeley Software Distribution|BSD systems}}
 
*pkg is the package management tool on {{w|Berkeley Software Distribution|BSD systems}}
 
*apt-get is the package management tool of {{w|Debian}} and derivatives (e.g. Ubuntu)
 
*apt-get is the package management tool of {{w|Debian}} and derivatives (e.g. Ubuntu)
*steamcmd refers to the command line version of {{w|Steam (software)|Steam}}, the computer game client
+
*steamcmd refers to {{w|Stream (software)|Steam}}, the computer game client
 
*git is the revision control software used for many projects and gained a lot of traction through the {{w|GitHub}} platform
 
*git is the revision control software used for many projects and gained a lot of traction through the {{w|GitHub}} platform
 
*configure/make/make install refers to the standard way of compiling software from source (on Linux/Unix)
 
*configure/make/make install refers to the standard way of compiling software from source (on Linux/Unix)
 
*curl is a tool for loading data via http:// (i.e. from a website), this data is then pushed to the shell interpreter (in order to install)
 
*curl is a tool for loading data via http:// (i.e. from a website), this data is then pushed to the shell interpreter (in order to install)
 
**Note: While this is a security nightmare, some projects (like Homebrew) still use it as the preferred or only method of installation.
 
**Note: While this is a security nightmare, some projects (like Homebrew) still use it as the preferred or only method of installation.
* a similar comic is [[1987]] which concerns only Python.
 
  
 
{{comic discussion}}
 
{{comic discussion}}
[[Category:Linux]]
+
 
 
[[Category:Programming]]
 
[[Category:Programming]]
[[Category:Version Control]]
 
[[Category:Comics with lowercase text]]
 

Please note that all contributions to explain xkcd may be edited, altered, or removed by other contributors. If you do not want your writing to be edited mercilessly, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource (see explain xkcd:Copyrights for details). Do not submit copyrighted work without permission!

To protect the wiki against automated edit spam, we kindly ask you to solve the following CAPTCHA:

Cancel | Editing help (opens in new window)