![\"Redirects](\"../../images/carpentries-cli/redirects-and-pipes.svg\")
grep to select lines from text files that match simple patterns.\n- Use find to find files and directories whose names match simple patterns.\n- Use the output of one command as the command-line argument(s) to another command.\n- Explain what is meant by 'text' and 'binary' files, and why many common tools don't handle the latter well.\n\n**Time Estimation: 2H**\nThis tutorial will walk you through the basics of how to use the Unix command line.
\n\n\nComment\nThis tutorial is significantly based on the Carpentries “The Unix Shell” lesson, which is licensed CC-BY 4.0. Adaptations have been made to make this work better in a GTN/Galaxy environment.
\n
\n\nAgenda\nIn this tutorial, we will cover:
\n\n
Now that we know a few basic commands,\nwe can finally look at the shell’s most powerful feature:\nthe ease with which it lets us combine existing programs in new ways.\nWe’ll start with the directory called shell-lesson-data/molecules\nthat contains six files describing some simple organic molecules.\nThe .pdb extension indicates that these files are in Protein Data Bank format,\na simple text format that specifies the type and position of each atom in the molecule.
Let’s go into that directory with cd and run an example command wc cubane.pdb:
wc is the ‘word count’ command:\nit counts the number of lines, words, and characters in files (from left to right, in that order).
If we run the command wc *.pdb, the * in *.pdb matches zero or more characters,\nso the shell turns *.pdb into a list of all .pdb files in the current directory:
Note that wc *.pdb also shows the total number of all lines in the last line of the output.
If we run wc -l instead of just wc,\nthe output shows only the number of lines per file:
The -m and -w options can also be used with the wc command, to show\nonly the number of characters or the number of words in the files.
\n\n\nWhat happens if a command is supposed to process a file, but we\ndon’t give it a filename? For example, what if we type:
\n\n$ wc -l\nbut don’t type
\n*.pdb(or anything else) after the command?\nSince it doesn’t have any filenames,wcassumes it is supposed to\nprocess input given at the command prompt, so it just sits there and waits for us to give\nit some data interactively. From the outside, though, all we see is it\nsitting there: the command doesn’t appear to do anything.If you make this kind of mistake, you can escape out of this state by holding down\nthe control key (Ctrl) and typing the letter C once and\nletting go of the Ctrl key.\nCtrl+C
\n
Which of these files contains the fewest lines?\nIt’s an easy question to answer when there are only six files,\nbut what if there were 6000?\nOur first step toward a solution is to run the command:
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-9", "source": [ "wc -l *.pdb > lengths.txt" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">The greater than symbol, >, tells the shell to redirect the command’s output\nto a file instead of printing it to the screen. (This is why there is no screen output:\neverything that wc would have printed has gone into the\nfile lengths.txt instead.) The shell will create\nthe file if it doesn’t exist. If the file exists, it will be\nsilently overwritten, which may lead to data loss and thus requires\nsome caution.
\n\n\nYou can rewrite this using the tee command which writes out a file, while also showing the output to
\nstdout.\nwc -l *.pdb | tee lengths.txt\nOr you can use copy and paste to copy the
\n>character from the materials.
ls lengths.txt confirms that the file exists:
We can now send the content of lengths.txt to the screen using cat lengths.txt.\nThe cat command gets its name from ‘concatenate’ i.e. join together,\nand it prints the contents of files one after another.\nThere’s only one file in this case,\nso cat just shows us what it contains:
\n\n\nWe’ll continue to use
\ncatin this lesson, for convenience and consistency,\nbut it has the disadvantage that it always dumps the whole file onto your screen.\nMore useful in practice is the commandless,\nwhich you use withless lengths.txt.\nThis displays a screenful of the file, and then stops.\nYou can go forward one screenful by pressing the spacebar,\nor back one by pressingb. Pressqto quit.
Next we’ll use the sort command to sort the contents of the lengths.txt file.\nBut first we’ll use an exercise to learn a little about the sort command:
\n\nQuestion: What Does\nsort -nDo?The file
\nshell-lesson-data/numbers.txt\ncontains the following lines:\n10\n2\n19\n22\n6\nIf we run
\nsorton this file, the output is:\n10\n19\n2\n22\n6\nIf we run
\nsort -non the same file, we get this instead:\n2\n6\n10\n19\n22\nExplain why
\n-nhas this effect.\n👁 View solution
\n\nThe
\n-noption specifies a numerical rather than an alphanumerical sort.
We will also use the -n option to specify that the sort is\nnumerical instead of alphanumerical.\nThis does not change the file;\ninstead, it sends the sorted result to the screen:
We can put the sorted list of lines in another temporary file called sorted-lengths.txt\nby putting > sorted-lengths.txt after the command,\njust as we used > lengths.txt to put the output of wc into lengths.txt.\nOnce we’ve done that,\nwe can run another command called head to get the first few lines in sorted-lengths.txt:
Using -n 1 with head tells it that\nwe only want the first line of the file;\n-n 20 would get the first 20,\nand so on.\nSince sorted-lengths.txt contains the lengths of our files ordered from least to greatest,\nthe output of head must be the file with the fewest lines.
\n\n\nIt’s a very bad idea to try redirecting\nthe output of a command that operates on a file\nto the same file. For example:
\n\n$ sort -n lengths.txt > lengths.txt\nDoing something like this may give you\nincorrect results and/or delete\nthe contents of
\nlengths.txt.
\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-19", "source": [ "# Explore the possible solutions here!" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Question: What Does\n>>Mean?We have seen the use of
\n>, but there is a similar operator>>\nwhich works slightly differently.\nWe’ll learn about the differences between these two operators by printing some strings.\nWe can use theechocommand to print strings e.g.\n\nCode In: Bash\n\n$ echo The echo command prints text\n\n\nCode Out\n\nThe echo command prints text\nNow test the commands below to reveal the difference between the two operators:
\n\n\nCode In: Bash\n\n$ echo hello > testfile01.txt\nand:
\n\n\nCode In: Bash\n\n$ echo hello >> testfile02.txt\nHint: Try executing each command twice in a row and then examining the output files.
\n\n👁 View solution
\nSolution
\nIn the first example with
\n>, the string ‘hello’ is written totestfile01.txt,\nbut the file gets overwritten each time we run the command.We see from the second example that the
\n>>operator also writes ‘hello’ to a file\n(in this casetestfile02.txt),\nbut appends the string to the file if it already exists\n(i.e. when we run it for the second time).
\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-21", "source": [ "# Explore the possible solutions here!" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Question: Appending Data\nWe have already met the
\nheadcommand, which prints lines from the start of a file.\ntailis similar, but prints lines from the end of a file instead.Consider the file
\nshell-lesson-data/data/animals.txt.\nAfter these commands, select the answer that\ncorresponds to the fileanimals-subset.txt:\n$ head -n 3 animals.txt > animals-subset.txt\n$ tail -n 2 animals.txt >> animals-subset.txt\n\n
\n- The first three lines of
\nanimals.txt- The last two lines of
\nanimals.txt- The first three lines and the last two lines of
\nanimals.txt- The second and third lines of
\nanimals.txt\n👁 View solution
\n\nOption 3 is correct.\nFor option 1 to be correct we would only run the
\nheadcommand.\nFor option 2 to be correct we would only run thetailcommand.\nFor option 4 to be correct we would have to pipe the output ofheadintotail -n 2\nby doinghead -n 3 animals.txt | tail -n 2 > animals-subset.txt
In our example of finding the file with the fewest lines,\nwe are using two intermediate files lengths.txt and sorted-lengths.txt to store output.\nThis is a confusing way to work because\neven once you understand what wc, sort, and head do,\nthose intermediate files make it hard to follow what’s going on.\nWe can make it easier to understand by running sort and head together:
The vertical bar, |, between the two commands is called a pipe.\nIt tells the shell that we want to use\nthe output of the command on the left\nas the input to the command on the right.
This has removed the need for the sorted-lengths.txt file.
Nothing prevents us from chaining pipes consecutively.\nWe can for example send the output of wc directly to sort,\nand then the resulting output to head.\nThis removes the need for any intermediate files.
We’ll start by using a pipe to send the output of wc to sort:
We can then send that output through another pipe, to head, so that the full pipeline becomes:
This is exactly like a mathematician nesting functions like log(3x)\nand saying ‘the log of three times x’.\nIn our case,\nthe calculation is ‘head of sort of line count of *.pdb’.
The redirection and pipes used in the last few commands are illustrated below:
\nwc -l *.pdb will direct the output to the shell. wc -l *.pdb > lengths will\ndirect output to the file lengths. wc -l *.pdb | sort -n | head -n 1 will\nbuild a pipeline where the output of the wc command is the input to the sort\ncommand, the output of the sort command is the input to the head command and\nthe output of the head command is directed to the shell
\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-29", "source": [ "# Explore the possible solutions here!" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Question: Piping Commands Together\nIn our current directory, we want to find the 3 files which have the least number of\nlines. Which command listed below would work?
\n\n
\n- \n
wc -l * > sort -n > head -n 3- \n
wc -l * | sort -n | head -n 1-3- \n
wc -l * | head -n 3 | sort -n- \n
wc -l * | sort -n | head -n 3\n👁 View solution
\n\nOption 4 is the solution.\nThe pipe character
\n|is used to connect the output from one command to\nthe input of another.\n>is used to redirect standard output to a file.\nTry it in theshell-lesson-data/moleculesdirectory!
This idea of linking programs together is why Unix has been so successful.\nInstead of creating enormous programs that try to do many different things,\nUnix programmers focus on creating lots of simple tools that each do one job well,\nand that work well with each other.\nThis programming model is called ‘pipes and filters’.\nWe’ve already seen pipes;\na filter is a program like wc or sort\nthat transforms a stream of input into a stream of output.\nAlmost all of the standard Unix tools can work this way:\nunless told to do otherwise,\nthey read from standard input,\ndo something with what they’ve read,\nand write to standard output.
The key is that any program that reads lines of text from standard input\nand writes lines of text to standard output\ncan be combined with every other program that behaves this way as well.\nYou can and should write your programs this way\nso that you and other people can put those programs into pipes to multiply their power.
\n\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-31", "source": [ "# Explore the possible solutions here!" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Question: Pipe Reading Comprehension\nA file called
\nanimals.txt(in theshell-lesson-data/datafolder) contains the following data:\n2012-11-05,deer\n2012-11-05,rabbit\n2012-11-05,raccoon\n2012-11-06,rabbit\n2012-11-06,deer\n2012-11-06,fox\n2012-11-07,rabbit\n2012-11-07,bear\nWhat text passes through each of the pipes and the final redirect in the pipeline below?
\n\n$ cat animals.txt | head -n 5 | tail -n 3 | sort -r > final.txt\nHint: build the pipeline up one command at a time to test your understanding
\n\n👁 View solution
\n\nThe
\nheadcommand extracts the first 5 lines fromanimals.txt.\nThen, the last 3 lines are extracted from the previous 5 by using thetailcommand.\nWith thesort -rcommand those 3 lines are sorted in reverse order and finally,\nthe output is redirected to a filefinal.txt.\nThe content of this file can be checked by executingcat final.txt.\nThe file should contain the following lines:\n2012-11-06,rabbit\n2012-11-06,deer\n2012-11-05,raccoon\n
\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-33", "source": [ "# Explore the possible solutions here!" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Question: Pipe Construction\nFor the file
\nanimals.txtfrom the previous exercise, consider the following command:\n$ cut -d , -f 2 animals.txt\nThe
\ncutcommand is used to remove or ‘cut out’ certain sections of each line in the file,\nandcutexpects the lines to be separated into columns by a Tab character.\nA character used in this way is a called a delimiter.\nIn the example above we use the-doption to specify the comma as our delimiter character.\nWe have also used the-foption to specify that we want to extract the second field (column).\nThis gives the following output:\ndeer\nrabbit\nraccoon\nrabbit\ndeer\nfox\nrabbit\nbear\nThe
\nuniqcommand filters out adjacent matching lines in a file.\nHow could you extend this pipeline (usinguniqand another command) to find\nout what animals the file contains (without any duplicates in their\nnames)?\n👁 View solution
\n\n\n$ cut -d , -f 2 animals.txt | sort | uniq\n
\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-35", "source": [ "# Explore the possible solutions here!" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Question: Which Pipe?\nThe file
\nanimals.txtcontains 8 lines of data formatted as follows:\n2012-11-05,deer\n2012-11-05,rabbit\n2012-11-05,raccoon\n2012-11-06,rabbit\n...\nThe
\nuniqcommand has a-coption which gives a count of the\nnumber of times a line occurs in its input. Assuming your current\ndirectory isshell-lesson-data/data/, what command would you use to produce\na table that shows the total count of each type of animal in the file?\n
\n- \n
sort animals.txt | uniq -c- \n
sort -t, -k2,2 animals.txt | uniq -c- \n
cut -d, -f 2 animals.txt | uniq -c- \n
cut -d, -f 2 animals.txt | sort | uniq -c- \n
cut -d, -f 2 animals.txt | sort | uniq -c | wc -l\n👁 View solution
\n\nOption 4. is the correct answer.\nIf you have difficulty understanding why, try running the commands, or sub-sections of\nthe pipelines (make sure you are in the
\nshell-lesson-data/datadirectory).
Nelle has run her samples through the assay machines\nand created 17 files in the north-pacific-gyre/2012-07-03 directory described earlier.\nAs a quick check, starting from her home directory, Nelle types:
The output is 18 lines that look like this:
\n300 NENE01729A.txt\n300 NENE01729B.txt\n300 NENE01736A.txt\n300 NENE01751A.txt\n300 NENE01751B.txt\n300 NENE01812A.txt\n... ...\nNow she types this:
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-39", "source": [ "wc -l *.txt | sort -n | head -n 5" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Whoops: one of the files is 60 lines shorter than the others.\nWhen she goes back and checks it,\nshe sees that she did that assay at 8:00 on a Monday morning — someone\nwas probably in using the machine on the weekend,\nand she forgot to reset it.\nBefore re-running that sample,\nshe checks to see if any files have too much data:
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-41", "source": [ "wc -l *.txt | sort -n | tail -n 5" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Those numbers look good — but what’s that ‘Z’ doing there in the third-to-last line?\nAll of her samples should be marked ‘A’ or ‘B’;\nby convention,\nher lab uses ‘Z’ to indicate samples with missing information.\nTo find others like it, she does this:
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-43", "source": [ "ls *Z.txt" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Sure enough,\nwhen she checks the log on her laptop,\nthere’s no depth recorded for either of those samples.\nSince it’s too late to get the information any other way,\nshe must exclude those two files from her analysis.\nShe could delete them using rm,\nbut there are actually some analyses she might do later where depth doesn’t matter,\nso instead, she’ll have to be careful later on to select files using the wildcard expressions\nNENE*A.txt NENE*B.txt.
\n\nQuestion: Removing Unneeded Files\nSuppose you want to delete your processed data files, and only keep\nyour raw files and processing script to save storage.\nThe raw files end in
\n.datand the processed files end in.txt.\nWhich of the following would remove all the processed data files,\nand only the processed data files?\n
\n- \n
rm ?.txt- \n
rm *.txt- \n
rm * .txt- \n
rm *.*\n👁 View solution
\n\n\n
\n- This would remove
\n.txtfiles with one-character names- This is correct answer
\n- The shell would expand
\n*to match everything in the current directory,\nso the command would try to remove all matched files and an additional\nfile called.txt- The shell would expand
\n*.*to match all files with any extension,\nso this command would delete all files
Loops are a programming construct which allow us to repeat a command or set of commands\nfor each item in a list.\nAs such they are key to productivity improvements through automation.\nSimilar to wildcards and tab completion, using loops also reduces the\namount of typing required (and hence reduces the number of typing mistakes).
\nSuppose we have several hundred genome data files named basilisk.dat, minotaur.dat, and\nunicorn.dat.\nFor this example, we’ll use the creatures directory which only has three example files,\nbut the principles can be applied to many many more files at once. First, go\ninto the creatures directory.
The structure of these files is the same: the common name, classification, and updated date are\npresented on the first three lines, with DNA sequences on the following lines.\nLet’s look at the files:
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-47", "source": [ "head -n 5 basilisk.dat minotaur.dat unicorn.dat" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">We would like to print out the classification for each species, which is given on the second\nline of each file.\nFor each file, we would need to execute the command head -n 2 and pipe this to tail -n 1.\nWe’ll use a loop to solve this problem, but first let’s look at the general form of a loop:
for thing in list_of_things\ndo\n operation_using $thing # Indentation within the loop is not required, but aids legibility\ndone\nand we can apply this to our example like this:
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-49", "source": [ "for filename in basilisk.dat minotaur.dat unicorn.dat\n", "do\n", " head -n 2 $filename | tail -n 1\n", "done" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">\n\n\nThe shell prompt changes from
\n$to>and back again as we were\ntyping in our loop. The second prompt,>, is different to remind\nus that we haven’t finished typing a complete command yet. A semicolon,;,\ncan be used to separate two commands written on a single line.
When the shell sees the keyword for,\nit knows to repeat a command (or group of commands) once for each item in a list.\nEach time the loop runs (called an iteration), an item in the list is assigned in sequence to\nthe variable, and the commands inside the loop are executed, before moving on to\nthe next item in the list.\nInside the loop,\nwe call for the variable’s value by putting $ in front of it.\nThe $ tells the shell interpreter to treat\nthe variable as a variable name and substitute its value in its place,\nrather than treat it as text or an external command.
In this example, the list is three filenames: basilisk.dat, minotaur.dat, and unicorn.dat.\nEach time the loop iterates, it will assign a file name to the variable filename\nand run the head command.\nThe first time through the loop,\n$filename is basilisk.dat.\nThe interpreter runs the command head on basilisk.dat\nand pipes the first two lines to the tail command,\nwhich then prints the second line of basilisk.dat.\nFor the second iteration, $filename becomes\nminotaur.dat. This time, the shell runs head on minotaur.dat\nand pipes the first two lines to the tail command,\nwhich then prints the second line of minotaur.dat.\nFor the third iteration, $filename becomes\nunicorn.dat, so the shell runs the head command on that file,\nand tail on the output of that.\nSince the list was only three items, the shell exits the for loop.
\n\n\nHere we see
\n>being used as a shell prompt, whereas>is also\nused to redirect output.\nSimilarly,$is used as a shell prompt, but, as we saw earlier,\nit is also used to ask the shell to get the value of a variable.If the shell prints
\n>or$then it expects you to type something,\nand the symbol is a prompt.If you type
\n>or$yourself, it is an instruction from you that\nthe shell should redirect output or get the value of a variable.
When using variables it is also\npossible to put the names into curly braces to clearly delimit the variable\nname: $filename is equivalent to ${filename}, but is different from\n${file}name. You may find this notation in other people’s programs.
We have called the variable in this loop filename\nin order to make its purpose clearer to human readers.\nThe shell itself doesn’t care what the variable is called;\nif we wrote this loop as:
or:
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-53", "source": [ "for temperature in basilisk.dat minotaur.dat unicorn.dat\n", "do\n", " head -n 2 $temperature | tail -n 1\n", "done" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">it would work exactly the same way.
\nPrograms are only useful if people can understand them,\nso meaningless names (like x) or misleading names (like temperature)\nincrease the odds that the program won’t do what its readers think it does.
\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-55", "source": [ "# Explore the possible solutions here!" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Question: Variables in Loops\nThis exercise refers to the
\nshell-lesson-data/moleculesdirectory.\nlsgives the following output:\ncubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb\nWhat is the output of the following code?
\n\nfor datafile in *.pdb\ndo\n ls *.pdb\ndone\nNow, what is the output of the following code?
\n\nfor datafile in *.pdb\ndo\n ls $datafile\ndone\nWhy do these two loops give different outputs?
\n\n👁 View solution
\n\nThe first code block gives the same output on each iteration through\nthe loop.\nBash expands the wildcard
\n*.pdbwithin the loop body (as well as\nbefore the loop starts) to match all files ending in.pdb\nand then lists them usingls.\nThe expanded loop would look like this:\n$ for datafile in cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb\n> do\n> ls cubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb\n> done\n\ncubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb\ncubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb\ncubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb\ncubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb\ncubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb\ncubane.pdb ethane.pdb methane.pdb octane.pdb pentane.pdb propane.pdb\nThe second code block lists a different file on each loop iteration.\nThe value of the
\ndatafilevariable is evaluated using$datafile,\nand then listed usingls.\ncubane.pdb\nethane.pdb\nmethane.pdb\noctane.pdb\npentane.pdb\npropane.pdb\n
\n\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-57", "source": [ "# Explore the possible solutions here!" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Question: Limiting Sets of Files\nWhat would be the output of running the following loop in thei\n
\nshell-lesson-data/moleculesdirectory?\nfor filename in c*\ndo\n ls $filename\ndone\n\n
\n- No files are listed.
\n- All files are listed.
\n- Only
\ncubane.pdb,octane.pdbandpentane.pdbare listed.- Only
\ncubane.pdbis listed.\n👁 View solution
\n\n4 is the correct answer.
\n*matches zero or more characters, so any file name starting with\nthe letter c, followed by zero or more other characters will be matched.How would the output differ from using this command instead?
\n\nfor filename in *c*\ndo\n ls $filename\ndone\n\n
\n- The same files would be listed.
\n- All the files are listed this time.
\n- No files are listed this time.
\n- The files
\ncubane.pdbandoctane.pdbwill be listed.- Only the file
\noctane.pdbwill be listed.👁 View solution
\n\n4 is the correct answer.
\n*matches zero or more characters, so a file name with zero or more\ncharacters before a letter c and zero or more characters after the letter c will be matched.
\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-59", "source": [ "# Explore the possible solutions here!" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Question: Saving to a File in a Loop - Part One\nIn the
\nshell-lesson-data/moleculesdirectory, what is the effect of this loop?\nfor alkanes in *.pdb\ndo\n echo $alkanes\n cat $alkanes > alkanes.pdb\ndone\n\n
\n- Prints
\ncubane.pdb,ethane.pdb,methane.pdb,octane.pdb,pentane.pdband\npropane.pdb, and the text frompropane.pdbwill be saved to a file calledalkanes.pdb.- Prints
\ncubane.pdb,ethane.pdb, andmethane.pdb, and the text from all three files\nwould be concatenated and saved to a file calledalkanes.pdb.- Prints
\ncubane.pdb,ethane.pdb,methane.pdb,octane.pdb, andpentane.pdb,\nand the text frompropane.pdbwill be saved to a file calledalkanes.pdb.- None of the above.
\n\n👁 View solution
\n\n1 is correct. The text from each file in turn gets written to the
\nalkanes.pdbfile.\nHowever, the file gets overwritten on each loop iteration, so the final content ofalkanes.pdb\nis the text from thepropane.pdbfile.
\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-61", "source": [ "# Explore the possible solutions here!" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Question: Saving to a File in a Loop - Part Two\nAlso in the
\nshell-lesson-data/moleculesdirectory,\nwhat would be the output of the following loop?\nfor datafile in *.pdb\ndo\n cat $datafile >> all.pdb\ndone\n\n
\n- All of the text from
\ncubane.pdb,ethane.pdb,methane.pdb,octane.pdb, and\npentane.pdbwould be concatenated and saved to a file calledall.pdb.- The text from
\nethane.pdbwill be saved to a file calledall.pdb.- All of the text from
\ncubane.pdb,ethane.pdb,methane.pdb,octane.pdb,pentane.pdb\nandpropane.pdbwould be concatenated and saved to a file calledall.pdb.- All of the text from
\ncubane.pdb,ethane.pdb,methane.pdb,octane.pdb,pentane.pdb\nandpropane.pdbwould be printed to the screen and saved to a file calledall.pdb.\n👁 View solution
\n\n3 is the correct answer.
\n>>appends to a file, rather than overwriting it with the redirected\noutput from a command.\nGiven the output from thecatcommand has been redirected, nothing is printed to the screen.
Let’s continue with our example in the shell-lesson-data/creatures directory.\nHere’s a slightly more complicated loop:
The shell starts by expanding *.dat to create the list of files it will process.\nThe loop body\nthen executes two commands for each of those files.\nThe first command, echo, prints its command-line arguments to standard output.\nFor example:
prints:
\nhello there\nIn this case,\nsince the shell expands $filename to be the name of a file,\necho $filename prints the name of the file.\nNote that we can’t write this as:
because then the first time through the loop,\nwhen $filename expanded to basilisk.dat, the shell would try to run basilisk.dat as a program.\nFinally,\nthe head and tail combination selects lines 81-100\nfrom whatever file is being processed\n(assuming the file has at least 100 lines).
\n\n\nSpaces are used to separate the elements of the list\nthat we are going to loop over. If one of those elements\ncontains a space character, we need to surround it with\nquotes, and do the same thing to our loop variable.\nSuppose our data files are named:
\n\nred dragon.dat\npurple unicorn.dat\nTo loop over these files, we would need to add double quotes like so:
\n\n$ for filename in \"red dragon.dat\" \"purple unicorn.dat\"\n> do\n> head -n 100 \"$filename\" | tail -n 20\n> done\nIt is simpler to avoid using spaces (or other special characters) in filenames.
\nThe files above don’t exist, so if we run the above code, the
\nheadcommand will be unable\nto find them, however the error message returned will show the name of the files it is\nexpecting:\nhead: cannot open ‘red dragon.dat’ for reading: No such file or directory\nhead: cannot open ‘purple unicorn.dat’ for reading: No such file or directory\nTry removing the quotes around
\n$filenamein the loop above to see the effect of the quote\nmarks on spaces. Note that we get a result from the loop command for unicorn.dat\nwhen we run this code in thecreaturesdirectory:\nhead: cannot open ‘red’ for reading: No such file or directory\nhead: cannot open ‘dragon.dat’ for reading: No such file or directory\nhead: cannot open ‘purple’ for reading: No such file or directory\nCGGTACCGAA\nAAGGGTCGCG\nCAAGTGTTCC\n...\n
We would like to modify each of the files in shell-lesson-data/creatures, but also save a version\nof the original files, naming the copies original-basilisk.dat and original-unicorn.dat.\nWe can’t use:
cp *.dat original-*.dat\nbecause that would expand to:
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-69", "source": [ "cp basilisk.dat minotaur.dat unicorn.dat original-*.dat" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">This wouldn’t back up our files, instead we get an error.
\nThis problem arises when cp receives more than two inputs. When this happens, it\nexpects the last input to be a directory where it can copy all the files it was passed.\nSince there is no directory named original-*.dat in the creatures directory we get an\nerror.
Instead, we can use a loop:
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-71", "source": [ "for filename in *.dat\n", "do\n", " cp $filename original-$filename\n", "done" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">This loop runs the cp command once for each filename.\nThe first time,\nwhen $filename expands to basilisk.dat,\nthe shell executes:
cp basilisk.dat original-basilisk.dat\nThe second time, the command is:
\ncp minotaur.dat original-minotaur.dat\nThe third and last time, the command is:
\ncp unicorn.dat original-unicorn.dat\nSince the cp command does not normally produce any output, it’s hard to check\nthat the loop is doing the correct thing.\nHowever, we learned earlier how to print strings using echo, and we can modify the loop\nto use echo to print our commands without actually executing them.\nAs such we can check what commands would be run in the unmodified loop.
The following diagram\nshows what happens when the modified loop is executed, and demonstrates how the\njudicious use of echo is a good debugging technique.
Nelle is now ready to process her data files using goostats.sh —\na shell script written by her supervisor.\nThis calculates some statistics from a protein sample file, and takes two arguments:
Since she’s still learning how to use the shell,\nshe decides to build up the required commands in stages.\nHer first step is to make sure that she can select the right input files — remember,\nthese are ones whose names end in ‘A’ or ‘B’, rather than ‘Z’.\nStarting from her home directory, Nelle types:
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-73", "source": [ "cd ~/Desktop/shell-lesson-data/north-pacific-gyre/2012-07-03\n", "for datafile in NENE*A.txt NENE*B.txt\n", "do\n", " echo $datafile\n", "done" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Her next step is to decide\nwhat to call the files that the goostats.sh analysis program will create.\nPrefixing each input file’s name with ‘stats’ seems simple,\nso she modifies her loop to do that:
She hasn’t actually run goostats.sh yet,\nbut now she’s sure she can select the right files and generate the right output filenames.
\n\n\nTyping in commands over and over again is becoming tedious,\nthough,\nand Nelle is worried about making mistakes,\nso instead of re-entering her loop,\nshe presses ↑.\nIn response,\nthe shell redisplays the whole loop on one line\n(using semi-colons to separate the pieces):
\n\nfor datafile in NENE*A.txt NENE*B.txt; do echo $datafile stats-$datafile; done\n
Using the left arrow key,\nNelle backs up and changes the command echo to bash goostats.sh:
![\"The](\"../../images/carpentries-cli/shell_script_for_loop_flow_chart.svg\")
for datafile in NENE*A.txt NENE*B.txt; do bash goostats.sh $datafile stats-$datafile; done\nWhen she presses Enter,\nthe shell runs the modified command.\nHowever, nothing appears to happen — there is no output.\nAfter a moment, Nelle realizes that since her script doesn’t print anything to the screen\nany longer, she has no idea whether it is running, much less how quickly.\nShe kills the running command by typing Ctrl+C,\nuses ↑ to repeat the command,\nand edits it to read:
\nfor datafile in NENE*A.txt NENE*B.txt; do echo $datafile; bash goostats.sh $datafile stats-$datafile; done\n\n\n\nWe can move to the beginning of a line in the shell by typing Ctrl+A\nand to the end using Ctrl+E.
\n
When she runs her program now,\nit produces one line of output every five seconds or so:
\n1518 times 5 seconds,\ndivided by 60,\ntells her that her script will take about two hours to run.\nAs a final check,\nshe opens another terminal window,\ngoes into north-pacific-gyre/2012-07-03,\nand uses cat stats-NENE01729B.txt\nto examine one of the output files.\nIt looks good,\nso she decides to get some coffee and catch up on her reading.
\n\n\nAnother way to repeat previous work is to use the
\nhistorycommand to\nget a list of the last few hundred commands that have been executed, and\nthen to use!123(where ‘123’ is replaced by the command number) to\nrepeat one of those commands. For example, if Nelle types this:\n$ history | tail -n 5\n\n456 ls -l NENE0*.txt\n 457 rm stats-NENE01729B.txt.txt\n 458 bash goostats.sh NENE01729B.txt stats-NENE01729B.txt\n 459 ls -l NENE0*.txt\n 460 history\nthen she can re-run
\ngoostats.shonNENE01729B.txtsimply by typing\n!458. This number will be different for you, you should check your history before running it!
\n\n\nThere are a number of other shortcut commands for getting at the history.
\n\n
\n- Ctrl+R enters a history search mode ‘reverse-i-search’ and finds the\nmost recent command in your history that matches the text you enter next.\nPress Ctrl+R one or more additional times to search for earlier matches.\nYou can then use the left and right arrow keys to choose that line and edit\nit then hit Return to run the command.
\n- \n
!!retrieves the immediately preceding command\n(you may or may not find this more convenient than using ↑)- \n
!$retrieves the last word of the last command.\nThat’s useful more often than you might expect: after\nbash goostats.sh NENE01729B.txt stats-NENE01729B.txt, you can type\nless !$to look at the filestats-NENE01729B.txt, which is\nquicker than doing ↑ and editing the command-line.
\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-77", "source": [ "# Explore the possible solutions here!" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Question: Doing a Dry Run\nA loop is a way to do many things at once — or to make many mistakes at\nonce if it does the wrong thing. One way to check what a loop would do\nis to
\nechothe commands it would run instead of actually running them.Suppose we want to preview the commands the following loop will execute\nwithout actually running those commands:
\n\ncd ~/Desktop/shell-lesson-data/pdb/\nfor datafile in *.pdb\ndo\n cat $datafile >> all.pdb\ndone\nWhat is the difference between the two loops below, and which one would we\nwant to run?
\n\n\nCode In: Version 1\n\nfor datafile in *.pdb\ndo\n echo cat $datafile >> all.pdb\ndone\n\n\nCode In: Version 2\n\nfor datafile in *.pdb\ndo\n echo \"cat $datafile >> all.pdb\"\ndone\n\n👁 View solution
\n\nThe second version is the one we want to run.\nThis prints to screen everything enclosed in the quote marks, expanding the\nloop variable name because we have prefixed it with a dollar sign.
\nThe first version appends the output from the command
\necho cat $datafile\nto the file,all.pdb. This file will just contain the list;\ncat cubane.pdb,cat ethane.pdb,cat methane.pdbetc.Try both versions for yourself to see the output! Be sure to open the\n
\nall.pdbfile to view its contents.
\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-79", "source": [ "# Explore the possible solutions here!" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Question: Nested Loops\nSuppose we want to set up a directory structure to organize\nsome experiments measuring reaction rate constants with different compounds\nand different temperatures. What would be the\nresult of the following code:
\n\nfor species in cubane ethane methane\ndo\n for temperature in 25 30 37 40\n do\n mkdir $species-$temperature\n done\ndone\n\n👁 View solution
\n\nWe have a nested loop, i.e. contained within another loop, so for each species\nin the outer loop, the inner loop (the nested loop) iterates over the list of\ntemperatures, and creates a new directory for each combination.
\nTry running the code for yourself to see which directories are created!
\n
In the same way that many of us now use ‘Google’ as a\nverb meaning ‘to find’, Unix programmers often use the\nword ‘grep’.\n‘grep’ is a contraction of ‘global/regular expression/print’,\na common sequence of operations in early Unix text editors.\nIt is also the name of a very useful command-line program.
\ngrep finds and prints lines in files that match a pattern.\nFor our examples,\nwe will use a file that contains three haiku taken from a\n1998 competition in Salon magazine. For this set of examples,\nwe’re going to be working in the writing subdirectory:
\n\n\nWe haven’t linked to the original haiku because\nthey don’t appear to be on Salon’s site any longer.\nAs Jeff Rothenberg said,\n‘Digital information lasts forever — or five years, whichever comes first.’\nLuckily, popular content often has backups.
\n
Let’s find lines that contain the word ‘not’:
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-83", "source": [ "grep not haiku.txt" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Here, not is the pattern we’re searching for.\nThe grep command searches through the file, looking for matches to the pattern specified.\nTo use it type grep, then the pattern we’re searching for and finally\nthe name of the file (or files) we’re searching in.
The output is the three lines in the file that contain the letters ‘not’.
\nBy default, grep searches for a pattern in a case-sensitive way.\nIn addition, the search pattern we have selected does not have to form a complete word,\nas we will see in the next example.
\nLet’s search for the pattern: ‘The’.
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-85", "source": [ "grep The haiku.txt" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">This time, two lines that include the letters ‘The’ are outputted,\none of which contained our search pattern within a larger word, ‘Thesis’.
\nTo restrict matches to lines containing the word ‘The’ on its own,\nwe can give grep with the -w option.\nThis will limit matches to word boundaries.
Later in this lesson, we will also see how we can change the search behavior of grep\nwith respect to its case sensitivity.
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-87", "source": [ "grep -w The haiku.txt" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Note that a ‘word boundary’ includes the start and end of a line, so not\njust letters surrounded by spaces.\nSometimes we don’t\nwant to search for a single word, but a phrase. This is also easy to do with\ngrep by putting the phrase in quotes.
We’ve now seen that you don’t have to have quotes around single words,\nbut it is useful to use quotes when searching for multiple words.\nIt also helps to make it easier to distinguish between the search term or phrase\nand the file being searched.\nWe will use quotes in the remaining examples.
\nAnother useful option is -n, which numbers the lines that match:
Here, we can see that lines 5, 9, and 10 contain the letters ‘it’.
\nWe can combine options (i.e. flags) as we do with other Unix commands.\nFor example, let’s find the lines that contain the word ‘the’.\nWe can combine the option -w to find the lines that contain the word ‘the’\nand -n to number the lines that match:
Now we want to use the option -i to make our search case-insensitive:
Now, we want to use the option -v to invert our search, i.e., we want to output\nthe lines that do not contain the word ‘the’.
If we use the -r (recursive) option,\ngrep can search for a pattern recursively through a set of files in subdirectories.
Let’s search recursively for Yesterday in the shell-lesson-data/writing directory:
grep has lots of other options. To find out what they are, we can type:
\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-103", "source": [ "# Explore the possible solutions here!" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Question: Using\ngrepWhich command would result in the following output:
\n\nand the presence of absence:\n\n
\n- \n
grep \"of\" haiku.txt- \n
grep -E \"of\" haiku.txt- \n
grep -w \"of\" haiku.txt- \n
grep -i \"of\" haiku.txt\n👁 View solution
\n\nThe correct answer is 3, because the
\n-woption looks only for whole-word matches.\nThe other options will also match ‘of’ when part of another word.
\n\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-105", "source": [ "# Explore the possible solutions here!" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">\n
grep’s real power doesn’t come from its options, though; it comes from\nthe fact that patterns can include wildcards. (The technical name for\nthese is regular expressions, which\nis what the ‘re’ in ‘grep’ stands for.) Regular expressions are both complex\nand powerful; if you want to do complex searches, please look at the lesson\non our website. As a taster, we can\nfind lines that have an ‘o’ in the second position like this:\n$ grep -E \"^.o\" haiku.txt\n\nYou bring fresh toner.\nToday it is not working\nSoftware is like that.\nWe use the
\n-Eoption and put the pattern in quotes to prevent the shell\nfrom trying to interpret it. (If the pattern contained a*, for\nexample, the shell would try to expand it before runninggrep.) The\n^in the pattern anchors the match to the start of the line. The.\nmatches a single character (just like?in the shell), while theo\nmatches an actual ‘o’.
\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-107", "source": [ "# Explore the possible solutions here!" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Question: Tracking a Species\nLeah has several hundred\ndata files saved in one directory, each of which is formatted like this:
\n\n2013-11-05,deer,5\n2013-11-05,rabbit,22\n2013-11-05,raccoon,7\n2013-11-06,rabbit,19\n2013-11-06,deer,2\nShe wants to write a shell script that takes a species as the first command-line argument\nand a directory as the second argument. The script should return one file called
\nspecies.txt\ncontaining a list of dates and the number of that species seen on each date.\nFor example using the data shown above,rabbit.txtwould contain:\n2013-11-05,22\n2013-11-06,19\nPut these commands and pipes in the right order to achieve this:
\n\ncut -d : -f 2\n>\n|\ngrep -w $1 -r $2\n|\n$1.txt\ncut -d , -f 1,3\nHint: use
\nman grepto look for how to grep text recursively in a directory\nandman cutto select more than one field in a line.An example of such a file is provided in
\nshell-lesson-data/data/animal-counts/animals.txt\n👁 View solution
\n\n\ngrep -w $1 -r $2 | cut -d : -f 2 | cut -d , -f 1,3 > $1.txt\nActually, you can swap the order of the two cut commands and it still works. At the\ncommand line, try changing the order of the cut commands, and have a look at the output\nfrom each step to see why this is the case.
\nYou would call the script above like this:
\n\n$ bash count-species.sh bear .\n
\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-109", "source": [ "# Explore the possible solutions here!" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Question: Little Women\nYou and your friend, having just finished reading Little Women by\nLouisa May Alcott, are in an argument. Of the four sisters in the\nbook, Jo, Meg, Beth, and Amy, your friend thinks that Jo was the\nmost mentioned. You, however, are certain it was Amy. Luckily, you\nhave a file
\nLittleWomen.txtcontaining the full text of the novel\n(shell-lesson-data/writing/data/LittleWomen.txt).\nUsing aforloop, how would you tabulate the number of times each\nof the four sisters is mentioned?Hint: one solution might employ\nthe commands
\ngrepandwcand a|, while another might utilize\ngrepoptions.\nThere is often more than one way to solve a programming task, so a\nparticular solution is usually chosen based on a combination of\nyielding the correct result, elegance, readability, and speed.\n👁 View solution
\n\n\nfor sis in Jo Meg Beth Amy\ndo\n\techo $sis:\n\tgrep -ow $sis LittleWomen.txt | wc -l\ndone\nAlternative, slightly inferior solution:
\n\nfor sis in Jo Meg Beth Amy\ndo\n\techo $sis:\n\tgrep -ocw $sis LittleWomen.txt\ndone\nThis solution is inferior because
\ngrep -conly reports the number of lines matched.\nThe total number of matches reported by this method will be lower if there is more\nthan one match per line.Perceptive observers may have noticed that character names sometimes appear in all-uppercase\nin chapter titles (e.g. ‘MEG GOES TO VANITY FAIR’).\nIf you wanted to count these as well, you could add the
\n-ioption for case-insensitivity\n(though in this case, it doesn’t affect the answer to which sister is mentioned\nmost frequently).
While grep finds lines in files,\nthe find command finds files themselves.\nAgain,\nit has a lot of options;\nto show how the simplest ones work, we’ll use the directory tree shown below.
Nelle’s writing directory contains one file called haiku.txt and three subdirectories:\nthesis (which contains a sadly empty file, empty-draft.md);\ndata (which contains three files LittleWomen.txt, one.txt and two.txt);\nand a tools directory that contains the programs format and stats,\nand a subdirectory called old, with a file oldtool.
For our first command,\nlet’s run find . (remember to run this command from the shell-lesson-data/writing folder).
As always,\nthe . on its own means the current working directory,\nwhich is where we want our search to start.\nfind’s output is the names of every file and directory\nunder the current working directory.\nThis can seem useless at first but find has many options\nto filter the output and in this lesson we will discover some\nof them.
The first option in our list is\n-type d that means ‘things that are directories’.\nSure enough,\nfind’s output is the names of the five directories in our little tree\n(including .):
Notice that the objects find finds are not listed in any particular order.\nIf we change -type d to -type f,\nwe get a listing of all the files instead:
Now let’s try matching by name:
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-117", "source": [ "find . -name *.txt" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">We expected it to find all the text files,\nbut it only prints out ./haiku.txt.\nThe problem is that the shell expands wildcard characters like * before commands run.\nSince *.txt in the current directory expands to haiku.txt,\nthe command we actually ran was:
find did what we asked; we just asked for the wrong thing.
To get what we want,\nlet’s do what we did with grep:\nput *.txt in quotes to prevent the shell from expanding the * wildcard.\nThis way,\nfind actually gets the pattern *.txt, not the expanded filename haiku.txt:
\n\n\n\n
lsandfindcan be made to do similar things given the right options,\nbut under normal circumstances,\nlslists everything it can,\nwhilefindsearches for things with certain properties and shows them.
As we said earlier,\nthe command line’s power lies in combining tools.\nWe’ve seen how to do that with pipes;\nlet’s look at another technique.\nAs we just saw,\nfind . -name \"*.txt\" gives us a list of all text files in or below the current directory.\nHow can we combine that with wc -l to count the lines in all those files?
The simplest way is to put the find command inside $():
When the shell executes this command,\nthe first thing it does is run whatever is inside the $().\nIt then replaces the $() expression with that command’s output.\nSince the output of find is the four filenames ./data/one.txt, ./data/LittleWomen.txt,\n./data/two.txt, and ./haiku.txt, the shell constructs the command:
which is what we wanted.\nThis expansion is exactly what the shell does when it expands wildcards like * and ?,\nbut lets us use any command we want as our own ‘wildcard’.
It’s very common to use find and grep together.\nThe first finds files that match a pattern;\nthe second looks for lines inside those files that match another pattern.\nHere, for example, we can find PDB files that contain iron atoms\nby looking for the string ‘FE’ in all the .pdb files above the current directory:
\n\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "id": "cell-129", "source": [ "# Explore the possible solutions here!" ], "cell_type": "code", "execution_count": null, "outputs": [], "metadata": { "attributes": { "classes": [ ">Question: Matching and Subtracting\nThe
\n-voption togrepinverts pattern matching, so that only lines\nwhich do not match the pattern are printed. Given that, which of\nthe following commands will find all files in/datawhose names\nend ins.txtbut whose names also do not contain the stringnet?\n(For example,animals.txtoramino-acids.txtbut notplanets.txt.)\nOnce you have thought about your answer, you can test the commands in theshell-lesson-data\ndirectory.\n
\n- \n
find data -name \"*s.txt\" | grep -v net- \n
find data -name *s.txt | grep -v net- \n
grep -v \"net\" $(find data -name \"*s.txt\")- None of the above.
\n\n👁 View solution
\n\nThe correct answer is 1. Putting the match expression in quotes prevents the shell\nexpanding it, so it gets passed to the
\nfindcommand.Option 2 is incorrect because the shell expands
\n*s.txtinstead of passing the wildcard\nexpression tofind.Option 3 is incorrect because it searches the contents of the files for lines which\ndo not match ‘net’, rather than searching the file names.
\n
\n\n\nWe have focused exclusively on finding patterns in text files. What if\nyour data is stored as images, in databases, or in some other format?
\nA handful of tools extend
\ngrepto handle a few non text formats. But a\nmore generalizable approach is to convert the data to text, or\nextract the text-like elements from the data. On the one hand, it makes simple\nthings easy to do. On the other hand, complex things are usually impossible. For\nexample, it’s easy enough to write a program that will extract X and Y\ndimensions from image files forgrepto play with, but how would you\nwrite something to find values in a spreadsheet whose cells contained\nformulas?A last option is to recognize that the shell and text processing have\ntheir limits, and to use another programming language.\nWhen the time comes to do this, don’t be too hard on the shell: many\nmodern programming languages have borrowed a lot of\nideas from it, and imitation is also the sincerest form of praise.
\n
The Unix shell is older than most of the people who use it. It has\nsurvived so long because it is one of the most productive programming\nenvironments ever created — maybe even the most productive. Its syntax\nmay be cryptic, but people who have mastered it can experiment with\ndifferent commands interactively, then use what they have learned to\nautomate their work. Graphical user interfaces may be easier to use at\nfirst, but once learned, the productivity in the shell is unbeatable.\nAnd as Alfred North Whitehead wrote in 1911, ‘Civilization advances by\nextending the number of important operations which we can perform\nwithout thinking about them.’
\n\n\nQuestion:\nfindPipeline Reading ComprehensionWrite a short explanatory comment for the following shell script:
\n\nwc -l $(find . -name \"*.dat\") | sort -n\n\n👁 View solution
\n\n\n
\n- Find all files with a
\n.datextension recursively from the current directory- Count the number of lines each of these files contains
\n- Sort the output from step 2. numerically
\n
All of the commands you have run up until now were ad-hoc, interactive commands.
\n", "cell_type": "markdown", "metadata": { "editable": false, "collapsed": false } }, { "cell_type": "markdown", "id": "final-ending-cell", "metadata": { "editable": false, "collapsed": false }, "source": [ "# Key Points\n\n", "- `wc` counts lines, words, and characters in its inputs.\n", "- `cat` displays the contents of its inputs.\n", "- `sort` sorts its inputs.\n", "- `head` displays the first 10 lines of its input.\n", "- `tail` displays the last 10 lines of its input.\n", "- `command > [file]` redirects a command's output to a file (overwriting any existing content).\n", "- `command >> [file]` appends a command's output to a file.\n", "- `[first] | [second]` is a pipeline: the output of the first command is used as the input to the second.\n", "- The best way to use the shell is to use pipes to combine simple single-purpose programs (filters).\n", "- A `for` loop repeats commands once for every thing in a list.\n", "- Every `for` loop needs a variable to refer to the thing it is currently operating on.\n", "- Use `$name` to expand a variable (i.e., get its value). `${name}` can also be used.\n", "- Do not use spaces, quotes, or wildcard characters such as '*' or '?' in filenames, as it complicates variable expansion.\n", "- Give files consistent names that are easy to match with wildcard patterns to make it easy to select them for looping.\n", "- Use the up-arrow key to scroll up through previous commands to edit and repeat them.\n", "- Use Ctrl+R to search through the previously entered commands.\n", "- Use `history` to display recent commands, and `![number]` to repeat a command by number.\n", "- `find` finds files with specific properties that match patterns.\n", "- `grep` selects lines in files that match patterns.\n", "- `--help` is an option supported by many bash commands, and programs that can be run from within Bash, to display more information on how to use these commands or programs.\n", "- `man [command]` displays the manual page for a given command.\n", "- `$([command])` inserts a command's output in place.\n", "\n# Congratulations on successfully completing this tutorial!\n\n", "Please [fill out the feedback on the GTN website](https://training.galaxyproject.org/training-material/topics/data-science/tutorials/cli-advanced/tutorial.html#feedback) and check there for further resources!\n" ] } ] }![\"A](\"../../images/carpentries-cli/find-file-tree.svg\")