I have a post-install script that adds a basic set of tools on an Ubuntu or Linux Mint. Warning, this installs PHP 7.

https://gist.github.com/derrekbertrand/7668a695911260dee0c8

Mounts a folder on a remote host using sshfs, pokes it every minute to make sure the connection doesn't break. Honestly, I've stopped having a need for this, as I've changed workflows.

https://gist.github.com/derrekbertrand/8911d178ebe23b15670c

I also have this which provisions an account on a server with nginx for the Laravel framework (mostly). Mainly for my internal use.

https://github.com/derrekbertrand/laravel-5-nginx

Same here. I've always been able to get by with bash, Python, and of course non-awk coreutils... I know there are a few sed/grep lines I've written that would have been more elegant with awk.

Maybe someday.

I have no scripts good for sharing, but I would like to share a simple tip: Check out the zenity, dialog, and osd_cat packages of your distribution.

Nitpick mode activated. I realize that you didn't necessarily write all of this, and it's purpose is limited so the code doesn't have to be perfect, but we're here so I figure we might as well point out improvements that can be made.

To make it so that you don't have to modify the source itself you can make those environment variables inherited from the environment, and defaulted if unset. That's how I'd prefer to do it. That way the source doesn't have to change at all and the actual "config" can be put in a separate file sourced by your shell.

VARNAME="${VARNAME:-default_value}";

If they aren't environment variables then it's probably a better idea to make them lowercase so that they'll be less likely to collide, and they'll stand out differently too.

VARNAME=value;
export VARNAME
# vs
varname=value;

There's basically no value in doing this (unless maybe you're going to modify PATH and want to make sure it doesn't screw with things). An absolute path can be useful for processes running with elevated privileges, but only if the absolute path is known. Querying the system for the absolute path is no different than just letting the system resolve it.

Note that this relies on $ being literal because of what follows. It's probably acceptable, but it made me question it. Using a strategy like Chris did to interpolate parts, especially repeating ones, might make it easier to understand. This is a pretty common way to simplify regular expressions.

wrdchr_re="a-zA-Z0-9";
bndchr_re="[${wrdchr_re}]";
midchr_re="[${wrdchr_re}\-]"
domprt_re="(${bndchr_re}|${bndchr_re}${midchr_re}*${bndchr_re})";
domain_re="^(${domprt_re}\.)*${domprt_re}\$";

I chose somewhat obscure names just to make them short to keep the composition in a single line/terminal.

Unless you're targeting various shells with varying capabilities you can probably reduce this to:

read -p "Would you like use ssl on this site (y/N)?" USESSL

Alternatively, I'd at least use "echo -n" so that the response was on the same line as the prompt.

The thing that should be quoted, $USESSL, is not and the thing that doesn't need to be quoted, the y, is. Or just use [[ and neither needs to be quoted.

Lastly, I'd reiterate quoting of variable expressions and subcommands.

Append:

Thanks, Chris! That pretty much takes the mystery out of the first awk program. I'm speculating that what looks like array references in the latter is actually storing data into a dictionary, but I'm not sure? It seems like it would help an awful lot to understand those awk programs if you began them with a comment describing the input format as best as you understand it, or even offer a short sample input for reference. For somebody familiar with the input source it's probably not very necessary, but since I've never used nbtscan I have no idea what to expect.

Disagreed.

Bash (and sh-variants in general) tends to be something that people don't start out caring about, but eventually their usefulness works their way into your life. I gradually developed a somewhat intermediate level of skill hacking in bash. If you do choose to do any hacking in bash I recommend joining #bash on freenode to ask for criticisms or help. They are extremely helpful, extremely wise, have a ton of readily linkable best practices at their disposal, and they can seriously improve the quality of your bash code and give you a lot more confidence in it.

I also have experience being put onto a system with an ancient sh variant. You don't know how much you'll miss the bells and whistles until they're gone. It was still refreshing to see how many features were still supported even then, but it was still a bit of a fight to get comfortable with it.

Before you all start changing PAGER to most: http://unix.stackexchange.com/a/81131. The Web site for it is incredibly bare. Most ... of it is "under construction". It's basically undocumented from the looks of things (note: I haven't installed it). In any case, less is pretty featureful already so I'm not really in need of a better pager. And apparently most does lack a few less features. Still, thanks for spreading the knowledge, Chris. I'll be watching for it next time it comes up to see if the game has changed. In the meantime, as with probably most people, LESS is set to FRSX. If yours isn't Google why maybe it should be.

Thanks for the explanation of the 2nd awk program. It makes sense. It also helps me to see where inspiration for some Perl features may have come from.

As for subnets I think it's guaranteed to be 1's and then 0's (never mixed). The subnet is supposed to tell you which bits of the address are network address versus host address. And it wouldn't make sense to mix them up.

Awk can do many things. There's nothing it can't do, or nothing that would surprise me, after seeing someone make a raycaster with it...

But of the good bits of Unix, I think shell scripts are... terrible? They're a great idea, but the implementation is straight out of the 80s at best (and not in a good way).

It probably wouldn't be too difficult to make a simple shell that leverages a powerful scripting language for personal use. For example, I somewhat like the idea PowerShell (programs/actions being objects), but the syntax is even worse than any popular Unix shell... I've thought about it, but I can barely get out of bed so it's definitely out of my league right now.

(A shell that properly embraces and extends my favorite scripting language, Lua, would be pretty neat...)

On the contrary, Unix shell languages are timeless. They're more likely to be straight out of the 70s with most features and the 60s for others, but I digress. This was at a time when they understood well how best to communicate with computers. It's pretty interesting to see just how many things they got right compared with future generations and how their ideas from those origin days still remain the best ideas today.

PowerShell sounds like a great idea in theory. Communicate with objects? How cool! The idea really breaks down when you realize how complicated that makes everything. The brilliance of the standard stream interface is that any two programs can talk to each other regardless of whether or not they were designed to do so. In fact, they could have been written in completely isolated systems without ever knowing of each other's existence, and later you can interface them together easily either directly (a | b) or indirectly (a | c | b), if necessary.

My understanding of the PowerShell interface is that programs essentially need to be explicitly written with specific object interfaces in and out. While nothing should theoretically prevent you from writing that c program to transform one object into another one, consider the verbosity of statically-typed OOP and how that's supposed to fit onto a command line!

In practice, c is usually not some custom program a_to_b, but rather it's awk ..., sed ... or perl ... oneliners hacked right there on the command line in 30 to 60 characters or so. Certainly more lengthy solutions exist, and often they are indeed written to disk as a separate command (or module), but the power exists because the communication interface for every program is a stream of bytes (and that stream could very well represent an object, a file, text, or anything else).

Since I am predominantly a command-line user I always operate from a command shell when I can, even in Windows. A few years back I made the conscious choice to just bite the bullet and switch from cmd.exe to PowerShell. And I spent a good 6 months exclusively in PowerShell trying to figure out how to use it effectively (note: cmd.exe has always been pretty terrible so I was very anxious for something better). I still had MinGW/MSYS in my PATH so I was able to use the UNIX-like tools from the PowerShell console. What I found was that learning PowerShell commands and invoking them was extremely difficult compared to the UNIX-like tools I already had at my disposal. There was no benefit to learning the PowerShell commands, and in fact they lacked in power compared to what the standard shell interfaces already had. The console program (i.e., the window/emulator/etc.) itself was very disappointing in that it had no features that cmd.exe didn't have.

PowerShell cmdlets are not just a man page or Google away from figuring out some command and wiring it up. Instead, it is a painful process of trying to learn and understand an entire custom API for every command, and battling frustratingly with the fact that they just can't do certain things. IIRC, there's no proper concept of STDERR. They had some other custom error thing that worked way differently. It just made for very clumsy, looooooooong commands that barely worked. Things that you'd expect to work don't. And the shell lacked features that UNIX shells have had for nearly 50 years... I doubt that has improved much in the past 5 years. In general, it's a massive failure in my eyes.

It has been a long while since I used it or cared so it's possible that things or at least documentation has changed, but for a taste of what I mean see here: http://stackoverflow.com/questions/4998173/how-do-i-write-to-standard-error-in-powershell. The friendly PowerShell users trying to help just don't grasp the simplicity of what this guy is trying to do. Eventually, they're basically like, "well, you can't do that, ... but this is close...?"

I don't recall now where I got this idea, but I seem to recall reading that Microsoft was never setting out with the knowledge of UNIX-like shells to do a better job of it. As usual, they wanted incompatibility and vendor-lock in. Their motivation was apparently to write a brand new shell to become a standard of sorts. For embedded devices or CMOS systems or who knows what...

You'll note before PowerShell came "Windows Script Host" which allowed for "shell scripting" to be done in VBScript or JavaScript largely dependent on another Redmond atrocity, COM objects. These are terrible interfaces, despite JavaScript actually being a pretty great programming language.

I'd argue that if you find bash somehow deficient in features or clumsy then you just haven't spent enough time with it. This would be a fantastic thread to pose some questions or complaints and see if our collective knowledge can enlighten you.

Append:

Wow. That's awesome. I tried it. It was hard. I failed.

I just started touching Powershell a few months ago. I really liked it. It was a god-send trying to debug an exchange server. Grab all messages in the queue, filter them by these fields, and send then to the screen in a nice table. It worked great. I did some other stuff too. Nothing needed hardcore conversions like my typical awk program though. And EVERYTHING was self-documenting, including the data structures.

The thing is, powershell can link to almost everything that a GUI can in windows now. COM, WMI, and .NET all can be called. Microsoft provides a unified interface, and powershell easily links up to those interfaces. It also easily returns class objects with methods, and can easily be linked up with .NET sub-programs. To Microsoft's credit, Windows has very clear boundaries for their interfaces so adding a new interface or scripting language is a simple, entirely encapsulated job.

However, to Bam's issue, scripting tends to involve badly written programs (or programs used outside their intended purpose), and using only serialized data lets you survive that (BUT NOT ALWAYS) and keep going. However, we're also missing out because we HAVE to constantly write scripts that take that serialized data, unserialize it to work on it using REGEX, and then re-serialize it for the next program. At BEST we're running either each-line-is-field, or CSV where each row is a UNIT of fields separated by commas. But what happens when a program spits out data every TWO lines? Or worse, sometimes two, and if there's more data available, THREE or more? What happens when we're dumping multi-line text with special characters? (My living nightmare happens.)

I JUST had to write an awk program to parse nmap's output for host discovery. It dumps the host name, it dumps the type of "I'm here!" acknowledgement found (ARP vs TCP, etc), and IF it finds a MAC address it dumps that too. All of that crap is inbetween lines of description text and thanks to the MAC, the number of lines per "object" is variable! They actually say DON'T parse their text output in their man page. You should actually be using... and here it is... their XML output option.

OH WAIT. How the hell would you get BASH, a language without methods and member variables (AFAIK) to parse XML?

So not only do you have this huge productivity hit from the impedience mismatch. But any data you don't intentionally capture, is lost. And since shell programs can output differing data you might not even SEE some of the edge cases when you're building your data capture script.

To me--and I'd actually be willing to put work into doing this--the ideal Bash world would have the main syntax unchanged for compatibility-reasons, but an OO extension that directly supported XML, JSON, and classes with built-in serializer/deserializers as constructors or something. (So you can write your own, but they're clearly marked as entry and exit points in the class structure.)

I see no reason why most UNIX commands shouldn't have a -JSON command. That format is not going away and it's way more legible and space-efficient than XML. Although, MOST programs don't even have an XML format. Also, I think all programs should have a SCHEMA section of the man page but Powershell is so amazing you can just store the output in to a $object and then just start poking around and reading the object's fields to learn the output.

To me, PowerShell is much more "new user" friendly. You don't need a manpage for a lot of things the way Bash scripting does because the OO-nature lets you gleam a lot more information from a proper, rigid, known architecture. "Oh. Those are the methods, those are the variables and they're all clearly named." as opposed to "Oh god, what is this gibberish text screen supposed to mean?" Every one of us has seen a 300-column CSV where you constantly have to look back at the header line to see what field it is until you eventually give up and load a GUI CSV parser. (But when Excel/OpenOffice crash on very-large CSVs, then you're stuck with strange, crappy, shareware CSV readers. That happened, it sucked.)

Lastly, one thing that surprises me is how many people consider C++ a good language, and OO proper modern methodology, but would never consider using it in Bash/command-line scripting. Now, I'm not saying it should ever be REQUIRED, but as an option, I can't think of any good reason to oppose it.

zsh is pretty magical. I haven't been able to get into it yet except an hour or two. I can't decide whether I like it or not though.

ALSO, I forgot to mention: How do you serialize a program when it's written so poorly that it dumps error messages to STDOUT instead of STDERR? (The horror... the horror...)

Nmap::Parser! Tada! Perl has you covered (untested). Seriously, considering the nature of the work that you're constantly doing you should give Perl a close look. There's tons of modules for solving these kinds of problems. The motto in Perl is to not shell out (as you might in bash), but rather to prefer CPAN modules which already do the shelling out for you and have wrapped the command in a safety layer. Not only is there in this particular case a module that already knows the schema for nmap, but there are also modules for parsing XML or JSON with varying degrees of complexity or simplicity as required for cases where no module exists. For simple cases you could do a one-liner, and for more complex cases write an actual program. Perl isn't the only such platform that will have modules like this, but it does have a long history of being a sysadmin's best friend so there will likely already be many modules for that space. And where there isn't you can create your own!

I think you're asking too much of bash. Its primary purpose is as a shell language. Invoking commands and wiring them up to the user, file system, and other commands. It's not meant to do everything the best. There are existing tools that will always do that better than bash ever could. And really, by the very nature of a command shell, you can already "extend" the bash shell by just writing new commands that do what you want. For example, write a command that allowed you to simply parse and process XML data.

And maybe one that lets you translate an XML structure into JSON, and a separate one that allows you to easily extract data from JSON. Google first, these probably exist.

Well it appears to work. Here's a simple test script that just dumps the raw blessed objects of online "hosts". To keep things simple it separates nmap command line options from hosts (e.g., IP addresses) using the standard -- option.

bambams@sephiroth:~$ perl nmap.pl -- castopulence.org
$VAR1 = bless( {
  'addrs' => {
    'ipv4' => '64.85.162.126'
  },
  'distance' => undef,
  'hostnames' => [
    'castopulence.org',
    'b03s17le.corenetworks.net'
  ],
  'hostscript' => undef,
  'ipidsequence' => undef,
  'os' => undef,
  'ports' => {
    'extraports' => {
      'count' => '994',
      'state' => 'closed'
    },
    'tcp' => {
      '113' => {
        'service' => {
          'confidence' => '3',
          'extrainfo' => undef,
          'fingerprint' => undef,
          'method' => 'table',
          'name' => 'ident',
          'port' => '113',
          'product' => undef,
          'proto' => 'unknown',
          'rpcnum' => undef,
          'script' => undef,
          'tunnel' => undef,
          'version' => undef
        },
        'state' => 'open'
      },
      '22' => {
        'service' => {
          'confidence' => '3',
          'extrainfo' => undef,
          'fingerprint' => undef,
          'method' => 'table',
          'name' => 'ssh',
          'port' => '22',
          'product' => undef,
          'proto' => 'unknown',
          'rpcnum' => undef,
          'script' => undef,
          'tunnel' => undef,
          'version' => undef
        },
        'state' => 'open'
      },
      '443' => {
        'service' => {
          'confidence' => '3',
          'extrainfo' => undef,
          'fingerprint' => undef,
          'method' => 'table',
          'name' => 'https',
          'port' => '443',
          'product' => undef,
          'proto' => 'unknown',
          'rpcnum' => undef,
          'script' => undef,
          'tunnel' => undef,
          'version' => undef
        },
        'state' => 'open'
      },
      '554' => {
        'service' => {
          'confidence' => '3',
          'extrainfo' => undef,
          'fingerprint' => undef,
          'method' => 'table',
          'name' => 'rtsp',
          'port' => '554',
          'product' => undef,
          'proto' => 'unknown',
          'rpcnum' => undef,
          'script' => undef,
          'tunnel' => undef,
          'version' => undef
        },
        'state' => 'filtered'
      },
      '80' => {
        'service' => {
          'confidence' => '3',
          'extrainfo' => undef,
          'fingerprint' => undef,
          'method' => 'table',
          'name' => 'http',
          'port' => '80',
          'product' => undef,
          'proto' => 'unknown',
          'rpcnum' => undef,
          'script' => undef,
          'tunnel' => undef,
          'version' => undef
        },
        'state' => 'open'
      },
      '9000' => {
        'service' => {
          'confidence' => '3',
          'extrainfo' => undef,
          'fingerprint' => undef,
          'method' => 'table',
          'name' => 'cslistener',
          'port' => '9000',
          'product' => undef,
          'proto' => 'unknown',
          'rpcnum' => undef,
          'script' => undef,
          'tunnel' => undef,
          'version' => undef
        },
        'state' => 'open'
      }
    },
    'tcp_port_count' => 6,
    'udp_port_count' => 0
  },
  'status' => 'up',
  'tcpsequence' => undef,
  'tcptssequence' => undef,
  'trace' => {
    'hops' => []
  },
  'trace_error' => undef,
  'uptime' => undef
}, 'Nmap::Parser::Host' );

Requires List::MoreUtils and Nmap::Parser modules. It also requires Perl 5.22.xx or better, but that can be lifted by removing the appropriate use line. It will fall back on whatever the used modules require. Nevertheless, to install the modules, my advice is to install perlbrew:

cpan -L http://install.perlbrew.pl | bash

Follow the directions to source the environment (and add it to bashrc for later). Then install the latest stable perl (this takes a few minutes):

perlbrew install --as stable stable

When it's finished, assuming it went gracefully, install cpanm, switch to the new local perl environment, and install the dependencies.

perlbrew install-cpanm
perlbrew use stable
cpanm install List::MoreUtils Nmap::Parser

You can switch perlbrew instead of use if your account doesn't depend on running administrative tasks that rely on a particular environment. To be safe, I opted for use so you wouldn't post at 2 AM screaming that servers are down and I broke your system. Note: use only modifies your current environment so logging out and in would "restore" it. With either option, you can switch/use system to go back to the original PATH perl. If your environment is more complex than this then YMMV. I recommend familiarizing yourself with this stuff on a desktop rather than a production server for safety's sake...

The main thing that leads to Perl appearing "cryptic" is its use of sigils (grammatic symbols on variables/expressions) to express context. Unlike most other programming languages, Perl has a concept of plural expressed in the language itself which can change the meaning of an expression; the creator is a linguist. That combined with a few other interesting syntax variations which require learning Perl to understand (i.e., unlike something like Python, you probably wouldn't be able to read it without learning the language first). For example, the following sets of two statements are equivalent:

my @results1 = map { $_ + 2 } grep { /^[0-9]+$/ } @input;
my @results2 = map $_ + 2, grep /^[0-9]+$/, @input;

The first "parameter" to map and grep and family are code blocks, and they're magical in the sense that you can either pass a block (in braces) or an expression (note: no comma versus comma). You can think of both essentially turning into lambdas (anonymous functions) implicitly. So that one special expression is not evaluated and then the result passed into map or grep or friends, but instead is passed in as a chunk of code to be repeatedly called against the members of the list passed in. In Perl, a regular expression is implicitly applied to the default variable, $_. The grep code is equivalent to saying, "the current member matches this regular expression", i.e., is a positive integer. It can also be weird to a newcomer how the use of parenthesis is optional in a function call, as above. You could make the parenthesis explicit if you thought it reads better (but you'd be wrong):

my @results1 = map({ $_ + 2 } grep({ /^[0-9]+$/ } @input));
my @results2 = map($_ + 2, grep(/^[0-9]+$/, @input));

Note, still no commas between "arguments" in the block case. The code argument could also be passed explicitly as a subroutine reference (which is like a lambda). For example:

my $pos_int = sub { /^[0-9]+$/ };
my $plus_2 = sub { $_ + 2 };
my @results = map $plus_2, grep $pos_int, @input;

Whereas Python's motto is that there should only be one way to do it, as you can see, in Perl there's many ways to do it^[1]. This obviously can also lead to cryptic code since the styles can vary wildly and the reader obviously needs to understand all of this stuff for the code to make sense. That said, it gives a lot of expressive power to the author.

Perl is a very complex language with lots of exceptional syntax. It can sometimes be difficult to be sure that what you've tried to say is what you said. This is compounded by the fact that Perl was written as a replacement for things like awk and as such its default running mode is very relaxed. If variables don't exist they are implicitly ignored/created. If you don't quote a bareword where Perl thinks you meant a string it will magically become a string. This allows for very short, compact programs in oneliners on the command line, but also makes for difficult to debug programs. You generally need to enable strict and warnings pragmas to enforce "strict" syntax (so none of those things I just mentioned work) and to output warnings when you do something that is considered dangerous or obscure (which will print out a warning message and line number at run-time to warn you that you should change the code to be less reliant on such features).

If you're willing to learn Awk then I'd argue that you should learn Perl as well. Perl is like a super-awk/grep/sed/etc. all in one. There have been many additional features over time that improve the language a lot. Perl 6 has also just been released, which is a completely different language to Perl 5 (what people normally mean when they say Perl). Perl 6 is radically different, and has some really nice features, but it's too young to really know where it stands in practice. It will be years before Perl 5 begins to fade away so it's probably best to learn Perl 5 first and then if you feel inspired to then move on to Perl 6.