Erlang slave nodes and ssh login shells

The Erlang runtime environment is more similar to an operating system than to a traditional language runtime library. An Erlang "node" is an Erlang instance started with the flag -name (or -sname, for "short names" if your network does not rely on DNS). For example: 
$ erl -sname foo
will give you an interactive Erlang shell with a prompt like this:
Eshell V9.2.1  (abort with ^G)
(foo@rocka)1>
and you can see from the prompt that this is running as a "node" with the node name foo@rocka. In practice, this means that networking is enabled, allowing Erlang processes on this node to communicate with processes on other nodes, either on the same host machine or on other machines, through the ordinary Erlang message passing mechanism.

Working with multiple nodes

If I open a separate console and start another node bar@rocka in the same way, I can then connect these two and start doing interesting multi-node stuff:
(foo@rocka)1> net_adm:ping(bar@rocka).
pong
(foo@rocka)2> nodes().                                         
[bar@rocka]
For instance, if I have implemented a module mysrv that runs a simple counter server, I can load the code onto the other node like this - even if the nodes do not share a file system - and start it on that node.
(foo@rocka)3> BeamFile = code:which(mysrv).
(foo@rocka)4> {ok, BeamBin} = file:read_file(BeamFile).
(foo@rocka)5> rpc:call(bar@rocka, code, load_binary, [mysrv, "", BeamBin]).
(foo@rocka)6> Pid = spawn(bar@rocka, mysrv, run, []).
The last command should print something like <8112.88.0> where the non-zero leftmost number in the Pid indicates that it runs on another node than the local one. We can try to request numbers from the server to check that it is running: 
(foo@rocka)7> mysrv:next(Pid).
0
(foo@rocka)8> mysrv:next(Pid).
1
(foo@rocka)9> mysrv:next(Pid).
2
That's cool and all, but it requires that you first do the orchestration of starting the nodes on the various machines (and making sure they get restarted if need be). In some situations, it would be much nicer if one Erlang node could simply start another one, on demand. For example:
  • Having a node on one machine coordinate a number of worker nodes on a cluster of machines
  • Writing test suites for code that cooperates across two or more nodes
  • Protecting the main node from failure when running less dependable native code (NIFs or C drivers, or a WX-based GUI)
  • Not running certain operations in the same OS process context as the main Erlang node
And of course, you want to get notified if one of these nodes crashes, and you want them to disappear automatically if the main node dies.

Running slave nodes

Erlang has standard library support for starting and supervising such slave nodes, and it looks straightforward enough: 
  slave:start(Hostname, Nodeame, ErlangArgs)
for example: 
  slave:start("rocka", "baz", "-s io write hello")
For this to work, you need help from your operating system to actually get the new nodes running, and by default it is assumed that the command for doing this is rsh. (The original Unix rsh command is deprecated these days, but on many machines nowadays, the rsh name is simply a link to ssh.) Edit: the patch to switch to using ssh by default was merged upstream in February 2019 and should be included in the following major release, Erlang/OTP 22.

You can however pass a flag to Erlang to tell it to use any other command, like this: 
$ erl -sname foo -rsh /usr/bin/ssh
Easy! But there is a snag: the way this is implemented, you cannot pass additional flags to the command, and the default behaviour for ssh is to create a non-interactive non-login shell, if a command is specified.

This means that your attempt to start a node on a remote machine will probably fail because the command erl could not be found in the path, for a non-login shell (at least if you have standard .profile/.bashrc files). And -rsh "/usr/bin/ssh -l" does not work either, because the given string is treated by Erlang as a single path (which is allowed to contain spaces).

You may also be mystified by why the following might work: 
(foo@rocka)1> slave:start("rocka","baz", "").
{ok,baz@rocka}
while this might not: 
(foo@rocka)1> slave:start("localhost","baz", "").
bash: erl: command not found
and the reason is that if the given hostname matches the hostname of the local node, the use of the rsh command is bypassed, and the new node is simply launched as erl ... <UserArgs> instead of "<Rsh>" <Hostname> erl ... <UserArgs> - and you probably had erl in your current path already, so it just worked. Also note that there is no way of specifying an absolute path to the erl command - it has to be in the path.

To get around this, you could set up a specific environment on all the remote machines, either by customizing their ssh configurations, or by creating custom user accounts for running the nodes. But if you want it to "just work", using a regular user account with no special modifications, how can you get around these limitations?

Starting the nodes in a login shell


The answer is to use a simple wrapper script like this one, for rearranging the arguments and passing them on to ssh along with the -l flag: 
    #!/bin/bash
    host="$1"
    shift
    exec /usr/bin/ssh "$host" "/bin/bash -l -c '$@'"
Put the above in a file myrsh, make it executable with chmod a+x myrsh, and then start Erlang as follows: 
$ erl -sname foo -rsh myrsh
That's all! Now it should work, both for "localhost" and for any remote machine that you can log in to (only assuming you have Erlang installed): 
(foo@rocka)1> slave:start("localhost","baz", "").
{ok,baz@localhost}
You could augment the script as you please - for example, adding logging of the arguments being passed to the command, which was something I used quite a lot while debugging this stuff. The only tricky part of all of this was to figure out what was actually happening in the different cases, how the actual command line was constructed, whether there are any straightforward alternatives for getting ssh to behave like you want (not really), and whether your shell really should set up your user path only if it is a login shell (as far as I can tell, yes).

I hope this can be of help to anyone else who wants to try out the slave node feature, which understandably has been rather underused.

Comments

Post a Comment

Popular posts from this blog

The Hobbit (1982 game) graphics engine in Javascript

Tunnel like hall Great river Trolls' path Narrow place Dragon's desolation Straight smooth passage Dale valley Trolls' cave Forest gate Lake town Goblins' dungeon Dark dungeon Trolls' clearing Elvish clearing Lonelands Elvenking's cellar Goblins' big cavern Gloomy bewitched place Running river Lower halls Spider place Front gate One of the first computer games that really fascinated me was the 1982 text adventure based on The Hobbit, for the ZX Spectrum. I was 12 years old when I first played it, and it was immediately clear that this was different from any other game I'd seen. It was a text adventure, but some locations had illustrations in brightly coloured graphics, which although crude by today's standards really helped transport you to another world back in those days of mostly monochrome screens and text-only adventure games. There is much more to be said about what made the game so special - in fact, the graphics ...
Read more

Recreated ZX Spectrum (Bluetooth Keyboard)

Image
(Originally posted on Google+, Sep 30 2015.) So I got myself what has to be the nerdiest bluetooth keyboard in the world: http://sinclair.recreatedzxspectrum.com/ Quite expensive too, for a bluetooth keyboard. But build-wise, it's a perfect replica of the ZX Spectrum we know and love. Feels exactly right and has the correct key labels. Which means that used with a decent emulator, it recreates the feeling of typing on a real Speccy (without resorting to blind hunt-and-peck which is generally the case if you use a PC keyboard). And after some fiddling with figuring out how to unlock the full bluetooth QWERTY mode as well as the Speccy-specific mapping, I can confirm that it works as advertised as a generic keyboard for Android, iOS or Windows. I only had to ask myself, "can I think of anyone who's a more suitable target for this product" to realize that I really had to get it. It's certainly not for fast touch typing, but programming on a Speccy was more ...
Read more

The times they are a-changin'

Recently I got a call from my dad, who was having some problems with his computer, along the lines of "there's something wrong with the damn internet again". Since I was the one who set it up for him, I don't mind fixing problems when they occur, but I find that the hardest thing with these debugging sessions over the phone is to get him to quote the exact messages he is seeing. It's kind of interesting, as if the words and concepts are so alien to him that he zones out after only a second, and is actually unable to read them aloud. A typical exchange goes like this: -So, dad, what does the error message say? -Oh, something about the Internet. -OK, but what does the text say, precisely? -I dunno, Microsoft and Internet something. -Dad, can you please tell me exactly what the text says? -I just told you, it says it can't find the Internet. -Where is the message, I mean, what program does it come from? -Well, dammit, it's right here, on the screen! And so on...
Read more