Wednesday, 9 April 2014

Extending OTP

This question came in an Elixir discussion but it is definitely not Elixir specific so I thought it might interest people.

Many people seem to think that OTP is set in stone, encased inside an armoured concrete block and practically impossible to do anything with without launching a major attack. So if it doesn't have just the feature you are looking for then there is not much you can do except grit your teeth and accept it or avoid OTP completely and roll your own instead. This is a fundamental faulty assumption.

First, however, what is OTP? It is many things: it is a large collection of libraries for doing things; a set of design principles and patterns for building systems and tools; and code, the behaviours, for implementing and supporting these principles and patterns. Most of these principles and patterns link together. So a system is a release, which consists of applications, which consist of code and processes at runtime organised into supervision tree. And at each level you can define how it is to manage the underlying systems. Many of the libraries are independent of the design patterns, you can do all the wonderful stuff with TCP without being in a behaviour.

OTP is actually quite an open system which is very amenable to development and extension. This is actually not surprising as it all implemented in Erlang and the Erlang ecosystem is a very egalitarian system: all modules are equal, all processes are equal, and all applications are equal. The only difference is who wrote them. There are of course some properties you can't change, for example how releases, applications and supervision trees interact, but within those limitations you can do a lot.

For example it is easy to create a new behaviour. So if you feel that gen_server does not give you the functionality you need or want then the easiest solution to create a new behaviour which has the right functionality. One way to this is to build your new behaviour on top of an existing behaviour. For example implement your behaviour on top of a gen_server using the gen_server callbacks to implement a base to call the callbacks in your new behaviour. There is nothing strange about this, it is done in many places. For example the supervisor is implemented as a gen_server behaviour. Doing it this way allows you to inherit a lot of functionality for free, the new behaviour will automatically follow all the rules.

Another way is of course to write the behaviour yourself. For example if the functionality you want does not fit nicely into an existing behaviour. This is also very easy, in some ways even easier than building on top of something else. All the tools, modules, which are used inside OTP are available. By using them and following a simple set of rules it easy to create a process which fits smoothly into a supervision and behaves exactly as it should to make OTP happy. Everything is also documented, look under the proc_lib and sys modules for starters.

Sometimes you don't need or want another behaviours then using these tools it of course easy to a create a singleton type of process. Again all the tools for doing this is there.

My point is that it is easy to extend OTP to provide the functionality you need without having to do any serious hacking of the system. Unfortunately too few people seem to know this. It is sometimes amazing to see the lengths people go to squeeze something into a gen_server when it would have been much easier to roll their own.

OTP was never meant to be the answer to (life, the universe and) everything, it "just" provides the basic building blocks and all the tools necessary to extend it. View the existing behaviours as a starter kit. So if you feel that OTP is lacking something then don't complain but fix it.


Sunday, 16 March 2014

Erlang/Elixir communities

After the keynote by José Valim and Dave Thomas at the Erlang Factory in San Francisco there were of course many discussions on the IRC Elixir channel. Some were advocating a split in the community into two separate communities. I personally think this would be a bad thing for a number of reasons:
  • The Erlang/Elixir combined community is not that large and splitting it into smaller parts would not help any of us. It would also be an easy target of derision: "look only 2 members and already 3 groups".
  • Both Erlang and Elixir are part of the Erlang "ecosystem" and not really independant. This also applies to other languages like LFE and Joxa which are also based on the fundamental principle that the interface between the language and OTP should be seemless. This could easily and unnecessarily confuse people who aren't in the know.
Fortunately, from my point of view, there were others who disagreed with this and would prefer to keep it as one community.


Monday, 20 January 2014

Erlang syntax again ... and again ... and again ...

Every once in a while there is a blogg complaining about Erlang and its syntax. To save you the trouble here are the main syntax complaints:

- The use of ';' (and ',' and '.') is strange, different and very confusing. I.e. it's not like Java.
- 'if' is very strange. I.e. it is not like Java.
- Variables must start with an uppercase letter. I.e. it is not like Java.
- The record syntax is very strange.

Actually only the record syntax is strange. Unfortunately, given Erlang and the original requirements for records it is not really possible to do better. So while many have complained, very few have accepted the challenge of coming up with something that looks better and works, and no-one has succeeded.

What I don't understand is when people want Erlang to look like XXXX, their language of choice. This idea I find very strange for many reasons:

- Unless the XXXX is Lisp it usually has a much more complex and messy syntax than Erlang. While the Erlang syntax may be different it is actually quite small, simple, regular and concise; much more so than the common choices of XXXX are.

- Usually XXXX has completely different semantics from Erlang. In most languages the syntax is (hopefully) tuned to its semantics, at least Erlang's is, so using XXXX syntax for Erlang would definitely be trying to fit a square peg in a round hole.

- A more philosophical reason is that I think if a language looks like XXXX it should behave like XXXX, not like something completely different. If it behaves differently it will just cause confusion and disappointment in the long run: "It looks like XXXX, but my XXXX code doesn't work the way it is supposed to". This is actually a valid criticism.

While I can understand people may dislike the syntax of a certain language, even I dislike some syntaxes, I don't understand people who say "I was going to learn Erlang but the syntax was so strange I quit". For me the syntax is the easiest part of learning a new language, it is really just a RTFM. At last count I have actively programmed in the following languages: Basic, Fortran, Pascal, C, Lisp, Prolog, Erlang, Lua and a number of different assembly languages. I am now teaching myself Python and Haskell. Lisp and Prolog are really groups of languages but they are so similar I class them together. I don't know whether to include Awk and Sh as well. All have different syntaxes. And I don't think I am in anyway unique in this as most programmers will find that they actually have used many languages.

My point is that the syntax is the easy part of learning a new language, just look it up in the manual. It is learning the semantics of the new language and how to use it efficiently to solve your problems which are the major difficulties. How do I structure my solution to best make use of the language and its environment? This is where major rethinks will occur. This is what takes time to learn and understand. Not in what it looks like.


P.S. Yes, I have implemented another syntax front-end for Erlang, but it is Lisp based so that is quite a natural thing to do. Lisp has such a simple and powerful syntax so how could I not do it. It is the syntaxes from other languages which are more complicated and messy.

Tuesday, 14 January 2014

LFE, Lisp and Erlang

My LFE (Lisp Flavoured Erlang), here and here, is not a Lisp-1 but a Lisp-2 as I think this fits better with Erlang. Well, it is really a Lisp-2+ which I will explain in a moment.

What are Lisp-1 and Lisp-2? Richard P. Gabriel in his paper defined the two as follows:
Lisp-1 has a single namespace that serves a dual role as the function namespace and value namespace; that is, its function namespace and value namespace are not distinct. In Lisp-1, the functional position of a form and the argument positions of forms are evaluated according to the same rules. Scheme is a Lisp-1 dialect.
Lisp-2 has distinct function and value namespaces. In Lisp-2, the rules for evaluation in the functional position of a form are distinct from those for evaluation in the argument positions of the form. Common Lisp is a Lisp-2 dialect.

What does this mean? For example take the expression (f a). In Lisp-1 you take the value of f as the function to call with the value of a as the argument, while in Lisp-2 you take the function of f to call with the value of a as the argument; you don't use the value of f. In Lisp-2 if you want to use the value of f as a function, for example if it a local variable, you can't do this directly but need to do something like (funcall f a) to do this, which is how it is done in LFE.

Lisp-1 is generally considered more elegant and simpler to understand, so why not use it for LFE? Well, this is where the properties of Erlang and the BEAM step in.

In Erlang you can define many top-level functions with the same name but with different arities, for example f/1, f/2 and f/3. While it was not strictly necessary to be able to do this in LFE, it is a basic property of Erlang which I felt was necessary to handle. Lisp-1 is clearly one value/function per name so it is difficult to extend the concept in a clean way. If we allow a symbol to have many functions at the top-level it seemed natural to be able to do this for local symbols/variables as well.

So in LFE we can both many top-level functions with the same name but different arities like Erlang:

(defun f (x) x)
(defun f (x y) (+ x y))
(defun f (x y z) (+ x y z))

And we can also define many local functions with the same name but different arities:

(flet ((f (x) x)
       (f (x y) (+ x y))
       (f (x y z) (+ x y z)))
  (list (f 5) (f 5 6) (f 5 6 7)))

=> (5 11 18)

I felt this was easier to do in a clean way if LFE was a Lisp-2.

This is what I meant in the beginning when I said LFE is a Lisp-2+, it not only has separate value and function space but it also allows multiple function definitions for the same name. Like Erlang does.