├── .gitignore
├── Makefile
├── README.md
├── argus.html
├── argus.tex
├── emerald.html
├── emerald.pdf
├── emerald.tex
├── guesstimate.html
├── guesstimate.tex
├── hermes.html
├── hermes.pdf
├── hermes.tex
├── pl.bib
├── pl.html
├── pl.md
├── plits.html
├── plits.tex
├── pmldc.pdf
├── pmldc.tex
├── promises.html
├── promises.tex
├── rpc.html
├── rpc.pdf
├── rpc.tex
└── todo.tex
/.gitignore:
--------------------------------------------------------------------------------
1 | .texpadtmp
2 |
--------------------------------------------------------------------------------
/Makefile:
--------------------------------------------------------------------------------
1 | pandoc:
2 | pandoc promises.tex -o promises.html --bibliography pl.bib
3 | pandoc emerald.tex -o emerald.html --bibliography pl.bib
4 | pandoc hermes.tex -o hermes.html --bibliography pl.bib
5 | pandoc rpc.tex -o rpc.html --bibliography pl.bib
6 | pandoc plits.tex -o plits.html --bibliography pl.bib
7 | pandoc argus.tex -o argus.html --bibliography pl.bib
8 | pandoc guesstimate.tex -o guesstimate.html --bibliography pl.bib
9 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | ## Programming Models and Languages for Distributed Computing
2 |
3 | This repository is a work-in-progress curriculum on models and languages
4 | for distributed computing.
5 |
6 | I try to write one section a week, but I don't guarantee that because
7 | this is a side project to my Ph.D. research.
8 |
9 | If you use any of this content in a course or workshop, please let me
10 | know, I'd love to know where it's being used and reference it from this
11 | page.
12 |
13 | ### Book
14 |
15 | Download all of the posts here:
16 | https://github.com/cmeiklejohn/PMLDC/blob/master/pmldc.pdf
17 |
18 | ### Blog
19 |
20 | Or, view them on my blog:
21 | http://christophermeiklejohn.com
22 |
23 | ### Contributing
24 |
25 | TODO's on sections to be written are in `todo.tex`.
26 |
27 | ## Copyright
28 |
29 | Copyright 2016 Christopher Meiklejohn
30 |
--------------------------------------------------------------------------------
/argus.html:
--------------------------------------------------------------------------------
1 | Relevant Reading
2 |
3 | Abstraction Mechanisms in CLU, Liskov, Barbara and Snyder, Alan and Atkinson, Russell and Schaffert, Craig, CACM 1977 (Liskov et al. 1977).
Guardians and Actions: Linguistic Support for Robust, Distributed Programs, Liskov, Barbara and Scheifler, Robert, TOPLAS 1982 (Liskov and Scheifler 1983).
Orphan Detection in the Argus System, Walker, Edward Franklin, DTIC 1984 (Walker 1984).
Implementation of Argus, Liskov, Barbara and Curtis, Dorothy and Johnson, Paul and Scheifer, Robert, SIGOPS 1987 (Liskov et al. 1987).
Distributed Programming in Argus, Liskov, Barbara CACM 1988 (Liskov 1988).
9 |
10 |
11 | “However, regardless of advances in hardware, we believe atomic actions are necessary and are a natural model for a large class of applications. If the language does not provide actions, the user will be compelled to implement them, perhaps unwittingly reimplementing with each new application, and may implement them incorrectly.”
12 |
13 | Overview
14 | The focus of these papers is the ARGUS system (and it’s roots with the programming language CLU), developed in the Laboratory for Computer Science at the Massachusetts Institute of Technology. ARGUS was designed to solve the problem of programming language support for the construction and maintenance of distributed programs constructed from modules executing at, and communicating from, geographically distinct nodes. ARGUS was designed with the following goals in mind:
15 |
16 | Service: Programs should have localized failures, and be geographically distributed with replicated data for both fault-tolerance and availability.
Reconfiguration: Software should be able to be reconfigured while the system is running: this allows for the addition of additional capacity to increase processing power, decrease response times, or increase the availability of data.
Autonomy: Nodes in the system may be owned by individuals or organizations that need to control what data is replicated at the node for political or sociological reasons.
Distribution: Programs should be able to control explicit placement of data to ensure responsiveness and cost-effectiveness of hardware in the system.
Concurrency: Distribution should be able to exploit the available concurrency in the system.
Consistency: Consistency must be maintained, specifically for invariant preservation: for instance, conservation of funds during a transfer between two accounts.
23 | Of all of the aforementioned concerns, the authors posit that the most difficult of these to provide is consistency: consistency becomes much more difficult to preserve in a system where coordination is minimized to avoid interference and m ask failures of the network. Therefore, the authors present a technique for integrating atomicity as fundamental concept in a programming language.
24 | Atomicity and Actions
25 | Since data resiliency only guarantees consistency under a quiescent environment, it is necessary to make some operations in the system atomic. When the authors state atomic, they mean two things:
26 |
27 | Indivisibility: the execution of an activity in the system never appears to overlap with another activity in the system.
Recoverability: the overall effect of an activity is all or nothing: in the event of a failure, the activity must be completed after recovery or all objects must be restored to their initial state.
30 | ARGUS introduces actions: atomic transactions that need to either be committed or aborted and provide both the indivisibility and recoverability properties. If an action happens to abort, it is as if the the effects had never happened; if an action happens to commit, all modified objects in the system take on their new state. However, since ARGUS is a concurrent system, shared objects that will be manipulated by concurrent atomic actions must be done via synchronization, with read/and write locks.
31 | To achieve this, ARGUS introduces a type of atomic object, or as they call it, atomic abstract data types: sets of objects and primitive operations for interacting with those objects. These objects are similar to normal data types, but are extended with operations that ensure indivisibility and recoverability. Access to objects within an atomic abstract data type are done through locking: read/write locks with versioning is used to facilitate concurrency access in the system. Write operations operate against a copy of the current version, and when the action completes, the new version of the object is written, replacing the old.
32 | Nested Actions
33 | The authors argue for indivisibility as a desired property, even if it reduces the amount of potential concurrency in the system:
34 |
35 | “It has been argued that indivisibility is too strong a property for certain applications because it limits the amount of potential concurrency. We believe that indivisibility is the desired property for most application, if it is required only at the appropriate levels of abstraction. ARGUS provides a mechanism for user-defined atomic data types.”
36 |
37 | Nested actions, or subactions, can be used to further divide actions and introduce concurrency within an action: each action can contain any number of subactions that can be either executed sequentially or concurrently. Subactions can commit or abort independently and this behavior has no impact on the parent action. However, an abort of a parent will force the abort of all subactions.
38 | Nested actions provide support for the composition of actions into a larger action where some of the nested actions can run in parallel or might fail and have to be compensated for. For instance, if one subaction has to contact a remote guardian that may be unavailable, that subaction can abort and the operation retried against another remote guardian: the authors refer to this as “several ways to accomplish a task.”
39 | Because nested actions may fail, interact with several guardians, and have “checkpointing” behavior as they either abort or succeed, each nested actions ancestor needs to track metadata during the execution:
40 |
41 | Each subaction will inherit locks from its ancestor: read locks are directly inherited, but write locks can only be inherited if all read locks can also be inherited.
Each subaction can abort or succeed: if the action aborts, the ancestor will restore the state, or “checkpoint”, of the execution to before the operation was executed.
Each version created by a subaction needs to be propagated, to ensure that the effects of a previous write to a successful nested action are available to a subsequent subaction.
Each top-level action will track what’s referred to as a plist, which is a list of all of the guardians contacted during the execution of the action to be used in the final two-phase commit protocol to commit the action and write to stable storage.
46 | Remote Procedure Call and the Network
47 | The authors argue for Remote Procedure Call and state that nested actions can be used for masking the communication failures inherent in the paradigm:
48 |
49 | “In fact, we believe the form of communication that is needed is remote procedure call, with at-most-once semantics, namely, that (effectively) either the message is delivered and acts on exactly once, with exactly one reply received, or the message is never delivered and the sender is so informed.”
50 |
51 | The authors believe that low-level issues from the system should be shielded from the user, such as packet retransmission, but do believe that the system should make a reasonable attempt of delivery for messages. However, the users believe that the notion that long delays are possible and that ultimate failure is possible, should be made present to the user. The authors propose that the subaction should be used for this: individual subactions can abort without causing the entire parent action to abort: in the case where these subactions do abort, the user should be able to take action, such as try an alternative replica to service the request.
52 | In this model, some top-level actions can not be aborted. Consider the case of an airline reservation system: the clerk can initiate operations for making a reservation and subactions can take action and try multiple replicas if, for instance, the primary can not be contacted. However, an external event performed by a top-level action, such as printing a check, can not be undone and needs to have a compensating action for dealing with this behavior. To alleviate this problem, the authors suggest breaking these into two separate top-level actions, where they are sequenced to ensure the completion of one before the execution of the subsequent operation.
53 | Finally, the authors acknowledge that even in this model timeouts are necessary to resolve issues that might arise from contention or deadlocks between atomic resources.
54 | Semantics
55 | We won’t provide the full semantics of how the ARGUS language works, but provide a brief write up that should give you the general idea of how the system works.
56 | In ARGUS, distributed programs are composed of a group of guardians. Guardians encapsulate and control access to one or more resources and makes them available through operations called handlers, which are called by other guardians. Guardians contain both data and processes: process execute handlers and are spawned for each call to a handler by another guardian. Processes have access to objects that make up the state of the guardian and this is how access to objects is controlled.
57 | Guardians contain both stable and volatile objects: volatile objects are lost when nodes running a guardian fail. When a guardian fails, the language support system is responsible for recreating the guardian with the objects that were persisted to stable storage. Stable state is versioned when modifications are performed, and volatile objects live on the heap. Guardians are created dynamically and the node where the guardian runs is specified by the programmer: guardians represent a logical node of the system where communication is performed by handlers, an abstraction of state and the physical network.
58 | Processes in guardians execute concurrently and the language controls access to atomic objects using the locking mechanisms described above: in addition, a coroutining facility allows subactions to be further divided into concurrent executions, yielding more concurrency from the system.
59 | ARGUS provides static type checking at compile time and loading of new guardians and handlers while the system is running: to ensure this happens safely, types must be known by the system before loading a handler that uses these types. Guardians and handlers are first class and can be used as both arguments and return values from handler invocations.
60 | As the authors highlight, the major drawbacks of the system are in security and scheduling. ARGUS provides no mechanism for securing guardians from particular guardians or preventing nodes from launching certain types of guardians. ARGUS also has no mechanism for priority servicing of calls, backpressure, or load-shedding, leading the system susceptible to scheduling problems when overloaded.
61 | Finally, the authors acknowledge that concurrent programming is hard. While ARGUS goes to great efforts to simplify the problem of maintaining consistency in a distributed application, as highlighted by the banking example, the ARGUS system made it all to easy to introduce deadlocks by having concurrent processes in a guardian, or concurrent actions, take out read/write locks in the wrong order.
62 | Impact and Implementations
63 | ARGUS was arguably the first system to provide a distributed programming paradigm where consistency was integrated in the language as a first class concept. ARGUS built on top of CLU, and provided a series of abstractions directly assisting programmers in writing code that maintained consistency, as the authors felt this was the main challenge in distributed programming.
64 | ARGUS served as the platform where other ideas would blossom: both Remote Procedure Call was used as the primary abstraction for making remote calls, where subactions were used to mask omissions. The optimizations that Promises provided to pipeline RPCs and ensure ordering of requests and responses also were originally developed in the ARGUS system.
65 | The idea of compensating transactions in ARGUS for top-level actions that can not be “recovered” also appeared later in the SAGA design (Garcia-Molina and Salem 1987).
66 | Finally, languages like Bloom (Alvaro et al. 2011), while not providing abstractions for consistency, take an alternative approach by providing analysis tools for determining where consistency could be violated and cause program anomalies. The authors propose to plug in a distributed consensus mechanism, such as Zookeeper, to ensure consistency while performing these changes.
67 |
68 |
69 |
Alvaro, Peter, Neil Conway, Joseph M Hellerstein, and William R Marczak. 2011. “Consistency Analysis in Bloom: A Calm and Collected Approach.” In CIDR, 249–60. Citeseer.
70 |
71 |
72 |
Garcia-Molina, Hector, and Kenneth Salem. 1987. Sagas. Vol. 16. 3. ACM.
73 |
74 |
75 |
Liskov, Barbara. 1988. “Distributed Programming in Argus.” Communications of the ACM 31 (3). ACM: 300–312.
76 |
77 |
78 |
Liskov, Barbara, and Robert Scheifler. 1983. “Guardians and Actions: Linguistic Support for Robust, Distributed Programs.” ACM Transactions on Programming Languages and Systems (TOPLAS) 5 (3). ACM: 381–404.
79 |
80 |
81 |
Liskov, Barbara, Dorothy Curtis, Paul Johnson, and Robert Scheifer. 1987. “Implementation of Argus.” ACM SIGOPS Operating Systems Review 21 (5). ACM: 111–22.
82 |
83 |
84 |
Liskov, Barbara, Alan Snyder, Russell Atkinson, and Craig Schaffert. 1977. “Abstraction Mechanisms in Clu.” Communications of the ACM 20 (8). ACM: 564–76.
85 |
86 |
87 |
Walker, Edward Franklin. 1984. “Orphan Detection in the Argus System.” DTIC Document.
88 |
89 |
Relevant Reading
2 |
3 | The development of the Emerald programming language, Black, Andrew P and Hutchinson, Norman C and Jul, Eric and Levy, Henry M, HOPL 2007 (A. P. Black et al. 2007).
Distribution and Abstract Types in Emerald, A. Black and N. Hutchinson and E. Jul and H. Levy and L. Carter, IEEE 1987 (A. Black et al. 1987).
Emerald: A general-purpose programming language, Raj, Rajendra K. and Tempero, Ewan and Levy, Henry M. and Black, Andrew P. and Hutchinson, Norman C. and Jul, Eric, Software Practice and Experience, 1991 (Raj et al. 1991).
Object Structure in the Emerald System, Black, Andrew and Hutchinson, Norman and Jul, Eric and Levy, Henry, OOPLSA ’86 (Andrew Black et al. 1986).
Typechecking Polymorphism in Emerald, Black, Andrew P. and Hutchinson, Norman, Technical Report CRL 91/1, Digital Cambridge Research Laboratory, 1991.
Getting to Oz, Hank Levy, Norm Hutchinson, and Eric Jul, April 1984.
10 | These texts, and more, are available from the languages website, http://www.emeraldprogramminglanguage.org.
11 |
12 | The Eden Programming Language (EPL) was a distributed programming language developed on top of Concurrent Euclid (Holt 1982) that extended the existing language with support for remote method invocations. However, this support was far from ideal: incoming method invocation requests would have to be received and dispatched by a single thread while the programmer making the request would have to manually inspect error codes to ensure that the remote invocation succeeded.
13 | Eden also provided location-independent mobile objects, but the implementation was extremely costly. In the implementation, each object was a full Unix process that could send and receive messages to each other: these messages would be sent using interprocess communication if located on the same node, resulting in latencies in the milliseconds. Eden additionally implemented a “kernel” object for dispatching messages between processes, resulting in a single message between two objects on the same system taking over 100 milliseconds, the cost of two context switches at the time. To make applications developed in EPL more efficient, application developers would use a lightweight heap-based object implemented in Concurrent Euclid (that, appeared as a single Eden object consuming a single Unix process) for objects that needed to communicate, but were located on the same machine, that would communicate through shared memory. The next problem follows naturally: the single abstraction provided resulted in extremely slow applications, so, a new abstraction was provided to compensate leaving the user with two different object models.
14 | In a legendary memo entitled “Getting to Oz”, the language designers of Eden and the soon-to-be language designers began discussions to improve the design of Eden. This new language would be entitled “Emerald”.
15 | We enumerate here the list of specific goals the language designers had for Emerald, outside of the general improvements they wanted to make on Eden.
16 |
17 | Convinced distributed objects were a good idea and the right way to construct distributed programs, they sought to improve the performance of distributed objects.
Objects should stay relatively cost-free, for instance, if they do not take advantage of distribution: no-use, no-cost.
Simplify and reduce the dual object model, remove explicit dispatching, error handling and other warts in the Eden model.
To support the principle of information hiding and have a single semantics for both large and small, local or distributed, objects.
Distributed programs can fail: the network can be down, a service can be unavailable, and therefore a language for building distributed applications needs to provide the programmer tools for dealing with these failures.
Minimization of the language by removing many of the features seen in other languages and building abstractions that could be used to extend the language.
Object location needs to be explicit even as much as the authors wanted to follow the principle of information hiding as it directly impacts performance. Objects should be able to be moved, but moving an object should not change the operational semantics of the language.
25 | Impact and Implementations
26 | The technical innovations for Emerald, a system that was under primary development from 1983 to 1987 (and later continued by various graduate students) are numerous. We highlight a few of the most important technical innovations below:
27 |
28 | Emerald presented a single object model for both distributed and local objects. Each object has a globally unique identifier, internal state, and a set of methods that could be invoked. These objects could run in their own process, if necessary, or not. (In fact, objects encapsulated their processes and launched them on object invocation.)
Objects could exist with different implementations in this unified model: global objects, or objects that could be accessed either locally or remote; local objects, that were optimized for local access only as determined as best as possible at compile time; and direct, or objects that represented primitive types such as integers, booleans, etc.
Emerald was a statically-typed language, that had dynamic type checking for objects that were received over the wire. This was achieved using a notion of protocols and conformity-based typing. Dynamic type checking would be performed by ensuring types at runtime were compatible based on their interfaces. This was done by forming a type lattice and computing both the join and meet based on the abstract, or the compile-time provided type, and the concrete implementation that were provided at runtime.
Emerald’s type system also provided capabilities, where types could either be restricted to a higher type in the type lattice, or viewed at a lower type in the type lattice.
Synchronization between processes in Emerald was achieved using monitors to achieve mutual exclusion with condition signaling and waiting (Hoare 1974).
Mobility in Emerald was provided using explicit placement primitives. Processes could be moved to a new location, fixed at a precise location, and located. Emerald also provided two new parameter evaluation modes based on mobility: call-by-move, or move the parameter object to the invocation location, and call-by-visit, by remote access of the parameter object from the invocation’s location. When objects were moved, the old placement would store a forwarding address that would be used to route messages forward; timestamps were used to detect routing loops and reference the most recent object and to avoid stable storage of the forwarding addresses, reliable broadcast was used to find lost pointers.
Errors related to network availability were not considered exceptions; therefore special notation for handling these errors was provided to the programmer.
36 | Emerald (and Eden’s) influence on programming languages throughout the history of programming languages is paramount. Emerald specifically innovated in two main areas: distributed objects and type systems, which is interesting because the innovations in type systems were only done to support the development of distributed objects in the Emerald system.
37 | The idea of type conformity over a type lattice with both concrete and abstract types influenced the further development of protocols, mechanisms to specify the external behavior of an objects, in the ANSI 1997 Smalltalk standard.
38 | Developers of the Modula-3 Network Objects (Birrell et al. 1993) system took what they felt was the most essential and the best of both the Emerald and SOS (Shapiro et al. 1989) systems. This system forewent the mobility of objects in favor of marshalling. In the author’s own words:
39 |
40 | “We believe it is better to provide powerful marshaling than object mobility. The two facilities are similar, because both of them allow the programmer the option of communicating objects by reference or by copying. Either facility can be used to distribute data and computation as needed by applications. Object mobility offers slightly more flexibility, because the same object can be either sent by reference or moved; while with our system, network objects are always sent by reference and other objects are always sent by copying. However, this extra flexibility doesn’t seem to us to be worth the substantial increase in complexity of mobile objects.” (A. P. Black et al. 2007)
41 |
42 | Both Java’s RMI system and Jini (and their predecessor OMG’s CORBA) were influenced by Emerald as well, however not without a fierce discussion on the merits of distinguishing between remote and local method invocations, motivated by legendary technical report (Kendall et al. 1994) by Sun Microsystems’ research division.
43 | Jim Waldo, author of the aforementioned technical report, writes:
44 |
45 | “The RMI system (and later the Jini system) took many of the ideas pioneered in Emerald having to do with moving objects around the network. We introduced these ideas to allow us to deal with the problems found in systems like CORBA with type truncation (and which were dealt with in Network Objects by doing the closest-match); the result was that passing an object to a remote site resulted in passing [a copy of] exactly that object, including when necessary a copy of the code (made possible by Java bytecodes and the security mechanisms). This was exploited to some extent in the RMI world, and far more fully in the Jini world, making both of those systems more Emerald-like than we realized at the time.” (A. P. Black et al. 2007)
46 |
47 | The authors eventually come to a similar conclusion to many distributed systems practitioners today and other critics in their research area: that availability, reliability, and the network remain the paramount challenges and add fuel to the fire against any location-transparent semantics provided by mobile objects. These still remain as much of a challenge for the developers of distributed programs today, as they did in 1983 at the start of the development lineage from Eden to Emerald:
48 |
49 | “Mobile objects promise to make that same simplicity available in a distributed setting: the same semantics, the same parameter mechanisms, and so on. But this promise must be illusory. In a distributed setting the programmer must deal with issues of availability and reliability. So programmers have to replicate their objects, manage the replicas, and worry about ’one copy semantics’. Things are not so simple any more, because the notion of object identity supported by the programming language is no longer the same as the application’s notion of identity. We can make things simple only by giving up on reliability, fault tolerance, and availability — but these are the reasons that we build distributed systems.” (A. P. Black et al. 2007)
50 |
51 | The authors of Eden and Emerald express an extremely interesting point early on in their paper on the history of Emerald (A. P. Black et al. 2007): the reason for the poor abstractions requiring manual dispatch of method invocations to threads, and the explicit error handling from network anomalies, was because as researchers working on a language, they were not implementing distributed applications themselves. To quote the authors:
52 |
53 | “Eden team had real experience with writing distributed applications, we had not yet learned what support should be provided. For example, it was not clear to us whether or not each incoming call should be run in its own thread (possibly leading to excessive resource contention), whether or not all calls should run in the same thread (possibly leading to deadlock), whether or not there should be a thread pool of a bounded size (and if so, how to choose it), or whether or not there was some other, more elegant solution that we hadn’t yet thought of. So we left it to the application programmer to build whatever invocation thread management system seemed appropriate: EPL was partly a language, and partly a kit of components. The result of this approach was that there was no clear separation between the code of the application and the scaffolding necessary to implement remote calls.” (A. P. Black et al. 2007)
54 |
55 | I will close this section on Emerald with a quote from the authors.
56 |
57 | “We are all proud of Emerald, and feel that it is one of the most significant pieces of research we have ever undertaken. People who have never heard of Emerald are surprised that a language that is so old, and was implemented by so small a team, does so much that is ’modern’. If asked to describe Emerald briefly, we sometimes say that it’s like Java, except that it has always had generics, and that its objects are mobile.” (A. P. Black et al. 2007)
58 |
59 |
60 |
61 |
Birrell, Andrew, Greg Nelson, Susan Owicki, and Edward Wobber. 1993. “Network Objects.” In Proceedings of the Fourteenth Acm Symposium on Operating Systems Principles, 217–30. SOSP ’93. New York, NY, USA: ACM. doi:10.1145/168619.168637.
62 |
63 |
64 |
Black, A., N. Hutchinson, E. Jul, H. Levy, and L. Carter. 1987. “Distribution and Abstract Types in Emerald.” IEEE Transactions on Software Engineering SE-13 (1): 65–76. doi:10.1109/TSE.1987.232836.
65 |
66 |
67 |
Black, Andrew P, Norman C Hutchinson, Eric Jul, and Henry M Levy. 2007. “The Development of the Emerald Programming Language.” In Proceedings of the Third Acm Sigplan Conference on History of Programming Languages, 11–11. ACM.
68 |
69 |
70 |
Black, Andrew, Norman Hutchinson, Eric Jul, and Henry Levy. 1986. “Object Structure in the Emerald System.” In Conference Proceedings on Object-Oriented Programming Systems, Languages and Applications, 78–86. OOPLSA ’86. New York, NY, USA: ACM. doi:10.1145/28697.28706.
71 |
72 |
73 |
Hoare, Charles Antony Richard. 1974. Monitors: An Operating System Structuring Concept. Springer.
74 |
75 |
76 |
Holt, Richard C. 1982. “A Short Introduction to Concurrent Euclid.” ACM Sigplan Notices 17 (5). ACM: 60–79.
77 |
78 |
79 |
Kendall, Samuel C, Jim Waldo, Ann Wollrath, and Geoff Wyant. 1994. “A Note on Distributed Computing.” Sun Microsystems, Inc.
80 |
81 |
82 |
Liskov, Barbara. 1988. “Distributed Programming in Argus.” Communications of the ACM 31 (3). ACM: 300–312.
83 |
84 |
85 |
Raj, Rajendra K., Ewan Tempero, Henry M. Levy, Andrew P. Black, Norman C. Hutchinson, and Eric Jul. 1991. “Emerald: A General-Purpose Programming Language.” Software: Practice and Experience 21 (1). John Wiley & Sons, Ltd.: 91–118. doi:10.1002/spe.4380210107.
86 |
87 |
88 |
Shapiro, Marc, Yvon Gourhant, Sabine Habert, Laurence Mosseri, Michel Ruffin, and Celine Valot. 1989. “SOS: An Object-Oriented Operating System―-Assessment and Perspectives.” Computing Systems 2 (4): 287–338.
89 |
90 |
Relevant Reading
2 |
3 | Guesstimate: A Programming Model for Collaborative Distributed Systems, Rajan, Rajamani, Yaduvanshi, PLDI 2010 Rajan, Rajamani, and Yaduvanshi (2010).
5 |
6 | Guesstimate: A Programming Model for Collaborative Distributed Systems
7 | As we’ve seen previously, distributed applications have to be shoehorned into either the CP or AP model as outlined by the CAP thereom. This results from the fact that applications that must remain available under partition, and as the CAP theorem states that applications cannot be simultaneously AP and CP, the application has to choose whether to remain consistent or sacrifice consistency for availability when partitions inevitably occur.
8 | Guesstimate is a programming model for collaborative applications that aims to reduce latency by replicating objects on participating nodes in a network. Guesstimate provides the user with an object-oriented programming model: objects can be replicated and shared by different users on the network and method invocations that are side-effecting are eventually replicated to all users that contain an instance, or replica, of the object. Replication is provided transparently by the runtime, so the user doesn’t need their own backing store, differing from approoaches such as the previously discussed, IPA from Holt et al.
9 | Each object in Guesstimate stores two states: a “guestimated” state, which is the result of local side-effecting method invocations and a “committed” state, which is the result of atomically committed state taken from the updates each node has made to their own side-effecting “guestimated” state. These method invocations that cause side-effects must take a particular form: they must validate the change against the current state using a guard and return either true or false, depending on if the state mutation has taken effect. To verify methods take this form, the Boogie verification language is used to analyze the implementation. These method invocations must take a particular form: a method that causes modifications to the local “guestimated” state and a delegate (read: anonymous function) to be invoked when the state is finally committed on all of the nodes.
10 | As nodes modify their local “guestimated” state, periodically a desginated “master” node begins synchronization rounds. These synchronization rounds begin at the master, walk each of the nodes in turn preventing updates from occuring while the synchronization round runs, and aggregates the pending updates from each nodes “guestimated“ state. These updates are stored in a tentative log that’s aggregated at the master. Once the master node aggregates all of the updates from each node, the updates, each with a pair consisting of the node identifier and operation identifier, using lexicographical ordering on the set of pairs. This determines the commit order for each update, and once established, this commit order and the associated updates are sent to each of the nodes. Once these updates are accepted by all nodes, they are then applied to the “committed” state, the delegates invoked, and the lock on updates released, so local updates are allowed to resume.
11 | The ideal way of programming with Guesstimate is to have operations modify “guestimated” state first and update an associated UI afterwards. Then, the delegate is invoked and should be used to update the UI with a notification on whether or not the update has been successfully applied or was refused during committment and must be tried again.
12 | One problem that can occur is issues around update interleaving at committment time. Consider the case of a ride sharing application from the paper: a guard for a method invocation that states that a user should get a ride from driver X may yieldd the user getting a ride from driver Y under a particular update reordering: therefore, guards should be written in a way where any acceptable outcome is allowed.
13 | To address the problems of ordering, Guesstimate provides atomic operations: if two updates have to happen together, or there’s a causal relationship between updates (think: references, pointers, secondary indexes), these operations can be grouped together in the programming model to ensure that they commit together. To address operations that must commit before proceeding, Guesstimate provides a primitive for blocking an operation until committment.
14 | Users of Guesstimate have to also deal with the reality that updates will be executed multiple times, although against different components of the objects state: “guesstimated” and ”committed”.
15 | Guesstimates’ runtime takes care of most of the synchronization concerns: nodes are allowed to leave and join the system, where failed nodes are evicted after a certain amount of time and forced to rejoin the cluster and repopulate their state. Nodes form a full mesh, where each node can talk to each other node, and there’s currently no mechanism outlined for handling the failure of a “master” node that begins and coordinates the synchronization rounds.
16 | We can see the genesis of ideas that made it into systems such as the Global Sequence Protocol and CAPtain, which are covered in other articles.
17 |
18 |
19 |
Rajan, Kaushik, Sriram Rajamani, and Shashank Yaduvanshi. 2010. “Guesstimate: A Programming Model for Collaborative Distributed Systems.” In ACM Sigplan Notices, 45:210–20. 6. ACM.
20 |
21 |
Relevant Reading
2 |
3 | Implementing Location Independent Invocation, Black, Andrew P and Artsy, Yeshayahu, IEEE Transactions on Parallel and Distributed Systems, 1990 (Andrew P Black and Artsy 1990).
Customizable and extensible deployment for mobile/cloud applications, Zhang, Irene and Szekeres, Adriana and Van Aken, Dana and Ackerman, Isaac and Gribble, Steven D and Krishnamurthy, Arvind and Levy, Henry M, 2014 (Zhang et al. 2014).
6 |
7 | The general idea behind the Remote Procedure Call (RPC) paradigm is that it supports the transfer of control between address spaces. This paradigm allows programmers to write distributed applications without having to have knowledge of data representations or specific network protocols. Even though we know that there is quite a bit semantically different between remote and local calls (Kendall et al. 1994; Andrew P Black and Artsy 1990), the authors posit that the most fundamental difference is that of binding, or, how to figure out which address space to direct the call to.
8 | Traditionally, this has been done one of two ways: default or automatic binding where the RPC system makes the choice for the programmer; or clerks, an application specific module used for determining where the place the call. Default binding is fairly straightforward when there is only one server (or a group of semantically equivalent servers) to service the request. Clerks are fairly expensive, as one must be written for each type of request that needs to be serviced. If the service the RPC call is being made to is pure, for instance providing as fast Fourier transform as the authors put it, it is easy to choose automatic binding to select a server based on latency or availability. However, it is more challenging if services host application data. In their example, they consider an employee directory at Digital where application data is partitioned by company, and further by other groupings. If this mapping changes infrequently, a static mapping can be distributed to all of the clients; but, what happens if objects are mobile and this changes more frequently?
9 | One of the fantastic things about this paper is how forward thinking the design is for an actual industrial problem at Digital Equipment Corporation. I consider this one of the early versions of what we now call an “industry” research report, even though the system never was productized and the work was mainly performed by researchers in a lab. The application deals with expense vouchers for employees: each form needs to be filled in by an employee, approved by various managers, filed, and eventually results in a payout of actual cash. Each of the managers that are involved in approving the form may be located in different buildings in different continents. The application design assumes Digital’s global network of 36,000 machines and assumes that centralizing the records for each form in a centralized database is infeasible. Instead, the design is based on mobile objects for both data and code; forms should be able to move around the network as required by the application.
10 | The Hermes system is broken into three components: a naming service, a persistent store known as a collection of storesites, and routing layer that sits above the RPC system. Each object in the system is given a globally unique identifier, a source storesite and a temporal address descriptor or tad. The temporal address descriptor is a pair composed of a Hermes node identifier and a monotonically advancing timestamp: this pair represents where an object is located at a given time. This information is also persisted in the objects storesite. As objects move around the network, the tad is updated at the source node and 2PC used to coordinate a change with the record at the objects’s storesite.
11 | When remote procedure calls are issued, the callee attempts to issue the call locally if the objects is local. If not, and a forwarding pointer, or tad exists, the message is routed to that node. Forwarding pointers are followed a number of times until a maximum hop count is reached; at this point the call is returned to the callee who begins the process again with the last known forwarding pointer. Along the path of forwarding, the tad is updated as each hop occurs, reducing the number of hops needed for the next request through that node. This is possible because of the monotonicity of the temporal addresses.
12 | If a node has no local knowledge of where that object is, either because it is not running locally or because there exists no temporal address, a request is made to the naming service to request the storesite for the object, and the address of the current location retrieved from the storesite.
13 | However, in this model failures may occur. If the RPC arrives at the destination of the object and the call invoked and completed, but the response packets dropped, what happens? In this case, an invocation sequencer is required to ensure that the operation only performed if it has not previously completed. The authors suggest developers write operations that are idempotent, to ensure they can be replayed without issue or additional overhead.
14 | Impact and Implementations
15 | Both the Eden (Andrew P. Black 1985) and Emerald (Andrew P Black et al. 2007) programming languages both had notions of distributed objects. Eden used hints to identify where to route messages for objects, but timed them out quickly. Once timed out, a durable storage location called a checksite would be checked, and if that yielded no results, broadcast messages would be used. Emerald, a predecessor to Hermes, used forwarding addresses, but used a broadcast mechanism to find objects when forwarding addresses were not available. In the event the broadcast yielded no results, an exhaustive search of every node in the cluster was performed. All of these decisions were fine for a language and operating system designed mainly for research.
16 | Emerald was more advanced in several ways. Emerald’s type system allowed for the introduction of new types of objects, whereas the Hermes system assumed at system start all possible object types were known to the system. Emerald could also migrate processes during invocation, something that the Hermes system could not.
17 | While the system could tolerate some notion of failures while following forwarding addresses, by resorting to usage of the information located at the storesite, the system had no way to prevent issues with partitions: where an invocation may fail because the object is inaccessible. However, given the relative independence of objects in the system, this would only affect objects (or users) located on the partitioned machine.
18 | The design of Hermes was completed in a year and a half, written in Modula-2+, and was demonstrated functional in the laboratory with a LAN composed of a small number of nodes. According to one of the authors of the paper, the system never was turned into a product, mainly because Digital did not have a team at the time responsible for turning advanced research projects into actual distributed systems products.
19 | The Sapphire (Zhang et al. 2014) system presented at OSDI ’14 bears a similar resemblance to the Hermes system and its Emerald roots. While Sapphire focuses on the separation of application logic from deployment logic through the use of interfaces and interface inheritance in object-oriented programming languages, Sapphire uses many of the techniques presented in both Emerald and Hermes: transparent relocation based on annotations or for load balancing; location independent method invocation through the use of forwarding pointers, and fallback to a persistent data store to find the canonical location of a particular object.
20 | Today, idempotence (Helland 2012) has been a topic of study in distributed systems, as it assists in designing deterministic computations that must happen on unreliable, asynchronous networks; a place where it is impossible to reliably detect failures (Fischer, Lynch, and Paterson 1985). Shapiro et al. (Shapiro et al. 2011) propose the use of data structures that are associative, commutative, and idempotent as the basis for shared state in distributed databases. Meiklejohn and Van Roy (Meiklejohn and Van Roy 2015) propose similar for large-scale distributed computations; whereas Conway et al. (Conway et al. 2012) propose similar for protocol development. Lee et al. propose a system called RIFL for ensuring exactly-once semantics for remote procedure calls by uniquely identifying each call and fault-tolerant storage of the results (Lee et al. 2015).
21 |
22 |
23 |
Black, Andrew P, and Yeshayahu Artsy. 1990. “Implementing Location Independent Invocation.” Parallel and Distributed Systems, IEEE Transactions on 1 (1). IEEE: 107–19.
24 |
25 |
26 |
Black, Andrew P, Norman C Hutchinson, Eric Jul, and Henry M Levy. 2007. “The Development of the Emerald Programming Language.” In Proceedings of the Third Acm Sigplan Conference on History of Programming Languages, 11–11. ACM.
27 |
28 |
29 |
Black, Andrew P. 1985. “Supporting Distributed Applications: Experience with Eden.” In Proceedings of the Tenth Acm Symposium on Operating Systems Principles, 181–93. SOSP ’85. New York, NY, USA: ACM. doi:10.1145/323647.323646.
30 |
31 |
32 |
Conway, Neil, William R Marczak, Peter Alvaro, Joseph M Hellerstein, and David Maier. 2012. “Logic and Lattices for Distributed Programming.” In Proceedings of the Third Acm Symposium on Cloud Computing, 1. ACM.
33 |
34 |
35 |
Fischer, Michael J, Nancy A Lynch, and Michael S Paterson. 1985. “Impossibility of Distributed Consensus with One Faulty Process.” Journal of the ACM (JACM) 32 (2). ACM: 374–82.
36 |
37 |
38 |
Helland, Pat. 2012. “Idempotence Is Not a Medical Condition.” Queue 10 (4). New York, NY, USA: ACM: 30:30–30:46. doi:10.1145/2181796.2187821.
39 |
40 |
41 |
Kendall, Samuel C, Jim Waldo, Ann Wollrath, and Geoff Wyant. 1994. “A Note on Distributed Computing.” Sun Microsystems, Inc.
42 |
43 |
44 |
Lee, Collin, Seo Jin Park, Ankita Kejriwal, Satoshi Matsushita, and John Ousterhout. 2015. “Implementing Linearizability at Large Scale and Low Latency.” In Proceedings of the 25th Symposium on Operating Systems Principles, 71–86. ACM.
45 |
46 |
47 |
Meiklejohn, Christopher, and Peter Van Roy. 2015. “Lasp: A Language for Distributed, Eventually Consistent Computations with Crdts.” In Proceedings of the First Workshop on Principles and Practice of Consistency for Distributed Data, 7. ACM.
48 |
49 |
50 |
Shapiro, Marc, Nuno Preguiça, Carlos Baquero, and Marek Zawirski. 2011. “A Comprehensive Study of Convergent and Commutative Replicated Data Types.” PhD thesis, Inria–Centre Paris-Rocquencourt.
51 |
52 |
53 |
Zhang, Irene, Adriana Szekeres, Dana Van Aken, Isaac Ackerman, Steven D Gribble, Arvind Krishnamurthy, and Henry M Levy. 2014. “Customizable and Extensible Deployment for Mobile/Cloud Applications.” In 11th Usenix Symposium on Operating Systems Design and Implementation (Osdi 14), 97–112.
54 |
55 |
Promises
2 | Relevant Reading
3 |
4 | Promises: linguistic support for efficient asynchronous procedure calls in distributed systems, Liskov and Shrira, PLDI 1988 Liskov and Shrira (1988).
Multilisp: A language for concurrent symbolic computation, Halstead, TOPLAS 1985 Halstead Jr (1985).
7 |
8 | Outside of early mentions from Friedman and Wise on a cons cell with placeholder values Friedman and Wise (1978) and Baker and Hewitt’s work on incremental garbage collection Baker and Hewitt (1977), futures originally appeared as one of the two principal constructs for parallel operations in MultiLisp. MultiLisp attempted to solve a main challenge of designing a language for parallel computation: how can parallel computation be introduced into a language in a way that fits with the existing programming paradigm. This problem is motivated by the fact that computer programmers will need to introduce concurrency into applications because automated analysis may not be able to identify all of the points for parallelism. Halstead decides there is quite a natural fit with a Lisp/Scheme: expression evaluation can be done in parallel. MultiLisp introduces two main concepts: pcall, to evaluate the expressions being passed to a function in parallel and introduce concurrency into evaluation of arguments to a function, and futures, to introduce concurrency between the computation of a value and the use of that value. Halstead also notes that futures closely resemble the “eventual values” in Hibbard’s Algol 68, however were typed distinctly from the values they produced and later represented. Halstead Jr (1985)
9 | In 1988, Liskov and Shrira introduce the concept of a promise: an efficient way to perform asynchronous remote procedure calls in a type-safe way Liskov and Shrira (1988). Simply put, a promise is a placeholder for a value that will be available in the future. When the initial call is made, a promise is created and the asynchronous call to compute the value of the promise runs in parallel with the rest of the program. When the call completes, the value can be “claimed“ by the caller.
10 | An excerpt motivation from Promises: linguistic support for efficient asynchronous procedure calls in distributed systems (Liskov and Shrira, PLDI 1988):
11 |
12 | “Remote procedure calls have come to be the preferred method of communication in a distributed system because programs that use procedures are easier to understand and reason about than those that explicitly send and receive messages. However, remote calls require the caller to wait for a reply before continuing, and therefore can lead to lower performance than explicit message exchange.”
13 |
14 | The general motivation behind the work by Liskov and Shrira can be thought as the following critiques of two models of distributed programming.
15 |
16 | The Remote Procedure Call (RPC) paradigm is preferable by programmers because it is a familiar programming model. However, because of the synchronous nature of RPC, this model does not scale in terms of performance.
The message passing paradigm is harder for programmers to reason about, but provides the benefit of decoupling of request and response, allowing for asynchronous programming and the subsequent performance benefits.
19 | Promises attempts to bridge this gap by combining the remote procedure call style of building applications, with the asynchronous execution model seen in systems that primarily use message passing.
20 | The first challenge in combining these two programming paradigms for distributed programming is that of order. Synchronous RPC imposes a total order across all of the calls in an application: one call will fully complete, from request to response, before moving to the next call, given a single thread of execution. If we move to an asynchronous model of RPC, we must have a way to block for a given value, or result, of an asynchronous RPC if required for further processing.
21 | Promises does this by imagining the concept of a call-stream. A call-stream is nothing more than a stream of placeholder values for each asynchronous RPC issued by a client. Once a RPC is issued, the promise is considered blocked asynchronous execution is performed, and once the value has been computed, the promise is considered ready and the value can be claimed by the caller. If an attempt to claim the value is issued before the value is computed, execution blocks until the value is available. The stream of placeholder values serves as an implicit ordering of the requests that are issued; in the Argus system that served as the implementation platform for this work, multiple streams were used and related operations sequenced together in the same stream.
22 | Impact and Implementations
23 | While promises originated as a technique for decoupling values from the computations that produced them, promises, as proposed by Liskov and Shrira mainly focused on reducing latency and improving performance of distributed computations. The majority of programming languages in use today by practitioners contain some notion of futures or promises. Below, we highlight a few examples.
24 | The Oz Henz, Smolka, and Würtz (1993) language, designed for the education of programmers in several different programming paradigms, provides a functional programming model with single assignment variables, streams, and promises. Given every variable in Oz is a dataflow, and therefore every single value in the system is a promise. Both Distributed Oz Haridi, Van Roy, and Smolka (1997) and Derflow (an implementation of Oz in the Erlang programming language) Bravo et al. (2014) provide distributed versions of the Oz programming model. The Akka library for Scala also provides Oz-style dataflow concurrency with Futures.
25 | More recently, promises have been repurposed by the JavaScript community to allow for asynchronous programs to be written in direct style instead of continuation-passing style. ECMAScript 6 contains a native Promise object, that can be used to perform asynchronous computation and register callback functions that will fire once the computation either succeeds or fails.
26 |
27 |
28 |
Baker, Henry C., Jr., and Carl Hewitt. 1977. “The Incremental Garbage Collection of Processes.” SIGPLAN Not. 12 (8). New York, NY, USA: ACM: 55–59. doi:10.1145/872734.806932.
29 |
30 |
31 |
Bravo, Manuel, Zhongmiao Li, Peter Van Roy, and Christopher Meiklejohn. 2014. “Derflow: Distributed Deterministic Dataflow Programming for Erlang.” In Proceedings of the Thirteenth ACM SIGPLAN Workshop on Erlang, 51–60. Erlang ’14. New York, NY, USA: ACM. doi:10.1145/2633448.2633451.
32 |
33 |
34 |
Friedman, D. P., and D. S. Wise. 1978. “Aspects of Applicative Programming for Parallel Processing.” IEEE Transactions on Computers C-27 (4): 289–96. doi:10.1109/TC.1978.1675100.
35 |
36 |
37 |
Halstead Jr, Robert H. 1985. “Multilisp: A Language for Concurrent Symbolic Computation.” ACM Transactions on Programming Languages and Systems (TOPLAS) 7 (4). ACM: 501–38.
38 |
39 |
40 |
Haridi, Seif, Peter Van Roy, and Gert Smolka. 1997. “An Overview of the Design of Distributed Oz.” In Proceedings of the Second International Symposium on Parallel Symbolic Computation, 176–87. ACM.
41 |
42 |
43 |
Henz, Martin, Gert Smolka, and Jörg Würtz. 1993. “Oz-a Programming Language for Multi-Agent Systems.” In IJCAI, 404–9.
44 |
45 |
46 |
Liskov, Barbara, and Liuba Shrira. 1988. Promises: Linguistic Support for Efficient Asynchronous Procedure Calls in Distributed Systems. Vol. 23. 7. ACM.
47 |
48 |
Relevant Reading
2 |
5 |
6 |
7 | “A significant conclusion was that parallelism and data sharing are inherently difficult to combine effectively.”
8 |
9 | The Programming Language in the Sky (PLITS) project was an effort by the University of Rochester Computer Science department started in 1974 to take a serious look at how to make programming languages more declarative, and to see what benefits could be gained from using state-of-the-art compiler technology. This paper, specifically focuses on the problem of addressing the need for programming language constructs for distributed computing, that normally do not arise in conventional programming.
10 | At a high level, PASCAL-PLITS views distributed computing as a group of computers communicating over low bandwidth, unreliable communication paths, applications would consist of communication between modules via an asynchronous message protocol instead of through the use of subroutines to avoid having to wait for a response to a request. Modules only communicate through message passing, and each message is composed of a set of name-value pairs which are called slots: names are uninterpreted string and values are an element of some primitive type (for instance, integers.)
11 | To avoid discussion of many of the PASCAL specific details outlined in the formal specification section of the paper, the intuition on how message passing between two modules operates is rather straightforward:
12 |
13 | Messages can be sent between two modules where slots are compatible: for instance, both a server and client must define the types of messages they can receive and send, and they must specify the slots of the message they are going to read. It’s perfectly fine in this system to define a message that underspecifies the message slots – readers can only access what they know about through their local specification.
Messages are sent to recipients by specifying the recipient’s module identifier and can be selectively received at the recipient by sender’s module identifier.
Messages in PASCAL-PLITS arrive in FIFO order between modules and waiting on a message that never arrives after a particular time interval will throw an application-level exception.
Messages support a grouping behavior that allow for fixed size repetitions of values by type, nested one level deep, as values in slots.
18 | An interesting idea that the paper presents is the idea of forwarding the message to another module for handling processing. For instance, a server might need to accept a message and pass that message to another client to handle specific processing of that request. In that case, the path from original sender, to server, to client handling the request must be retraced when the request is complete and the response is being returned to the original caller. PASCAL-PLITS offers an alternative method: a unique transaction identifier is transmitted with the original message, so the client handling the request can send it directly back to the original sender, by passing the coordinating server module. In PASCAL-PLITS, this is referred to as a transaction key, and receive messages can specifically request they want to selectively receive messages about a key, instead of from a specific recipient; because of this proxy-like behavior, modules need to be able to read only particular slots of the message, and ignore, but forward, the remainder.
19 | The authors argue for message passing over subroutines:
20 |
21 | “The message paradigm has several advantages over subroutine calls. If the modules were in different languages, the subroutine call mechanisms would have to be made compatible. Any sophisticated lockout procedure would require the internal coding of queues equivalent to what the message switcher provides. In the subroutine discipline, a module which tries to execute a locked subroutine is unable to proceed with other computation. The total picture on the relative value of messages and calls is much more complex...”
22 |
23 | The authors also argue for the encapsulation provided by message passing as the only interface to data:
24 |
25 | “There are other interesting features that arise when messages are combined with the idea of modules. The most obvious feature of PLITS programming is the high degree of locality and protection it provides. Each PLITS module is totally self-contained and communicates solely through messages. This means that no local variables can even be examined from the outside, no procedures invoked, etc. A module can be asked to return or update a value, execute a function, etc. It now becomes quite natural to screen requests for validity (much more than type checking), to guard against conflicting demands on a data structure, etc. This does not solve all the problems attacked by structured programming strictures, but does make it clear what has to be done and where.“
26 |
27 | The authors use this encapsulation to solve the exclusion problem: if module B is storing data that A wants to compare and swap, B simply ignores messages that do not contain the transaction key generated by A until the swap is complete (or, does not handle messages related to the keys that are beings swapped until the operation is complete.) However, this ultimately brings additional complexities, as now the module itself must deal with the queue management of messages that are waiting to be processed while resources are locked.
28 | Perhaps the most interesting part of the PLITS system is the distribution system for running distributed jobs. Distribution is tackled through a layered approach:
29 |
30 | Networks are made up of a set of machines.
Machines are made up of multiple sites, each with a kernel that keeps track of scheduling and which modules are waiting to receive messages: within a site, messages must have the same format and same primitive data representation.
Kernels are responsible for distribution of messages within a site, forwarding messages between sites and resource allocation.
Co-located with a kernel is a Host Control Program (HCP) that is responsible for forwarding within sites on the same machine, and forwarding messages between machines.
Finally, the HCP is divided into two components: a distributed job manager and communication manager: one for job-lifecycle related operations and one for managing communication between machines.
36 | Messages to the recipients that are overloaded are buffered by the sender: this procedure is done on the sender side through process suspension. If nodes are located together at the same site, the kernel is responsible for this process, if not, the distributed communication manager is responsible for control flow management.
37 | Identification of where a module are located is done through the module name, an incarnation number, site number and local module number: this means that the identifier of a process can locate where its running on the network alone, without the need for a global registry. However, given this is done on a module instance basis, modules only ever live on the same machine. To get around this, the authors suggest using equivalent modules across machines and having the caller make the decision where to send the message. As the authors state it quite succinctly “contrary to current fantasies about distributed computing”.
38 | Impact and Implementations
39 | So, what’s the contribution of this paper?
40 | Well, as the authors describe it, it’s the “module-message” paradigm: something that you’ve probably seen before and the reason this paper has sounded so familiar: Erlang, or more specifically Distributed Erlang.
41 | PLITS, only briefly mentioned in Joe Armstrong’s thesis (Armstrong 2010), bears a striking resemblance to Erlang and brings some of the features that are used on a daily basis by Erlang developers. We quickly summarize those features now.
42 |
43 | Selective receive through the use of a transaction identifier has a strong resemblance to Erlang’s ability to generate a globally unique reference and use it as criteria to select from the process mailbox. This supports the forwarding behavior of processes that serve mainly for the routing of a request.
Repetition in slot values, where each value is tagged in the initial position by type bear a strong resemblance of how tagged tuples are used in records to specify a typed value purely as tuples.
The idea of encapsulation and locking or queueing messages for exclusion is similar to how processes control access to resources using the “generic abstractions” provided by Erlang: specifically, the “generic server” or “gen_server”. The generic server and generic finite state machine are both to control access to some state and has the ability to either select specific messages for processing related to a message identifier, or interleave requests if possible based on selective receive and asynchronous messaging.
The design of both the PLITS kernel, as a process scheduler responsible for waking and suspending processes waiting on messages, as well as the Host Control Program work very similar to the implementation of Distributed Erlang: processes running on remote machines are uniquely identified by a process identifier that encodes the machine name and messages are transparently delivered to them under what aims to achieve reliable transmission, but sometimes falls short.
48 | We should be clear, however, that PLITS is not an actor-based language such as Erlang. While PLITS shares many of the same design patterns and abstractions, the language is fundamentally a module-message system: modules are identified by name, which includes a location and incarnation number and are the target of messages, not processes.
49 |
50 |
51 |
Armstrong, Joe. 2010. “Erlang.” Communications of the ACM 53 (9). ACM: 68–75.
52 |
53 |
54 |
Feldman, Jerome A. 1979. “High Level Programming for Distributed Computing.” Communications of the ACM 22 (6). ACM: 353–68.
55 |
56 |
Relevant Reading
2 |
3 | Promises: linguistic support for efficient asynchronous procedure calls in distributed systems, Liskov and Shrira, PLDI 1988 (Liskov and Shrira 1988).
Multilisp: A language for concurrent symbolic computation, Halstead, TOPLAS 1985 (Halstead Jr 1985).
6 |
7 | Outside of early mentions from Friedman and Wise on a cons cell with placeholder values (Friedman and Wise 1978) and Baker and Hewitt’s work on incremental garbage collection for speculative execution in parallel processes (Baker and Hewitt 1977), futures originally appeared as one of the two principal constructs for parallel operations in MultiLisp. MultiLisp attempted to solve a main challenge of designing a language for parallel computation: how can parallel computation be introduced into a language in a way that fits with the existing programming paradigm. This problem is motivated by the fact that computer programmers will need to introduce concurrency into applications because automated analysis may not be able to identify all of the points for parallelism. Halstead decides there is quite a natural fit with a Lisp/Scheme: expression evaluation can be done in parallel. MultiLisp introduces two main concepts: pcall, to evaluate the expressions being passed to a function in parallel and introduce concurrency into evaluation of arguments to a function, and futures, to introduce concurrency between the computation of a value and the use of that value. Halstead also notes that futures closely resemble the “eventual values” in Hibbard’s Algol 68, however were typed distinctly from the values they produced and later represented. (Halstead Jr 1985)
8 | In 1988, Liskov and Shrira introduce the concept of a promise: an efficient way to perform asynchronous remote procedure calls in a type-safe way (Liskov and Shrira 1988). Simply put, a promise is a placeholder for a value that will be available in the future. When the initial call is made, a promise is created and the asynchronous call to compute the value of the promise runs in parallel with the rest of the program. When the call completes, the value can be “claimed” by the caller.
9 | An excerpt motivation from Promises: linguistic support for efficient asynchronous procedure calls in distributed systems (Liskov and Shrira, PLDI 1988):
10 |
11 | “Remote procedure calls have come to be the preferred method of communication in a distributed system because programs that use procedures are easier to understand and reason about than those that explicitly send and receive messages. However, remote calls require the caller to wait for a reply before continuing, and therefore can lead to lower performance than explicit message exchange.”
12 |
13 | The general motivation behind the work by Liskov and Shrira can be thought as the following critiques of two models of distributed programming.
14 |
15 | The Remote Procedure Call (RPC) paradigm is preferable by programmers because it is a familiar programming model. However, because of the synchronous nature of RPC, this model does not scale in terms of performance.
The message passing paradigm is harder for programmers to reason about, but provides the benefit of decoupling of request and response, allowing for asynchronous programming and the subsequent performance benefits.
18 | Promises attempts to bridge this gap by combining the remote procedure call style of building applications, with the asynchronous execution model seen in systems that primarily use message passing.
19 | The first challenge in combining these two programming paradigms for distributed programming is that of order. Synchronous RPC imposes a total order across all of the calls in an application: one call will fully complete, from request to response, before moving to the next call, given a single thread of execution. If we move to an asynchronous model of RPC, we must have a way to block for a given value, or result, of an asynchronous RPC if required for further processing.
20 | Promises does this by imagining the concept of a call-stream. A call-stream is nothing more than a stream of placeholder values for each asynchronous RPC issued by a client. Once a RPC is issued, the promise is considered blocked asynchronous execution is performed, and once the value has been computed, the promise is considered ready and the value can be claimed by the caller. If an attempt to claim the value is issued before the value is computed, execution blocks until the value is available. The stream of placeholder values serves as an implicit ordering of the requests that are issued; in the Argus (Liskov 1988) system that served as the implementation platform for this work, multiple streams were used and related operations sequenced together in the same stream.
21 | Impact and Implementations
22 | While promises originated as a technique for decoupling values from the computations that produced them, promises, as proposed by Liskov and Shrira mainly focused on reducing latency and improving performance of distributed computations. The majority of programming languages in use today by practitioners contain some notion of futures or promises. Below, we highlight a few examples.
23 | The Oz (Henz, Smolka, and Würtz 1993) language, designed for the education of programmers in several different programming paradigms, provides a functional programming model with single assignment variables, streams, and promises. Every variable in Oz is a dataflow, and therefore every single value in the system is a promise. Both Distributed Oz (Haridi, Van Roy, and Smolka 1997) and Derflow (an implementation of Oz in the Erlang programming language) (Bravo et al. 2014) provide distributed versions of the Oz programming model. The Akka library for Scala also provides Oz-style dataflow concurrency with Futures.
24 | More recently, promises have been repurposed by the JavaScript community to allow for asynchronous programs to be written in direct style instead of continuation-passing style. ECMAScript 6 contains a native Promise object, that can be used to perform asynchronous computation and register callback functions that will fire once the computation either succeeds or fails (Wikipedia 2016).
25 |
26 |
27 |
Baker, Henry C., Jr., and Carl Hewitt. 1977. “The Incremental Garbage Collection of Processes.” SIGPLAN Not. 12 (8). New York, NY, USA: ACM: 55–59. doi:10.1145/872734.806932.
28 |
29 |
30 |
Bravo, Manuel, Zhongmiao Li, Peter Van Roy, and Christopher Meiklejohn. 2014. “Derflow: Distributed Deterministic Dataflow Programming for Erlang.” In Proceedings of the Thirteenth Acm Sigplan Workshop on Erlang, 51–60. Erlang ’14. New York, NY, USA: ACM. doi:10.1145/2633448.2633451.
31 |
32 |
33 |
Friedman, D. P., and D. S. Wise. 1978. “Aspects of Applicative Programming for Parallel Processing.” IEEE Transactions on Computers C-27 (4): 289–96. doi:10.1109/TC.1978.1675100.
34 |
35 |
36 |
Halstead Jr, Robert H. 1985. “Multilisp: A Language for Concurrent Symbolic Computation.” ACM Transactions on Programming Languages and Systems (TOPLAS) 7 (4). ACM: 501–38.
37 |
38 |
39 |
Haridi, Seif, Peter Van Roy, and Gert Smolka. 1997. “An Overview of the Design of Distributed Oz.” In Proceedings of the Second International Symposium on Parallel Symbolic Computation, 176–87. ACM.
40 |
41 |
42 |
Henz, Martin, Gert Smolka, and Jörg Würtz. 1993. “Oz-a Programming Language for Multi-Agent Systems.” In IJCAI, 404–9.
43 |
44 |
45 |
Liskov, Barbara. 1988. “Distributed Programming in Argus.” Communications of the ACM 31 (3). ACM: 300–312.
46 |
47 |
48 |
Liskov, Barbara, and Liuba Shrira. 1988. Promises: Linguistic Support for Efficient Asynchronous Procedure Calls in Distributed Systems. Vol. 23. 7. ACM.
49 |
50 |
53 |