## Abstract

We give a rigorous framework for the interaction of physical computing devices with abstract computation. Device and program are mediated by the non-logical *representation relation*; we give the conditions under which representation and device theory give rise to commuting diagrams between logical and physical domains, and the conditions for computation to occur. We give the interface of this new framework with currently existing formal methods, showing in particular its close relationship to refinement theory, and the implications for questions of meaning and reference in theoretical computer science. The case of hybrid computing is considered in detail, addressing in particular the example of an Internet-mediated *social machine*, and the abstraction/representation framework used to provide a formal distinction between heterotic and hybrid computing. This forms the basis for future use of the framework in formal treatments of non-standard physical computers.

## 1. Introduction

*Computer Science is no more about computers than astronomy is about telescopes.*— E. Dijkstra (attrib.)

*That is incorrect.*— Lt. Cmdr. Data (Star Trek)

Heterotic hybrid computing involves the combination of two or more distinct computational substrates into a computing system whose power is greater than that of its individual components [1,2]. Such devices have come into prominence with the slowdown of Moore's law, and the subsequent search for different ways to increase computing power beyond standard silicon-based digital computers. The most well-developed such system is quantum computing [3]; while not usually presented as a hybrid device, a realistic quantum computer requires a large and integrated classical control computer, in particular for performing error correction [4]. Other examples include proposed chemical [5,6] and biological [7,8] non-standard computers, again requiring classical control. Also included are biological systems interfacing with digital software (for instance [9]). A different direction for heterotic systems is opened up by the enormous data-gathering and communication possibilities available online. A key recent insight there has been the recognition of ‘social machines’: computational ecosystems comprising both digital computers and multiple human users, all acting towards a computational goal [10,11]. Important social machines include Wikipedia [12] and crowdsourced science such as Galaxy Zoo [13]. As the space of these devices and interactions expands, heterotic forms of computing will continue to be a growth area in the very near future.

With the expansion of importance of such systems, questions of their computational foundations become newly prominent. Discussion within the unconventional computing community has often become stuck in discussion over (super-)Turing computable behaviour. For practical hybrid devices there are much more pressing concerns. The large toolkit of mathematical and abstract methods for computer science (semantics, logical calculi, programming, verification, etc.) has historically been developed in parallel with standard digital computers. Unfortunately, non-standard computing substrates and devices almost never admit a native description in terms of such standard abstract methods. This means that their computing power is both largely unknown and underutilized. They also lack the formal underpinnings that have proved so powerful in standard computer science. For hybrid devices, in particular, a lack of formalization of the individual components means a lack of a combined formalization when they are put together.

The lack of formalism for novel devices stems not (just) from a lack of understanding about their abilities, but from the way in which standard formal methods have been developed and used. The groundbreaking understanding of Lovelace, Turing and Shannon, which gave rise to modern computer science, was that the computers they were considering had a theoretical representation separate from their specific physical implementation. While inspired by certain physical devices (the Analytical Engine, the Bombe, the differential analyser, etc.), the associated formal systems could be manipulated without reference to any physically realized system. The interface between the abstract methods of computer science and the physical engineering of computers was, however, not itself formalized. Computers appear within the formalism, if at all, as mathematical or logical abstractions, not as the physical devices themselves.

This separation proved to be extremely powerful and was largely responsible for the rapid expansion of the modern science of computers. However as non-standard, and in particular hybrid, devices now come to prominence, having only an informal connection between the physical device and the abstract formal theory becomes significantly problematic. With a purely informal interface between the two, there is no straightforward way to reach standard abstract computer science from a novel computing substrate. Even if an ad hoc mapping can be considered for some particular non-standard devices (e.g. a certain neural net ‘implements an XOR’), this cannot be straightforwardly combined with another system with a different mapping to create a hybrid computational device. There is no way formally to combine such informal mappings, meaning that until now there has been no integrated framework that allows for the abstract treatment of non-standard and hybrid computers.

In this paper, we give formal foundations for the interaction of hybrid computing systems through the use of *Abstraction/Representation Theory*^{1} (AR theory). Recently introduced, AR theory allows us to make rigorous the interface between physical computing systems and their abstract, mathematical, representation. AR theory was developed in [14] to address the question of which physical dynamical systems are properly represented as computing systems. In this paper, we give a formal treatment of the theory in the context of the foundations of computer science, centring on the *representation relation* between data-processing physical systems and the abstract objects of standard computing theory. AR theory comes equipped with both an algebraic and a graphical language of commuting diagrams, and we show how this interfaces with similar, known, formal methods. The discussion is conducted with particular reference to three elements in the foundations of computer science—not as an exhaustive list, but to give the reader a broader understanding of the place AR theory occupies in computing theory. We show how AR theory supplements *refinement and reification*, allowing upwards/downwards simulation diagrams to be embedded in physical computing devices. We touch on the category-theoretic underpinnings of such diagrams, and consider also the relationship to the interactions between concrete and abstract semantics given by *Abstract Interpretation*. By grounding computer science in the physical world, questions of ontology and semantics for computational formal systems can also be addressed, and we will focus in particular on *lambda calculi* in this regard.

In the presence of AR theory, computer science becomes the natural science of the computing abilities of physical systems. While seemingly a very different foundational view of computing, it is in fact a supplement to, rather than a repudiation of, standard computability theory. It is a theory of comput*ing* to add to our current theories of comput*ation*. AR theory allows more physical devices, both individually and as heterotic systems, to be brought under the umbrella of abstract computation theory, while remaining agnostic about questions around their universality and/or Turing completeness. The addition of AR theory to the standard toolkit of computer science promises to greatly increase our power to reason about complex computational physical devices.

## 2. Basic elements of Abstraction/Representation Theory

AR theory was introduced in [14], with that paper giving the full physical and philosophical background for the framework. The present paper is concerned with its formalization for use specifically in computer science. That previous work should be referenced for questions concerning the broad ontology and context of AR theory, which will not in general be reiterated here.

AR theory concerns objects in the domain of physical systems, abstract objects (including mathematical and logical entities) and the representation relation which mediates between the two. The distinction between the two spaces, abstract and physical, is fundamental in the theory, as is their connection *only* by the (directed) representation relation. An intuitive example is given in figure 1: a physical switch is represented by an abstract bit by some representation.

This separation of physical and abstract domains necessitates our first definitions:

### Definition 1

The **physical domain**, **P**, consists of all physical objects, **p**∈**P**.

### Definition 2

The **abstract domain**, *M*, consists of all abstract objects, *m*∈*M*.

A *computer* is an object in the domain of physical entities, usually with internal degrees of freedom and physical evolution occurring between initial and final output states. A *computation* is a set of objects and relations within the domain of abstract entities, as described in the logical formalisms of theoretical computer science. Bold font is used to indicate where an object or evolution is physical. Abstract objects are represented using an italic font.

The elementary *representation relation* is the directed map . When two objects are connected by we write them as . The abstract object *m*_{p} is then said to be the *abstract representation* of the physical object **p**, and together they form one of the basic composites of AR theory, the *representational triple* . The basic representational triple is shown in figure 2*a*.

The next map we introduce is that of abstract evolution. This takes abstract objects to abstract objects: . An individual example is shown in figure 2*b*, for the mapping *C*(*m*_{p}) taking . The corresponding physical evolution map is given by . For individual elements in figure 2*c* this is **H(p)** which takes .

In order to reach the next key concept in AR theory, we now apply the representation relation to the outcome state of the physical evolution to give its abstract representation *m*_{p′}. This forms another representational triple , figure 2*d*. We now have two abstract objects, *m*′_{p} and *m*_{p′}. For some error quantity *ε* and norm ||, if |*m*_{p′}−*m*′_{p}|≤*ε* then figure 2*d* *commutes*. Commuting diagrams are fundamental to the use of AR theory. Definitionally,

### Definition 3

A **commuting diagram** in AR theory comprises two representational triples and , and a pair of abstract and physical evolutions and , which satisfy the condition |*m*_{p′}−*m*′_{p}|≤*ε*.

If a set of abstract and physical objects form a commuting diagram under representation, then *m*_{p} is a *faithful abstract representation* of physical system **p** for the evolutions *C*(*m*_{p}) and **H(p)**.

The main implication of having a faithful abstract representation for a physical system is that the final state of a physical object undergoing evolution can be known either by tracking the physical evolution and then representing the output abstractly, or by theoretically evolving the representation of the system. In the first case, the ‘lower path’ of a commuting diagram is followed; in the latter, the ‘upper path’. Finding out which diagrams commute is the business of basic experimental science, and their initial exploitation that of fundamental engineering and technology.

In experimental science, a test for commutation of a diagram involves producing a controlled physical set-up (the experiment) about which one has both an abstract representation and an abstract prediction of how it will behave, *C*. The physical system **p** is evolved under the physical experimental dynamics **H**, and the outcome compared with the theoretical prediction. If they coincide within the error tolerance of the experiment and the desired outcome confidence, then the diagram commutes.

This is not, of course, the *purpose* of an experiment. Experiments are designed in order to test not a single scenario but a *theory*. We now bring in the final element of basic AR theory:

### Definition 4

A **theory**, , is a set of representation relations for physical objects, a domain of such objects for which it is purported to be valid, and a set of abstract predictive dynamics for the output of the representations, *m*_{p}, *C*(*m*_{p}).

If a theory supports commuting diagrams for all scenarios in which it has been both defined and tested, then it is a *valid* theory.^{2}

A physical system or device that is both well tested and well understood will in general have a large number of commuting diagrams supporting it. This is a necessary condition for a theory to be *good*, but not a sufficient one. Furthermore, if a theory is a *good* theory then we can be confident that it will give rise to commuting diagrams outside the domain of either objects or dynamics for which it has been tested.

## 3. What representation is not

Computing relies on having a good and valid well-developed theory of the physical computing device. Commuting diagrams such as figure 3 do not, however, describe the actual process of computing itself; rather, they are the prerequisites for us to use the physical system as a computer. A computer must, however, satisfy a set of these diagrams. It is therefore useful at this point to consider such commuting diagrams for a computer, and specify exactly what is being described in the AR description of such a device.

A commuting diagram in the context of computation connects the physical computing device, **p**, and its abstract representation, *m*_{p}. *m*_{p} can be a number of different abstract representations—we go into this in more detail below, for now we take *m*_{p} to be drawn from the set of binary strings (figure 3). The abstract evolution is then the (binary) program to be run on the computer, and the physical evolution is how the state of the computer changes during the program (change of voltages, etc.). The full commuting diagram describes the parallel evolution of the physical computer and the abstract algorithm. They are connected via the representation given by the theory of the computing device, .

What precisely does the relationship given by AR theory between computer and computation consist in? That is, how does AR theory describe and define the relation between physical and abstract computational objects and evolutions, and what attributes does it give to this relationship? It is instructive at this point to compare the AR diagram in figure 3 with an existing formal framework that can appear at first sight to be very similar. The representation of computation given by Abstract Interpretation [15] involves a concrete semantics modelling the operation of the computing system, an abstract semantics representing the algorithm or program to be run, and the functional *abstraction relation* mapping between the two. Is this not, then, precisely what we have described with AR theory?

The reason why the answer to this question is ‘no’ gets to the heart of AR theory and what it is modelling in the representation relation. In Abstract Interpretation, the concrete semantics is designed to represent the operation of the physical device, but is *not the physical device*. It is an abstract, mathematical representation of the physical system, not the physical system itself. As a consequence, the abstraction relation between concrete and abstract semantics is a mathematical function, taking mathematical objects as both its domain and range.

By contrast, reference to the physical system within AR theory is to *the physical system itself*, not a representation of it. **p** is a physical object, *m*_{p} its mathematical/logical representation. The concrete semantics of Abstract Interpretation is the objects in the space *M* not the space **P**. AR theory concerns how physical objects *qua* physical objects relate to their own abstract representation, rather than how different types of abstract representation interact with each other. As a consequence, **p** must stand for the physical system and not some pre-defined representation; a model, a program, or a semantics, however low level, is still a *representation* of the physical system, *m*_{p}, not the physical system, **p**, itself.

This is the core of AR theory, and it cannot be over-emphasized: *as a representation relation is not a mathematical relation*. Neither is it a logical relation. It is a relation whose domain is physical objects and whose range is abstract/mathematical objects: it is a

*representation*relation. This is an entirely new kind of relation, one that is key to understanding how physical computing devices operate by mediating between the level of physics and the level of data manipulation.

## 4. Computing in Abstraction/Representation Theory

The parallel evolution of physical device and abstract program in figure 3 is not yet the AR representation of using that device to perform a computation. In order to describe computing, we need one final element of AR theory, this time not a basic element but a composed one.

The representation relation defined here is directed, from physical to abstract objects. This is *modelling*: giving an abstract representation of a physical object. The question can now be posed: is it possible to give a reversed representation relation, an *instantiation* relation? This will not be a basic relation in the same way as the ordinary (modelling) representation relation is basic: abstract representation can be given for any physical object (this is language), but there are plenty of abstract objects that do not have a physical instantiation (‘unicorn’, ‘infinite-tape Turing machine’, , etc.). Only in very specific circumstances can an instantiation relation be given for a theory .

To find these circumstances, consider again the ‘upper’ and ‘lower’ paths of a commuting diagram, and respectively. Between them, these paths describe the process of finding some **p**_{0} such that, when it is subjected to the physical process , it becomes the physical system **p** whose abstract representation is *m*_{p}. In other words, if both paths are present and form a commuting diagram, the theory can be used to *engineer* system **p** from system **p**_{0} given a desired abstract specification *m*_{p}. Together, then, they stand as a shorthand for the instantiation relation , as in figure 4.

The instantiation relation is given only when is both good and valid. It then permits the construction of an *instantiational triple*, . The system **p** is then the *physical instantiation* of the abstract object *m*_{p} for theory .

A use of the instantiation relation can be seen as a counterfactual use of the representation relation: which physical system, when represented abstractly, would give the abstract representation that we are trying to instantiate? The method by which it is achieved will vary considerably given different scenarios: trial and error, abstract reasoning, numerical simulation, etc. What connects these methods is that they are not straightforward: it is generally a skilful and creative process to reverse a representation relation.

In computing, the use of an instantiation relation is the act of *encoding and initialization*. At its simplest, this is the encoding of abstract data in a physical device, from turning a dial to a specified input state, punching a set of holes on a card, to initializing a series of voltages across a semiconductor. In all cases, how data are represented by physical objects is determined both by the available physics of the system and by design choices.

Initialization is the first step in the AR cycle for computing. This has one immediate implication: that a device cannot be used as a computer until its theory, , is well understood, as good and valid. If it is not, then cannot be formed with any confidence, and only experiment or engineering or technology cycles can be constructed with the device, not a compute cycle.

The compute cycle starts from a set of abstract objects—the program and initial state that are to be computed. The existence of an abstract problem, embedded usually as an input and a program, is the reason why a physical computer is to be used (figure 5*a*).

Consider again the problem of figure 3: binary addition of two 2-bit numbers: 01+10=11. The abstract initial state, *m*_{p}={01,10}, is encoded, through , in the physical system **p** (figure 5*a*). The representation relation determines that detecting a high voltage corresponds to representing a ‘1’, and low voltage is ‘0’. The initial physical set-up therefore instantiates an initial abstract state. In this example, two parts of the hardware are designated by as ‘registers’, and the voltages in the components of those areas correspond to the representation of the initial state as ‘01’ and ‘10’ (the two numbers we wish to add).

At the abstract level, this initial state is now the input to a sequence of gate operations that takes ‘01, 10’ and performs addition (figure 5*b*). At the physical level, a physical evolution **H**(**p**) is applied to the state, producing the final physical state **p**^{′}. Here, this will be the hardware manipulation of voltages.

Finally, an application of takes the final physical state and represents it abstractly as some . After this decoding step, if the computer has the correct answer then . If we have confidence in the theory of the computer, then we are confident that , and that this would be the outcome of the abstract evolution.

While a computer can be described in the above terms, as a parallel evolution of abstract and physical systems, the most important use of a computing system is when the abstract outcome *m*′_{p} is unknown: when computers are used to solve problems. In our example, if the outcome of 01+10 were unknown, and the computing device being used to compute it, the final abstract state, , would not be evolved abstractly. Instead, confidence in the technological capabilities of the computer would enable the user to reach the final, abstract, output state using the physical evolution of the computing device alone.

This use of a physical computer is the *compute cycle* (figure 5*c*):
This gives us the following definitions within AR theory:

### Definition 5

A **computer** is a (usually highly engineered) device in the physical domain with a good and valid theory , together with representation and instantiation relations, , and the relevant commuting diagrams, which support compute cycles for its specific domain of inputs {*m*_{p}} and computational operations .

### Definition 6

**Physical computing** is the use of a computer (a physical system) to predict the outcome of a computation (an abstract evolution) through a compute cycle.

## 5. Refinement and representation

The definition of physical computing, definition 6, makes clear what was noted at the beginning of this paper: that AR theory is a *supplement* to existing theories of computation, rather than a replacement of them. With the main machinery of AR theory in place, we can now see exactly where this framework fits in terms of the standard formal methods of computer science.

Perhaps the most immediate similarity to existing methods is with those of *refinement theory* (e.g. [16]). Refinement (alternatively, *reification*), in its simplest form, takes an abstract algorithm and produces an equivalent concrete machine code that can be implemented on a computer. In general, it describes the relationship between different *levels of abstraction* in computing code. For a refinement to be correct, the more concrete representation must faithfully implement the abstract specification. The concrete and abstract semantics of Abstract Interpretation, discussed briefly above, can be viewed as related by a refinement.

Refinement between levels of abstraction is governed by logico-mathematical refinement relations. If these relations are correct, then the levels of abstraction form commuting diagrams. At first sight, these refinement relations seem very similar to the representation and instantiation relations, and we will see that there are connections between the two. However, as with Abstract Interpretation, the operations and outcomes of refinement take place entirely within the abstract domain: even once a refinement is complete down to a machine code, this is still an *abstract* specification of the higher-level algorithm.

AR theory sits underneath this bottom level of a refinement stack, connecting the abstract system specification with the physical computing device. Figure 6 shows the connected stack reaching into the physical domain via the compute cycle. Different machine codes and different physical computers are given in each example. As a consequence, the representation and instantiation relations in each example are different.

The connection between representation and refinement now becomes clear: a representation can be viewed as similar to a refinement between physical and abstract levels. and differ from *S*_{AB,BC} in being non-mathematical relations, and in not being bidirectional; however, there are important structural similarities between AR theory and refinement theory that can now be brought out.

The full use of a refinement for computation, shown in the upper levels of figure 6, is verified through the correctness of one of two separate cycles: those of *downwards/forwards* and *upwards/backwards* simulation (e.g. [17]). The use of forward versus backward refinement proofs is governed by the point in the computation that any indeterminism is resolved [18]. These cycles are correct if the corresponding diagrams commute. Replacing a downwards refinement arrow with and an upwards one with , we can give the AR diagrams spanning the abstract–physical divide which correspond to these two types of simulation (figure 7).

We see immediately that *downwards simulation is analogous to technology* (implementing an abstract specification into a physical device) and that *upwards simulation is analogous to scientific prediction* (determining an abstract prediction for a physical event). The alternative terms for these simulations, ‘history’ and ‘prophecy’, respectively, import directly into the AR framework and become extremely descriptive [19]. It is a fascinating implication of AR theory that the relationship between abstract and physical objects follows the analogy with levels of abstraction in refinement theory even *outside* the domain of computer science. One interesting point to note is that, for refinement, only one of upwards and downwards simulation is needed to prove the refinement. By contrast, AR theory requires both its analogous cycles for computing: physical computing is the combination of experimental device theories capable of prediction (equivalent to upwards simulation/prophecy), and of technological engineering of the device (equivalent to downwards simulation/history). In other words, a computer is a combination of scientific understanding and of technology use.

An interesting area for further investigation, seen also by the analogues with refinement, is the category–theory structure of the relationship between abstract and physical objects. There is an obvious categorical underpinning to AR theory in its commuting–diagrammatic form, and an interesting question arises as to whether representation, while itself not mathematical, can itself be represented in this way.

One categorical account of data refinement was given by Hoare [20], representing refinement as a lax natural transformation between two models of the same category. If the computational system is embedded in a category , and *M*,*N* are two models of this category, then if there is a lax natural transformation then *M* is a ‘representation’ of *N* which is ‘more defined’ (i.e. the functors of *M* are defined over a larger number of objects than the functors of *N*). Lax transformation is then the categorical representation of downwards/forwards simulation. More higher-level generalizations have also been studied, with the objective of finding a better categorical description of refinement. Such notions as lax *logical* transformation [21] point towards a way in which the similar concept of representation may be categorified, with a physical object and its own abstract representation given as generalized models of a single categorical structure.

## 6. Meaning and ontology

A common issue in the foundations of computing concerns the meaning of objects within the formal systems of computation. By connecting these formal methods to the physical domain, AR theory helps refine questions of meaning and reference within computer science.

One of the most fundamental formal models of computation is that of the *lambda calculus*. In itself, it is a formal system of symbol manipulation. It has interpretations and models as anything from the theory of functions, proofs, or programming languages, to definitions of effective computability and the internal mechanics of Cartesian closed categories. Depending on which model is being used, the semantic content to even the simple term *λx*.*x* varies enormously.

With an AR underpinning to the abstract domain, it is now possible to distinguish between a mathematical object *x* being the representation of a physical system and that same object belonging to a model of a *λ*-calculus. The set of abstract objects that are representations of physical objects and the set of abstract objects representing models of formal systems are not co-extensive. Some formal systems may have a concrete model but no physical referent; and some physical systems may have a mathematical abstraction that does not correspond to a model for any current formal system of computation.

Physical computing occurs in the special case where the abstract representation of the computing system, ‘bottom-up’, is *also* an object within a model of computation, ‘top-down’. In this case, the semantic content to a computational assertion is twofold, and the reference of the abstract object is similarly dual: the meaning of a term, say *λyx*.*xy*, is both a computational assertion and a description of the physical capabilities of the physical computing device (here, that the state of two data-instantiating systems can be swapped). While formal computation theory extends beyond objects that are representations of physical devices, it is based around, and must be consistent with, the core of objects and relations that are.

This dual ontology of mathematical objects, both physical and abstract, while unusual in the context of computer science, is the standard within the natural sciences. A comparison with the situation in theoretical physics is highly instructive here. Physics started bottom-up, with many physical systems which were then represented abstractly, and the formal mathematics of their representations developed. Sometimes it was noted that the mathematical representations of these physical systems also corresponded to elements of previously known mathematics.

By contrast, computer science has historically concentrated on formal systems, top-down, with only a very narrow range of physical systems corresponding to the abstract specifications. The challenge for non-standard and extended types of physical computing is to import some of the methodology of natural science into computer science. Just as theoretical physics is now starting to consider formal systems that may not correspond to physical universes (e.g. generalized probabilistic theories based on quantum mechanics [22]), so computer science needs to extend to deal with physical systems that do not immediately correspond with standard formal methods. Abstract representations of non-standard computing systems may be found to correspond through some embedding or refinement or change in representation with the abstract representation of standard computers. In some cases they may, however, require an abstract theory of computation that is different; or another instance of a more general theoretical framework. The framework of AR theory can help allow us to make that determination for novel computing devices.

This leads to what is not a definition, but a description of the new addition AR theory gives computer science, in the form of physical computing:
**Computing** is the natural science of the computational abilities of physical systems.

## 7. Heterotic and hybrid computing

We now apply AR theory and our understandings of refinement and ontologies to the central issue of this paper: the representation of heterotic and hybrid computing within AR theory.

In order to highlight the central feature of heterotic computing, we first make a distinction with non-heterotic hybrid computing. As noted in the Introduction, heterotic hybrid computing allows computations to be performed that are, in some sense to be determined, more powerful than, or otherwise significantly different from, the computations that can be run on the individual systems. By contrast, we will use the term ‘hybrid’, without modifier, where the computations on the joint system are a simple composition of the computations on the individual ones.

A simple example of a hybrid system would be the human–computer interaction when, in order to perform the calculation (10+10)/7, the human user mentally performs (10+10) and then uses a computer to calculate . The joint computation is hybrid, but is not outside the computing power of either individually. Contrast this with another human–computer interaction, a social machine. For a machine such as Galaxy Zoo [13,23], the computation performed is outside the abilities of either the humans or the computer—the machine is heterotic.

We will postulate here a type of hybrid-heterotic distinction that is native to AR theory. It may be *the* distinction, or it may be one type of a class of distinctions given by different types of composition; this and the relationship to heterotic systems as defined by the original authors [1,2] are areas for further investigation. The following can all be thought of as a consideration of *representational* hybrid versus heterotic systems.

A hybrid (unmodified) system in AR theory can be described using composition of computation in the abstract domain. Figure 8*a* shows this for two physical computing devices, **p** and **q** (these could be a human and a computer, respectively). They each encode and process information according to their own representation relations and theory. That is, the computational system forms two separate representational triples, and , where *τ*,*ν* are the theories of the two computing systems.

There are now two possibilities for hybrid data processing. The first, as shown in figure 8*a*, is parallel composition: the computation performed, *C*, itself a composition of *C*_{τ},*C*_{ν}, is performed on the composed input:
The second option is sequential composition. This is where the computations *C*_{τ},*C*_{ν} are performed on separate machines and then composed:

The case of heterotic computing differs in significant respects from straightforward hybrid computing. In the case of heterotic systems, the composition happens in the representation itself, before the abstract computation is reached. Figure 8*b* shows the production of a joint representation for the heterotic system composed of **p** and **q**. Importantly, there is only one representational triple formed for the joint system:
The single representational triple then means that there is a single set of operations allowed on the abstract input, given by the composed device theory *μ*. We will denote these operations *D*_{μ}. The system then proceeds as

The crucial difference in this heterotic case is that *D*_{μ} need not be a composition of *C*_{τ},*C*_{ν}. Because the representation is combined, the joint system can have access to computational operations not available to individual ones. This is the signature of heterotic computing: that composition occurs in the representation, rather than within the abstract domain.

It will, in general, be delicate work to pick out exactly where the composition is occurring, and so to argue whether a system is ‘truly’ heterotic or not. If the systems **p** and **q** are individually capable of computing, then they will participate in representational triples etc. individually, as well as jointly for the heterotic computation. Differentiating these two levels of representation will be important.

As an example, we will consider the Galaxy Zoo social machine. A human user interacts by classifying objects presented to them into one of several categories. The computer processes these inputs according to its programming and produces an output. The user can be described using , where *τ* is the computational theory of how a human computing device classifies visual objects as ‘round’, ‘square’, etc. The computer satisfies the triple , where *ν* is the theory of computers running Javascript. Individually, these computations make sense, and satisfy all the relevant commuting diagrams.

However, a full description of the system is not reducible to a description in terms of these two elements. The social machine itself is described by the triple . The theory of this machine, *μ*, is the theory of a computing device that takes as input astronomical pictures and outputs classifications of galaxies. Only in the context of theory *μ* are the individual actions of the human and the computer describable as part of this computation. Furthermore, this computation is not available to either individually (given our current understanding of the abilities of human beings and Javascript programs).

It is key to heterotic computing that the heterotic triple not be decomposable. For some composition of representational triples, then, we can give the following definition:

### Definition 7

If two physical systems **p** and **q** are elements of the representational triples and , and individually satisfy all the requirements for being a computer, and if the joint physical system **p**°**q** also participates in the representational triple and satisfies the requirements for being a computer, if
then the computing system is **heterotic**. If the equality holds, then the computing system is **hybrid**.

Defining composition between representational triples is an important area of further work.

This consideration of hybrid and heterotic brings us back to the centrality of representation in computing: the different abilities of heterotic systems over their hybrid cousins lie in their abilities to use composed representations of joint systems. We can also see another aspect of AR theory: that more than one representation can be given for the same physical system—even more than one *computational* representation, where the same physical system is taking part in more than one computation under different representational theories. By foregrounding representation, AR theory has given us the ability to distinguish these different computational scenarios; and so to reason rigorously about these novel computing devices and processes.

## 8. Conclusion

In this paper, we have seen how AR theory enables us to add physical systems—standard, unconventional and hybrid/heterotic—into the foundational language of computer science. By making rigorous the *representation relation* between physical and abstract objects, we are able to distinguish between computing in the physical domain and computation in the abstract—and to give a set of relations between the two. The algebraic and commuting–diagram forms of AR theory can be used to show the similarities and differences between different stages of computer design and use—from the initial experiments on a substrate, to its engineering, and finally its use as a computational device. This leads us to the AR definition of physical computing, whereby a computer is shown to be a physical device for predicting the outcome of an abstract evolution (a computation).

Representation is key to computing: a computer is a machine that must use the representation relation as part of its functioning, as unless it is present within a compute cycle no computing can be said to be occurring in a physical device.^{3} We have seen how the representation relation extends the refinement and abstraction relations of Abstract Interpretation and refinement theory. With the addition of AR theory, refinement diagrams can be given that commute ‘all the way down’ to the physical device (not simply the still-abstract machine code), and the connections between the physics of the substrate and the levels of abstraction they can support becomes clearer. We have seen furthermore the beginnings of a fascinating relationship between types of refinement simulation and different cycles in AR theory corresponding to science and the use of technology.

We have defined physical computing and described the development of the natural science of computing. We have seen how AR theory fits in with formal abstract systems of computation such as the lambda calculus and gives a physical ontology for computation theory. This now opens up the way to using AR theory in the formalization of computing using novel physical systems, such as quantum devices and the newly described set of Internet-enabled social machines. We have seen how these hybrid and heterotic systems can be defined and distinguished using AR theory. The power inherent in the AR separation of abstract and physical domains, and representation, comes out strongly as we are now able to distinguish between composition of representations (hybrid) and composition within representation (heterotic). By no longer conflating the representation of a device with the device itself, we are able to determine when a single device supports multiple computational representations, and the relationships between them.

There are many, many open questions and problems here; the development of AR theory and its application to computing is still in its first stages. It has already shown itself to be a powerful tool for reasoning about computational systems and processes, and further work will only add to its range of application. One immediate avenue is the extension of the analysis given above for social machines. A full construction for a specific machine, giving the precise connection between the different levels of representation present, and the formal definition of composition between representational triples, is outstanding work.

Another open, and very interesting, question is how AR theory in general relates in detail to the categorical substructures of theoretical computer science. In particular, the connection between refinement types and the AR commutation cycles for science and technology leads to the possibility that a categorification of AR theory would give an algebra for experimental theory confirmation in science—and, by the Curry–Howard correspondence, an associated logic. This would be a novel logical structure for theory choice—opening up in turn the possibility of importing it back into computer science as an addition to theory discovery and decision algorithms in artificial intelligence.

AR theory does not just add new rigorous tools for computing. The foundational gap between computational theory and physical device that has historically been present has now been bridged. Computers show themselves as machines that manipulate both physics and representation, and computing as a process that necessarily crosses the divide between physical and abstract. A missing piece of the puzzle in the foundations of computer science is now in place.

## Competing interests

I declare I have no competing interests.

## Funding

The author acknowledges funding from the NQIT EPSRC Quantum Technologies Hub and EPSRC project EP/L022303/1.

## Acknowledgements

D.C.H. thanks Viv Kendon and Susan Stepney for many and continuing discussions on AR theory. Thanks are also owing to two anonymous referees for very useful comments and suggestions.

## Footnotes

One contribution of 13 to a Theo Murphy meeting issue ‘Heterotic computing: exploiting hybrid computational devices’.

↵1 I am indebted to Aleks Kissinger for the name.

↵2 It may not, however, necessarily be a

*good*theory.↵3 One may be tempted to render this conclusion as ‘no computation without representation’.

- Accepted May 5, 2015.

- © 2015 The Author(s) Published by the Royal Society. All rights reserved.