Intentionally Fusing Direct Manipulations into SVG Programs

Abstract

Bidirectional live programming is a programming paradigm that enhances the developer experience by allowing direct modifications to the output, with changes instantly reflected back to the source program. Existing operation-based approaches allow modifying the constant values and expressions of programs, while they lack control over modifications originating from multiple updatable locations. To control the updatability of the source program, our approach introduces non-updatable expressions, which guide the fusion of updates to the intended locations. The correctness properties, GetPut and WeakPutGet, ensure output consistency. We have implemented a prototype, FuseSVG, and tested it with 14 nontrivial SVG-drawing benchmarks, demonstrating the effectiveness of our approach.

1. Introduction

Programming can be mentally taxing as it requires us to envision the execution of code during the editing process. Live programming [1,2,3], however, alleviates this strain by continuously re-executing the program during the editing phase, thereby establishing a seamless feedback loop between the programmer and the program. Bidirectional live programming [4] further enhances the developer’s experience by providing a distinctive feature, known as direct manipulations [5], which allows for direct modifications to the output. These changes are instantaneously reflected back to the source program, guaranteeing that the updated program can reproduce the manipulated output.

Sketch-n-Sketch [4,6,7,8] is a bidirectional live programming system that allows direct manipulation on the output, eventually causing the system to substitute old constants in the program with new ones.

The code in Figure 1 creates a rectangle with a width that is twice its height (expressed as mult 2 h in the code). If a programmer attempts to widen the rectangle horizontally while maintaining the height, they would be unsuccessful. This is because any change to the rectangle’s width in the SVG results in an update to the rectangle’s height in the program, due to the default update policy that modifies the right operand of the expression mult 2 h. Sketch-n-Sketch provides a freeze operation that uses ! to mark a constant, indicating that it cannot be modified. If the constant value of height 100 is marked with !, the rectangle becomes un-stretchable. Furthermore, if the programmer wants to change the expression from mult 2 h to mult 2 h + 50 by directly manipulating the output, thereby establishing a new correlation between width and height, this is unachievable as the system only permits updates to constants. To summarize, Sketch-n-Sketch allows for the modification of constants, yet it constrains the adjustment of expressions, such as the correlation between width and height.

FuseDM [9] fuses direct manipulations into source program by propagating them to constants whenever possible. In order to satisfy the correctness that the execution of the updated program yields the same manipulated output, sometimes direct manipulations will be embedded into a “proper position”.

Figure 2 depicts the same rectangle. If the programmer stretches the rectangle horizontally, the expression h*2, which represents the width of the rectangle, is updated to h*2+dh, where dh represents the displacement value, and the height h remains unchanged. Although FuseDM conveys the intention of updating the expression, it fails to express the intention of maintaining the relationship between width and height by adjusting the width. This is particularly important when drawing in SVG, as numerous relationships between SVG elements must be preserved.

In general, we propose an approach that sits between Sketch-n-Sketch and FuseDM, which is capable of preserving the relationships between SVG elements during manipulation, while also facilitating program updates. These updates encompass more than just changes to constant values; they also involve modifications to expressions. We developed FuseSVG with the following features: (1) We introduce a const keyword that can be used to designate an expression as non-changeable. This guides the fusion of manipulation, ensuring that marked expressions are not updated during propagation to prevent unintended changes. (2) The correctness is guaranteed by the GETPUT and WeakPUTGET properties. Since the relation is stored in the program, fusion followed by execution of updated program ensures that elements in SVG are consistent according to the relation. Moreover, the WeakPUTGET guarantees that updates are only reflected in related expressions.

The program shown on the left of Figure 3 conveys the intent that the value of the height variable h, and the expression that serves as the left operand of the binary expression h*2, are immutable. Consequently, any manipulation of the width will result in changes to the ratio value 2. If we further designate h*2 as const, as shown on the right of Figure 3, any manipulation of the width will lead to the transformation of the entire binary expression from const(h*2) to an updated expression const(h*2)+dh, where dh denotes the displacement value. This kind of transformation is not feasible in Sketch-n-Sketch.

FuseSVG builds upon our previous work, FuseDM [9], while FuseDM laid the foundation for bidirectional live programming with an operation-based approach for fusing updates into the source program, FuseSVG refines and enhances this framework by addressing key limitations identified during the development of FuseDM. Specifically, FuseSVG introduces improvements in the handling of update propagation, correctness in positioning updates, and user-intended relationships within the program. These extensions make FuseSVG a more robust and flexible approach, offering finer control over program updates while maintaining consistency.

In summary, our contributions are as follows:

• We introduce a manipulation-directed programming approach where manipulations represent changes on the SVG. These changes are fused into programs to construct a desired program. Furthermore, a const marker is utilized to guide the propagation and fusion of the manipulation in the intended location.

• We design both the forward evaluation and the backward fusion semantics to preserve manipulations. We have proven that the language semantics satisfy the GetPut and WeakPutGet properties to guarantee correctness, thereby ensuing the validity of our approach.

• We have implemented a prototype, FuseSVG, that supports manipulation-directed programming for SVG. It has been tested with 14 nontrivial benchmark examples to demonstrate the effectiveness of our approach.

2. A Core Language DF

We initially describe the syntax of the core language DF, which stands for the functional language F with Deltas. The DF language is designed as a minimal core language for expressing key concepts of our system, particularly the manipulation of program deltas and the structured application of updates. It formalizes how delta values are handled, enabling precise reasoning about how changes propagate through a program. The purpose of DF is to provide an abstract environment in which the logic of program updates can be clearly modeled and analyzed. Next, we present its formal semantics and provide inference rules that determine whether a given expression is modifiable, which are used during the manipulation propagation process.

2.1. Syntax of DF

In Figure 4, we outline the core functional language, DF, designed for crafting source programs. Basic expressions are composed of constants c, variables x, abstractions $\lambda p.e$, and applications $e_1{e}_2$. It also includes binary expressions, case expressions, and $\mathit{fix}e$, which is used to described recursive functions. Constructors include tuple constructions $(e_1,e_2)$ and list constructions $e_1::e_2$. The const keyword is introduced, which can be used to designate an expression as non-changeable (const e). Consequently, it includes a special expression $e^{dv}$ that retains the delta for the expression and a special value $c^{dv}$ that stores updates on the constant c, which are vital for demonstrating the correctness of our approach.

Constants encompass numbers n, Booleans b, strings s, and empty lists []. Values are made up of constants with deltas $c^{\mathit{dv}}$, tuples $(v_1,v_2)$, list constructions $v_1::v_2$, and function closures $(E,E_b,\lambda p.e)$. The environment E is responsible for mapping variables to their corresponding values v, while the environment $E_b$ maps variables to Boolean values $b_v$. $b_v$ represents a special Boolean value that includes $\mathit{true}$, $\mathit{false}$, a tuple of $b_v$, and a special value $(E,E_b,\lambda p.e)$, which is internally used for lambda expressions. Basic delta operations consist of $id$, replacement (repl$\mathit{c}$), composition (${\mathit{dv}}_1\circ {\mathit{dv}}_2$), plus delta (${+}_{\Delta }\mathit{atom}$) that augment a given expression’s value, delete that signifies an element’s removal from the list, insert that indicates an element’s addition to the list, and an extra function closure used exclusively during computation. The subscript _Δ is employed to differentiate the notations of the same operators in the arithmetic expression.

Example 1. 

A list of simple examples.

$\begin{array}{ccc}& \lambda x.(x,x)5^{{+}_{\Delta }\mathit{10}}\hfill & \hfill \mathrm{An}\mathrm{application}\mathrm{with}\mathrm{a}\mathrm{constant}\mathrm{value}\mathrm{marked}\mathrm{with}\mathrm{a}\mathrm{delta}\mathrm{value}.\\ & 5^{{+}_{\Delta }\mathit{10}}+5\hfill & \hfill \mathrm{A}\mathrm{binary}\mathrm{expression}\mathrm{whose}\mathrm{left}\mathrm{operand}\mathrm{is}\mathrm{a}\mathrm{delta}\mathrm{expression}.\\ & (\mathsf{const}5,5^{{+}_{\Delta }\mathit{10}})\hfill & \hfill \mathrm{The}\mathrm{first}\mathrm{element}\mathrm{of}\mathrm{the}\mathrm{tuple}\mathrm{is}\mathrm{not}\mathrm{modifiable}\mathrm{since}\mathrm{it}\mathrm{is}\mathrm{marked}\mathrm{by}\mathrm{const}.\\ & 5^{{+}_{\Delta }\mathit{10}}::\left[\right]\hfill & \hfill \mathrm{The}\mathrm{first}\mathrm{element}\mathrm{of}\mathrm{the}\mathrm{list}\mathrm{is}\mathrm{a}\mathrm{delta}\mathrm{expression}.\\ & 5^{\mathsf{delete}}::\left[\right]\hfill & \hfill \mathrm{The}\mathrm{first}\mathrm{element}\mathrm{of}\mathrm{the}\mathrm{list}\mathrm{is}\mathrm{deleted}\mathrm{through}\mathrm{manipulation}.\\ & 5^{\mathsf{insert}}::\left[\right]\hfill & \hfill \mathrm{The}\mathrm{first}\mathrm{element}\mathrm{of}\mathrm{the}\mathrm{list}\mathrm{is}\mathrm{newly}\mathrm{inserted}\mathrm{through}\mathrm{manipulation}.\end{array}$

Example 2. 

Suppose the programmer stretches the rectangle horizontally by 10; the program defined in Figure 3 will be updated to the following one.

$\begin{array}{c}\hfill \mathtt{main}=\mathtt{let}\mathtt{h}=\mathtt{const}100\mathtt{in}[\mathtt{rect}018270\left(\mathtt{const}\mathtt{h}\right)\ast 2^{{+}_{\Delta }\mathit{0}.\mathit{1}}\mathtt{h}].\end{array}$

Note that a program augmented with deltas acts as an intermediate representation, which is crucial to our approach and will be explained in detail later. Delta values can be seen as a form of lifting, where updates (represented as deltas) are encapsulated within a computational context and applied to the source program, similar to how applicative functors apply functions to wrapped values. Specifically, delta values are combined and propagated through program expressions, preserving the program’s original structure and intent, much like how applicative functors combine computations in a functional context. However, all deltas are ultimately eliminated in the final program. The resulting program derived from Example 2 is shown below:

$\begin{array}{c}\hfill \mathtt{main}=\mathtt{let}\mathtt{h}=\mathtt{const}100\mathtt{in}\left[\mathtt{rect}018270\right(\mathtt{const}\mathtt{h})\ast 2.1\mathtt{h}].\end{array}$

2.2. Semantics of DF

This section describes the semantics of DF as shown in Figure 5. While most of the evaluation rules are quite straightforward, we only explain those rules that involve the delta or const marker. Evaluating a const-marked expression is equivalent to directly evaluating the expression itself. When evaluating an expression with a delta denoted as $e^{dv}$, the expression e is first evaluated to a value v, resulting in a value $v^{dv}$. We impose a restriction on binary expressions $e_1+e_2$ and $e_1\ast e_2$ such that the delta $dv$ can only occur on one side. This does not reduce the expressiveness of our language, as the delta only occurs on one side during update propagation in our setting.

Example 3. 

The evaluation result of the program in Example 2 is as follows, where the width value is set to 200, augmented with a delta of ${+}_{\Delta }\mathit{10}$:

$\begin{array}{cc}& \left[\mathtt{rect}[0,182,70,{200}^{{+}_{\Delta }\mathit{10}},100]\right]\hfill \end{array}$

2.3. Updatability of Expression

When an expression is marked with const, the syntactic structure of the expression is not modifiable; however, the value of the expression may be subject to change if variables are present within the expression. To ascertain whether the computed value of an expression is updatable, we establish a set of inference rules. The const inference rules are outlined in Figure 6. If an expression is inferred as $\mathit{true}$, it signifies that the expression is not updatable. Conversely, if it is inferred as $\mathit{false}$, it implies that the expression is updatable.

As depicted in Figure 6, E signifies an environment for variables bound to their respective values, while $E_b$ represents a set containing variables bound to $b_v$. For a given expression e, we infer it as $b_v$ to ascertain the updatability of the expression. Nearly every expression has two rules: one marked with const, and one without. If a constant is marked with const, it is inferred as $\mathit{true}$. In the absence of the const keyword, it is inferred as $\mathit{false}$. If the expression is inferred as $\mathit{false}$, then the const-marked expression is also inferred as $\mathit{false}$. The updatability of a variable depends on whether it is marked with the const keyword and the binding of the variable x in $E_b$. However, if a variable x lacks the const keyword, it will invariably be inferred as $\mathit{false}$, indicating that the expression x is modifiable. For a lambda expression, we create a tuple that encapsulates E, $E_b$, and the lambda expression $\lambda p.e$, which is utilized in lambda application. For a binary expression marked with const, we initially infer the Boolean value of the left and right operands when marked with the const keyword. The computation of $b{v}_1\oplus b{v}_2$ is straightforward. Given that $b{v}_1$ and $b{v}_2$ can only be $\mathit{true}$ or $\mathit{false}$, the result is $\mathit{true}$ if both $b{v}_1$ and $b{v}_2$ is $\mathit{true}$, otherwise, it is $\mathit{false}$. If it lacks the const keyword, we directly return $\mathit{false}$. For a case expression, pattern $p_j$ is matched with $b_{v0}$ to construct $E_{m_b}$, which is subsequently used for evaluating $e_j$. When a tuple is marked with const, we infer $e_1$ and $e_2$ when marked with const. The const inference rules for lists are the same as for tuples. When an expression contains delta, it will always be inferred as $\mathit{false}$ if it is not marked by const. Otherwise, the delta should be $id$ and the result is the same as evaluating e.

Example 4. 

Given a lambda application, the environment E contains a variable y, and in the environment $E_b$, the variable y is bound to $\mathit{true}$, then the inferred result is $\mathit{true}$.

$\begin{array}{cc}& \{y\mapsto 1^{id}\},\{y\mapsto \mathit{true}\}⊢\left(\mathtt{const}\lambda x.x\right)\left(\mathtt{const}y\right)\to \mathit{true}\hfill \end{array}$

Example 5. 

Given a binary expression, the environment E contains variables x and y, and in the environment $E_b$, the variable x is bound to $\mathit{true}$, while the variable y is bound to $\mathit{false}$. Even if the binary expression is marked by const, it is inferred as $\mathit{false}$ since y is inferred as $\mathit{false}$.

$\begin{array}{cc}& \{x\mapsto 1^{id},y\mapsto 2^{id}\},\{x\mapsto \mathit{true},y\mapsto \mathit{false}\}⊢\mathtt{const}(x+y)\to \mathit{false}\hfill \end{array}$

3. A Direct Manipulation Language DM

We begin by introducing the syntax of the direct manipulation language DM, followed by a description of its semantics in detail.

3.1. Syntax of DM

The syntax of DM is outlined in Figure 7. The symbol $dv$ is used to denote a delta, which is capable of expressing a variety of direct manipulations. It is important to note that the same notation is employed in Figure 4. In the core language, DF, the symbol $dv$ signifies the basic deltas that can be incorporated into the source program. The identity delta, id, leaves the output unaffected, while the replacement delta, repl $\mathit{aexp}$, substitutes the output value with the computed value from $\mathit{aexp}$. The composition of two deltas, represented as ${\mathit{dv}}_1{\circ }_{\Delta }{\mathit{dv}}_2$, signifies the manipulation of ${\mathit{dv}}_1$ on the output, followed by ${\mathit{dv}}_2$. The addition delta, ${+}_{\Delta }\mathit{atom}$, appends the computed value from $\mathit{atom}$ to the output.

The delta list constructor fabricates a list of deltas, ${\mathit{dv}}_1:{:}_{\Delta }{\mathit{dv}}_2$, that operates on a list. The list deletion delta, delete n, eliminates the nth element in the list, while the list insertion delta, insert$n\mathit{atom}$, introduces the computed value from $\mathit{atom}$ as the nth element. The list modification delta, modify $n\mathit{dv}$, updates the nth element by $\mathit{dv}$, and the list folding of delta, dfold, is analogous to foldl in Haskell. $\mathit{derive}$ is a function, defined using the core language DF, that accepts the current accumulator, $\mathit{acc}$ (an atomic expression), as input and derives a new accumulator, ${\mathit{acc}}^{\prime }$. In addition, it computes the arguments to be applied by the function $\mathit{todelta}$, which computes a delta for each list element being recursively processed. The binding operation, denoted as $\mathtt{intro}x\mathtt{by}\mathit{select}\mathtt{into}\mathit{dv}$, introduces a new variable x, extracts a sub-value from the output value using the selector $\mathit{select}$, and then binds the variable x to that sub-value. The variable x is subsequently utilized in $\mathit{dv}$. The selector $\mathit{select}$ is designed to extract a specific value from a data structure. It includes the identity selector id, and composition selectors with projectors comprised of head, tail, fst, and snd. The atomic expression $\mathit{atom}$ encompasses constants c, variables x, tuples, and lists. The arithmetic expression $\mathit{aexp}$ incorporates atomic expressions $\mathit{atom}$, additions ${\mathit{aexp}}_1+{\mathit{aexp}}_2$, multiplications ${\mathit{aexp}}_1\ast {\mathit{aexp}}_2$, and so forth.

Example 6 

(Transforming a Rectangle into a Square). Consider a canvas with one rectangle, whose width is 200, and height is 100.

$\left[\mathsf{rect}182\right(70,70\left)200100\right]$

The operation to turn the rectangle into a square is written as follows: It first extracts the rectangle’s width using the selector $\mathsf{head}\circ \mathsf{nth}\mathsf{3}\circ \mathsf{id}$ and assigns it to the newly introduced variable w. Then, it modifies the rectangle’s h with w.

$\mathtt{intro}w\mathtt{by}(\mathtt{head}\circ \mathtt{nth}\mathtt{3}\circ \mathtt{id})\mathtt{into}\left(\mathtt{modify}4\right(\mathtt{repl}w\left)\right)$

Example 7 

(Turn Two Rectangles Horizontally Attached). Consider a canvas with two rectangles spread horizontally, but not attached. The operation to direct the second rectangle to attach to the first rectangle horizontally is written as follows: It extracts the first rectangle’s x and width using the selector $\mathsf{head}\circ \mathsf{nth}\mathsf{2}\circ \mathsf{fst}\circ \mathsf{id}$ and $\mathsf{head}\circ \mathsf{nth}\mathsf{3}\circ \mathsf{id}$, and assigns it to the newly introduced variables x and w, respectively. Then, it modifies the second rectangle’s x with $x+w$.

$\begin{array}{cc}& \mathtt{intro}x\mathtt{by}(\mathtt{head}\circ \mathtt{nth}\mathtt{2}\circ \mathtt{fst}\circ \mathtt{id})\mathtt{into}\hfill \\ & \mathtt{intro}w\mathtt{by}(\mathtt{head}\circ \mathtt{nth}\mathtt{3}\circ \mathtt{id})\mathtt{into}\hfill \\ & \mathtt{modify}1(\mathtt{repl}x+w,id)\hfill \end{array}$

3.2. Semantics of Language DM

The semantics of DM  are defined in Figure 8. The judgment $\mathit{dv}▹v\to v^{\prime }$ indicates that “applying a delta $\mathit{dv}$ to the value v results in $v^{\prime }$”. The definition of values v, which includes constants with delta, tuples, and lists, is provided in Figure 4.

The D-Id rule asserts that the identity id leaves the value v unchanged. The D-Repl rule introduces a new value, denoted as $v^{\prime }$, derived from the arithmetic expression $\mathit{aexp}$. It is important to note that the original value v is not directly replaced by $v^{\prime }$. Instead, the change is recorded as a delta. eval is an auxiliary function that evaluates arithmetic expressions into values. The D-Com rule states that the composition ${\mathit{dv}}_2\circ {\mathit{dv}}_1$ is applied to v, resulting in a delta value that stores the composed deltas. The D-Add rule states that, when applying ${+}_{\Delta }\mathit{atom}$ to a numerical value n, the value v computed from $\mathtt{eval}\left(\mathit{atom}\right)$ is not directly applied to the value n, but is stored in a delta value. The D-Tuple rule states that, when applying the tuple delta ${({\mathit{dv}}_1,{\mathit{dv}}_2)}_{\Delta }$ to the tuple value $(v_1,v_2)$, $v_1$ (resp. $v_2$) is transformed by ${\mathit{dv}}_1$ (resp. ${\mathit{dv}}_2$) and becomes $v_1^{\prime }$ (resp. $v_2^{\prime }$). The D-Cons rule operates similarly to D-Tuple.

The application rules for deletion, insertion, and modification apply to list values. The D-Ins-1 rule states that when applying $\mathtt{insert}n\mathit{atom}$ to a list construction $v_1::v_2$ and n is larger than zero, $v_2$ is transformed by $\mathtt{insert}(\mathit{n}-1)\mathit{atom}$ and becomes $v_2^{\prime }$. The D-Ins-2 rule states that when n is zero, the computed value $v_1$ marked with delta insert is inserted at the front of the list $v_2$. The rules for deletion and modification follow a similar pattern. The D-Fold rule applies deltas to each element from left to right. With the recursion logic(i.e., $\mathit{derive}$) and the delta generation (i.e., $\mathit{todelta}$), $\mathit{acc}$ is an atomic expression denoting an accumulator, usually the list index. Let us use an example to show how the rule D-Fold works. The D-Binding rule is utilized for creating bindings from elements in the output. It initially introduces a new variable, denoted as x, to represent the value selected from an existing set using the $select$ function. The semantics of these selectors are elaborated in Figure 9. Subsequently, this selected value x is used to evaluate the change $dv$. This mechanism is employed in various applications such as defining alignment, joining points, and applying a format brush.

Example 8. 

Consider a sequence of rectangles on the canvas. The following operation sets all the rectangles’ widths to 100. The command $\mathsf{modify}\mathsf{3}\left(\mathsf{repl}\mathsf{100}\right)$ changes a specific rectangle’s width to 100. The dfold function iterates over a sequence of rectangles from left to right.

$\mathtt{dfold}(\lambda i.(0,i+1\left)\right)(\lambda x.\mathtt{modify}3(\mathtt{repl}100\left)\right)0$

Example 9. 

The operation below sets the first rectangle’s width to 20, the second rectangle’s width to 40, and so on. Initially, i is set to 1, and the variable $dis$ is bound to 20. After one iteration, i is updated to 2, and the value of $dis$ becomes 40.

$\mathtt{dfold}(\lambda i.(i\ast 20,i+1\left)\right)(\lambda dis.\mathtt{repl}3dis)1$

4. Fusing Delta into the Program

We first introduce the well-behavedness properties, followed by an example to illustrate how fusion operates. Subsequently, we present the primary fusion algorithms that fuse delta into program. Particularly, when program expressions are marked with const, the fusion rules become more intricate.

4.1. Well-Behavedness

Our system incorporates a fundamental bidirectional transformation mechanism, which comprises the standard evaluation (get function) and the fusion (putback function). These components maintain consistency between source programs and their outputs by satisfying the GetPut law and the WeakPutGet law.

Lemma 1 

(GetPut). If $E⊢e\Rightarrow v$, then $\mathsf{id}▹E⊢e⇝E⊢e$.

The GetPut law (stability) states that if we obtain an output from a program and perform no direct manipulation, the program must remain unchanged. This is straightforward because we default the delta modification to be the identity function (id), which keeps the same source program.

Lemma 2 

(WeakPutGet). If $E⊢e\Rightarrow v$, $\mathit{dv}▹v\to v^{\prime }$, and $\mathit{dv}▹E⊢e⇝E^{\prime }⊢e^{\prime }$, $E^{\prime }⊢e^{\prime }\Rightarrow v^{{\mathit{dv}}^{\prime }}$, then ${\mathit{dv}}^{\prime }▹E⊢e⇝E^{\prime }⊢e^{\prime }$.

The WeakPutGet law (soundness) is more critical. It states that when a delta is fused into a program, even if the output of executing the modified program differs, by collecting the changes and fusing them again into the original program, we should obtain the same modified program.

Theorem 1. 

If a program satisfies the GetPut law and the WeakPutGet law, then the program is well behaved.

We need to emphasize that we use the WeakPutGet law instead of the PutGet law. The PutGet law is more restrictive, such that the output of executing the modified program is equal to the output generated by applying that delta to the original output. We relax it by allowing the updated outputs to change, which is important for reconstructing the correct relations in the output.

4.2. Fusion Example

Figure 10a depicts the fusion of a delta ${+}_{\Delta }2$, which adds 2 to the output of a program. The fusion involves a depth-first traversal of the program’s structure, as illustrated in Figure 10b (where let-bindings are represented as function applications). The green lines, labeled with numbers, indicate the path of the propagation. Once the timesExp node is reached, the traversal proceeds to the left branch, and the delta transforms into ${+}_{\Delta }1$. Ultimately, the delta is fused with the constant value 1 at the node highlighted in yellow. The final program is shown in the bottom of Figure 10a.

If we carefully analyze the propagation steps, it becomes apparent that there are other reasonable points for updating, rather than solely the “deepest” one. For instance, consider updating function definitions of plus or times. Since there are binary operations, we have the flexibility to update either operand. To address this, we introduce the const marker to indicate which parts we prefer to keep unchanged. For example, if we wish to preserve value of the let binding of the variable a and aim to update the plus function, we can mark the binding of a and the application times a as unmodifiable using const. The revised and updated program is presented as shown in Figure 11.

4.3. Fusion Rules

Fusion rules are the core mechanisms that propagate deltas across the main program structure. The key idea is to propagate the delta back to variables or sub-expressions using variable bindings (environment). Given a delta $\mathit{dv}$, under value environment E, and Boolean environment $E_b$, update expression e to $e^{\prime }$, and update the value environment to $E^{\prime }$. The fusion rules are depicted in Figure 12 and Figure 13.

• Constants. There are two rules for constant values. If a constant value is marked by the const keyword, it remains immutable (since the inference rule will always yield to true for a constant marked by the const keyword, only $\mathit{id}$ is allowed). If the constant value c is not marked by the const keyword, it can be updated to $c^{\mathit{dv}}$, as demonstrated in P-Con2. Note that we do not directly compute the result of applying $\mathit{dv}$ to value c, but preserve the delta, which is important for proving the WeakPutGet law.

• Variable Expressions. There are four rules for variable expressions based on whether the variable expression x is marked by the const keyword and whether the bound value for the variable is modifiable. It is crucial to differentiate between the variable expression x and the variable binding for x. The first rule P-Var1 states that if the bound value for the variable is not modifiable and the variable expression is marked by const, then only the delta $\mathit{id}$ is permitted. The second rule P-Var2 states that if the bound value for the variable is not modifiable, but the variable expression is modifiable, then modify the expression to $x^{\mathit{dv}}$. The third rule P-Var3 states that if the bound value for the variable is modifiable, and the variable expression is marked by const, then the variable x in the environment E is bound to a new delta ${\mathit{dv}}_{\mathit{1}}\circ \mathit{dv}$ that composes ${\mathit{dv}}_{\mathit{1}}$ with the original delta $\mathit{dv}$. The last rule P-Var4 states that if the bound value for the variable is modifiable, and the variable expression is not marked by const, then despite the fact that updating the variable expression (as in P-Var2) or the variable binding (as in P-Var3) is reasonable, updating the variable, which yields a result identical to P-Var3, is the preferred approach. The last rule P-Var4 is the same as the P-Var defined in FuseDM [9].

• Lambda Abstractions. The rule P-Lam1 states that, when applying the closure $(E^{\prime },E_b,\mathtt{const}\lambda p.e)$, the program $E,E_b⊢\mathtt{const}\lambda p.e$ becomes $E^{\prime },E_b⊢\mathtt{const}\lambda p.e$. Since the lambda abstraction is marked by the const keyword, the body of the lambda abstraction (e) cannot be modified; only the bindings of the variables in the environment are updated. The lambda abstraction in P-Lam2 is not marked by the const keyword, and the fusion can be decomposed into two updates: replacing the function body with $e^{\prime }$ and the environment with the updated one $E^{\prime }$.

• Function Applications. The rule P-App1 states that, if the inference result of the application is true, the application is not modifiable. The rule P-App2 states that, if the inference result of expression $e_2$ is true, then only $e_1$ can be updated. When fusing delta $dv$ to $e_f$, even though there exists a possibility to update x, the fusion algorithm will refrain from updating x. We first evaluate $e_1$ to a lambda closure, a tuple of an environment $E_f$, and a lambda abstraction $\lambda x.e_f$. For simplicity, we assume only variable patterns in the lambda abstraction. Under the binding of variable x to $v_2$, which is computed from $e_2$ and environment $E_f$, we update the expression $e_f$ and environment $E_f$ to $e_f^{\prime }$ and $E_f^{\prime }$, respectively. And finally, we use the updated expression $e_f^{\prime }$ and environment $E_f^{\prime }$ to update expression $e_1$ to $e_1^{\prime }$, and environment E to $E_1$. The rule P-App3 is a bit complicated, since both $e_1$ and $e_2$ can be updated, thus they may encounter conflicts. A merge operation is defined for resolving conflicts. The intuitive idea is explained by the example “fusing ${({+}_{\Delta }2,{+}_{\Delta }3)}_{\Delta }$ into the program $(\varnothing ,\{a\mapsto 0^{\mathtt{id}}\},\{a\mapsto \mathit{false}\})⊢(\lambda x.(x,a))a$” in Table 1.

The operator ${E_1}^{e_1^{\prime }}{\otimes }^{e_2^{\prime }}{E}_2$ in Step 7 first merges variable bindings obtained from fusion of $e_1$ and $e_2$, and then embeds inconsistent deltas into $e_1^{\prime }$ (resp. $e_2^{\prime }$) to obtain $e_1^{\prime \prime }$ (resp. $e_2^{″}$), which is formally defined in Section 4.4. The final program is as follows, which evaluates to $(2,3)$.

$(\{a\mapsto 0^{\mathtt{id}}\},\{a\mapsto \mathit{false}\})⊢(\lambda x.(x,a+3))(a+2):$

• Case Expressions. The rule P-Case1 states that, if the inference result of the case expression is true, the expression is not modifiable. The rule P-Case2 states that, if the inference result of $e_0$ is true, then only the matched case branch expression $e_j$ is updated. The rule P-Case3 first applies the delta $\mathit{dv}$ to the selected branch $e_j$, resulting in the updated $e_j^{\prime }$ and $E_m^{\prime }$. Then, it collects the delta from $E_m^{\prime }$ for $e_0$, and subsequently fuses it into $e_0$. Finally, the merge operator is used to resolve any conflicts.

• Binary Expressions. The rule P-C-Oplus1 states that if a binary expression marked with const is inferred as true, then the expression is not modifiable. The rule P-C-Oplus2 states that updating a const marked binary expression is the same as updating the binary expression with each operand marked by const. The rule P-Oplus1 states that if both $e_1$ and $e_2$ are inferred as true, then update the entire binary expression with $\mathit{dv}$. The rule P-Oplus2 states that if $e_1$ is inferred as true and $e_2$ is inferred as false, then update the right operand $e_2$ by a computed delta $d{v}_{{\oplus }_{v_1}}$. The rule P-Oplus3 updates $e_1$ instead. The rule P-Oplus4 states that, if both $e_1$ and $e_2$ are inferred as false, then it chooses to update $e_1$, but a merge operation is needed to resolve conflicts. How to apply deltas to binary expressions varies in different deployments; we give potential heuristic rule (P-Oplus4) for applying deltas to left operands. Other strategies include modifying the right operands, which can be achieved by marking the left operand with const.

• Tuples. The rule P-C-Tuple states that updating a const marked tuple is the same as updating each element marked by const. The rule P-Tuple states that one must update $e_1$ with $d{v}_1$ and $e_2$ with $d{v}_2$. Since $e_1$ and $e_2$ may share variables, then a conflict-resolving operation is needed. The rule P-C-List (resp. P-List) is similar to the rule P-C-Tuple (resp. P-Tuple).

4.4. Merge Operation

Intuitively, the merge operator ${E_1}^{e_1}{\otimes }^{e_2}{E}_2$ compares two structurally equivalent environments using the comparing operator ${}^{e_2}\times ^{e_1}$ and then embeds conflicting bindings into sub-expressions using the embedding operator ⊙. The merge operator and the comparing operator are the same as the definition in FuseDM, except the definition of embedding operator.

Definition 1 

(Merge Operator). The merge operator ${E_1}^{e_1}{\otimes }^{e_2}{E}_2$ reconciles conflicts of variable bindings as follows. $(E^M,e_1^{\prime },e_2^{\prime })={E_1}^{e_1}{\otimes }^{e_2}{E}_2$ holds if

$(E^M,E_1^I,E_2^I)={E_1}^{e_1}{\times }^{e_2}{E}_2,e_1^{\prime }=E_1^I\odot e_1,\mathrm{and}{e}_2^{\prime }=E_2^I\odot e_2.$

Definition 2 

(Comparing Operator). The comparing operator ${}^{e_2}\times ^{e_1}$ compares two environments $E_1$ and $E_2$ and returns the successfully merged part $E^M$ and two inconsistent parts ($E_1^I$ and $E_2^I$).

$\begin{array}{cc}\hfill & {(E_1,x\mapsto v^{{\mathit{dv}}_1})}^{e_1}{\times }_{E_b}^{e_2}(E_2,x\mapsto v^{{\mathit{dv}}_2})\hfill \\ \hfill & =\left\{\begin{array}{cccc}\hfill & ((E^M,x\mapsto v^{{\mathit{dv}}_1}),E_1^I,E_2^I)\hfill & \hfill & \mathit{if}{\mathit{dv}}_1={\mathit{dv}}_2\hfill \\ \hfill & ((E^M,x\mapsto v^{{\mathit{dv}}_1}),E_1^I,E_2^I)\hfill & \hfill & \mathit{if}x\notin \mathit{fv}\left(e_2\right)\hfill \\ \hfill & ((E^M,x\mapsto v^{{\mathit{dv}}_2}),E_1^I,E_2^I)\hfill & \hfill & \mathit{if}x\notin \mathit{fv}\left(e_1\right)\hfill \\ \hfill & ((E^M,x\mapsto v^{{\mathit{dv}}^{\prime }}),(E_1^I,x\mapsto v^{{\mathit{dv}}_1^{\prime }}),(E_2^I,x\mapsto v^{{\mathit{dv}}_2^{\prime }}))\hfill & \hfill & \mathit{otherwise}\hfill \\ \hfill & \mathit{where}({\mathit{dv}}^{\prime },{\mathit{dv}}_1^{\prime },{\mathit{dv}}_2^{\prime })={\mathit{dv}}_{\mathit{1}}{\wedge }_{\mathit{prefix}}{\mathit{dv}}_{\mathit{2}}\hfill & \hfill \end{array}\right.\hfill \\ \hfill & \mathit{where}(E^M,E_1^I,E_2^I)={E_1}^{e_1}{{\times }_{E_b}}^{e_2}{E}_2\hfill \end{array}$

Definition 3 

(Embedding Operator). The embedding operator $E^I\odot e$ embeds deltas bound in the mapping $E^I$ around free variables in e, denoted by

$(E^I,x\mapsto v^{\mathit{dv}})\odot e=E^I\odot e[x\mapsto x^{\mathit{dv}}],$

where the “$[x\mapsto x^{\mathit{dv}}]$” operator replaces the free occurrences of the variable x in e with $x^{\mathit{dv}}$.

The embedding operator introduces deltas at the variable expression x, by substituting x with $x^{\mathit{dv}}$ to encapsulate the delta $\mathit{dv}$ at the variable expression x. This approach differs from the mechanism employed in FuseDM [9]. However, when the program with deltas is evaluated to a program without deltas, the same embedding mechanism as described in FuseDM is utilized.

4.5. Well-Behavedness

The well-behavedness (Theorem 1) properties includes the GetPut law and the WeakPutGet law. The GetPut law is straightforward because when the direct manipulation is the identity id, it then keeps the same source program as it is. Proving the well-behavedness is essentially the same as verifying the WeakPutGet law. We give a sketch of the proof for the WeakPutGet law.

Proof. 

By induction on the fusion derivation below, we show the P-Con2, P-Var3, and P-App3 three cases as representatives:

1. The P-Con2 case:

   $\frac{ }{\mathit{dv}▹E,E_b⊢c⇝E,E_b⊢c^{\mathit{dv}}}$

   • (a). According to P-Con2, the updated program is $E,E_b⊢c^{\mathit{dv}}$.

   • (b). By E-dv, $E⊢c^{\mathit{dv}}\Rightarrow c^{\mathit{id}\circ \mathit{dv}}$.

   • (c). The delta is $\mathit{id}\circ \mathit{dv}$.

   • (d). By P-Con2, $\mathit{id}\circ \mathit{dv}▹E,E_b⊢c⇝E,E_b⊢c^{\mathit{id}\circ \mathit{dv}}$.

   • (e). Since $\mathit{id}\circ \mathit{dv}$ is the same as $\mathit{dv}$, $E,E_b⊢c^{\mathit{id}\circ \mathit{dv}}$ is the same as $E,E_b⊢c^{\mathit{dv}}$, which is the goal.

The P-Var3 case:

$\frac{E_b\left(x\right)=\mathit{false}E\left(x\right)=v^{id}}{\mathit{dv}▹E,E_b⊢\mathtt{const}x⇝E[x\mapsto v^{\mathit{dv}\circ \mathit{id}}],E_b⊢\mathtt{const}x}$

• (a). According to P-Var3, the updated program is the same as the original one, but the environment changes, $E[x\mapsto v^{\mathit{dv}\circ \mathit{id}}],E_b⊢\mathtt{const}x$.

• (b). By E-Const, and E-Var, $E[x\mapsto v^{\mathit{dv}\circ \mathit{id}}],E_b⊢\mathtt{const}x\Rightarrow v^{\mathit{dv}\circ \mathit{id}}$.

• (c). The delta is $\mathit{dv}\circ \mathit{id}$.

• (d). By P-Var3, $\mathit{dv}\circ \mathit{id}▹E,E_b⊢\mathtt{const}x⇝E[x\mapsto v^{\mathit{dv}\circ \mathit{id}\circ \mathit{id}}],E_b⊢\mathtt{const}x$.

• (e). Since $\mathit{dv}\circ \mathit{id}\circ \mathit{id}$ is the same as $\mathit{dv}\circ \mathit{id}$, $E[x\mapsto v^{\mathit{dv}\circ \mathit{id}\circ \mathit{id}}],E_b⊢\mathtt{const}x$ is the same as $E[x\mapsto v^{\mathit{dv}\circ \mathit{id}}],E_b⊢\mathtt{const}x$, which is the goal.

The P-App3 case:

$\frac{\begin{array}{c}E,E_b⊢e_2{\to }_e\mathit{false}E⊢e_2\Rightarrow v_{2}E⊢e_1\Rightarrow (E_f,\lambda x.e_f)\\ \mathit{dv}▹E_f\cup \{x\mapsto v_2^{\mathtt{id}}\},E_b\cup \{x\mapsto \mathit{false}\}⊢e_f⇝E_f^{\prime }\cup \{x\mapsto v_2^{{\mathit{dv}}_{\mathit{2}}\circ \mathtt{id}}\},E_b\cup \{x\mapsto \mathit{false}\}⊢e_f^{\prime }\\ (E_f^{\prime },\lambda x.e_f^{\prime })▹E,E_b⊢e_1⇝E_1,E_b⊢e_1^{\prime }\\ {\mathit{dv}}_2▹E,E_b⊢e_2⇝E_2,E_b⊢e_2^{\prime }(E^{\prime },e_1^{″},e_2^{″})={E_1}^{e_1^{\prime }}{{\otimes }_{E_b}}^{e_2^{\prime }}{E}_2\end{array}}{\mathit{dv}▹E,E_b⊢e_1{e}_2⇝E^{\prime },E_b⊢e_1^{\prime \prime }{e}_2^{″}}$

• (a). According to P-, ${\mathit{dv}}_2▹v_2\to v_2^{\prime }$.

• (b). By the induction hypothesis, $E_f^{\prime }\cup \{x\mapsto v_2^{{\mathit{dv}}_{\mathit{2}}\circ \mathtt{id}}\},E_b\cup \{x\mapsto \mathit{false}\}⊢e_f^{\prime }\Rightarrow v^{{\mathit{dv}}^{\prime }}$, $E_1,E_b⊢e_1^{\prime }\Rightarrow (E_f^{\prime },\lambda x.e_f^{\prime })$, and $E_2,E_b⊢e_2^{\prime }\Rightarrow v_2^{{\mathit{dv}}_{\mathit{2}}^{\prime }}$.

• (c). By Lemma 4, $E^{\prime },E_b⊢e_1^{″}\Rightarrow (E_f^{\prime },\lambda x.e_f^{\prime })$ and $E^{\prime },E_b⊢e_2^{″}\Rightarrow v_2^{{\mathit{dv}}_{\mathit{2}}^{\prime }}$.

• (d). By (c) and Lemma 3, $E_f^{\prime },x\mapsto v_2^{{\mathit{dv}}_2^{\prime }\circ \mathtt{id}}⊢e_f^{\prime }\Rightarrow v^{{\mathit{dv}}^{\prime }}$.

• (e). By E-App, $E^{\prime },E_b⊢e_1^{″}{e}_2^{″}\Rightarrow v^{{\mathit{dv}}^{\prime }}$.

• (f). By the induction hypothesis, ${\mathit{dv}}^{\prime }▹E_f\cup \{x\mapsto v_2^{\mathtt{id}}\},E_b\cup \{x\mapsto \mathit{false}\}⊢e_f⇝E_f^{\prime }\cup \{x\mapsto v_2^{{\mathit{dv}}_{\mathit{2}}^{\prime }\circ \mathtt{id}},E_b\cup \{x\mapsto \mathit{false}\}⊢e_f^{\prime }\}$, ${\mathit{dv}}_2^{\prime }▹E,E_b⊢e_2⇝E_2,E_b⊢e_2^{\prime }$.

• (g). According to definition, $(E_f^{\prime },\lambda x.e_f^{\prime })▹E,E_b⊢e_1⇝E_1,E_b⊢e_1^{\prime }$ and $(E^{\prime },e_1^{\prime \prime },e_2^{\prime \prime })={E_1}^{e_1^{\prime }}{{\otimes }_{E_b}}^{e_2^{\prime }}{E}_2$, the updated program is $E^{\prime },E_b⊢e_1^{″}{e}_2^{″}$, which is the goal.

   □

Definition 4 

(Equivalent Environment). $E_1$ and $E_2$ are equivalent (denoted $E_1\equiv E_2$) if they are structurally equivalent and $\forall x\in E_1$, $E_1⊢x\Rightarrow v$, and $E_2⊢x\Rightarrow v$.

Lemma 3. 

If $E_1\equiv E_2$ and $E_1⊢e\Rightarrow v$, then $E_2⊢e\Rightarrow v$.

Lemma 4 

(Merge Equivalency). If $(E,e_1^{\prime },e_2^{\prime })={E_1}^{e_1}{\otimes }_{E_b}^{e_2}{E}_2$. $\forall E_0,E_0\cup E_1⊢e_1\Rightarrow v$ implies $E_0\cup E⊢e_1^{\prime }\Rightarrow v$.

Since Lemmas 3 and 4 are straightforward, we omit the details here.

5. Evaluation

The evaluation aims to analyze the expressiveness of the language to cover the scenario of SVG design. In addition, we present two typical benchmark examples in detail to show how an SVG graphic is constructed.

5.1. Expressiveness

FuseSVG introduces the const keyword, enabling users to mark expression to narrow down the appropriate location from multiple options, ensure certain constants in the program remain unaltered, and preserve relationships between SVG elements within the program. To illustrate the expressiveness of FuseSVG (FuseSVG is available at https://github.com/zanlyun/FuseDMI, accessed on 19 February 2025), we have replicated 14 benchmark examples and conducted an analysis from these three perspectives, as shown in Figure 14. All of these examples originate from FuseDM [9]. The results are summarized in Table 2. The term ‘Multiple update choice’ refers to situations where there are several possible places for making updates, such as the left or right operand of a binary operation. ‘Immutability for a single value’ suggests that the programmer intends to set certain constant values as non-updatable, like the x and y coordinates of a rectangle or the center of an ellipse. The ‘Number of relations’ corresponds to the count of relation expressions defined in the program. For instance, the width of a rectangle being twice its height ($2\ast h$) is considered a relation.

Based on our analysis of the 14 benchmark examples, all examples encounter the problem of selecting suitable sub-expressions for updating when fusing deltas into the program. All examples necessitate the setting of certain constant values as non-updatable. For example, in the Precision Floor Plan example, there are three relations: the y coordinate of the second rectangle is identical to that of the first rectangle; the x coordinate of the second rectangle should be the sum of the first rectangle’s x coordinate and its width w; and both rectangles share the same height h. Relation expressions are defined across all examples and can be viewed as an important property in SVG design, emphasizing the importance of defining the invariance of relations in the source program.

Now, let us perform a detailed comparison between FuseDM and FuseSVG using the Precision Floor Plan Example. Listings 1 and 2 show the actual code for FuseDM and FuseSVG, respectively, which generates the output SVG shown in Figure 14(1). From a code perspective, compared to FuseDM, FuseSVG introduces const markers to specify that certain relations should be preserved, ensuring that certain expressions cannot be updated.

Listing 1. Precision Floor Plan Example in FuseDM.

main = let rect1_x=175 in let rect1_y=239 in let rect1_width=60 in let rect1_height=106 in [   rect[0,219,rect1_x,rect1_y,rect1_width,rect1_height],   rect[0,30,rect1_x+rect1_width,rect1_y,rect1_width*2,rect1_height] ];

Listing 2. Precision Floor Plan Example in FuseSVG.

main = let rect1_x=175 in let rect1_y=239 in let rect1_width=60 in let rect1_height=106 in {  rect{0,219,const rect1_x,rect1_y,rect1_width,rect1_height},  rect{0,30,const(rect1_x+rect1_width),rect1_y,const rect1_width*2,rect1_height} };

When generating the SVG, we have a specific intent: Suppose the left-hand rectangle is rect1, and the right-hand one is rect2. For instance, rect2’s width is defined as twice rect1’s width (rect1_width * 2), and rect2 adheres to rect1, where rect2’s x is rect1’s x plus rect1_width, and rect2’s y is the same as rect1’s y.

Next, we perform three update operations on both FuseDM and FuseSVG, as shown in Figure 15. First, if we shrink rect2’s width by 16, FuseDM modifies rect2’s width expression to $(\mathtt{rect}\mathtt{1}\_\mathtt{width}-8)\ast 2$, which breaks the intended relationship. However, in FuseSVG, this update proportionally adjusts rect1’s width, preserving the relationship by shrinking it from 60 to 52.

Similarly, if we shrink rect1’s width by 13, the expression for rect1_width becomes $\mathtt{rect}\mathtt{1}\_\mathtt{width}-13$. In FuseDM, this breaks the relationship where rect2’s width is always twice rect1’s width. However, in FuseSVG, the width of rect2 is proportionally adjusted, preserving the intended relationship between the two rectangles by updating rect1_width to 47.

Finally, when we move rect2 horizontally by +10, FuseDM updates the expression $\mathtt{rect}\mathtt{1}\_\mathtt{x}+\mathtt{rect}\mathtt{1}\_\mathtt{width}$ to $(\mathtt{rect}\mathtt{1}\_\mathtt{x}+10)+\mathtt{rect}\mathtt{1}\_\mathtt{width}$, breaking the alignment. In contrast, FuseSVG updates rect1_x directly, ensuring that the relative positioning of the rectangles is preserved. If we further want to keep rect1_x stable, we can mark rect1_x as const, ensuring that only rect1_width is updated.

In summary, FuseDM propagates updates to constants whenever possible while guaranteeing the PutGet property. However, this can sometimes lead to situations where relationships between elements are broken during updates, particularly when elements are interdependent or when updates occur in unintended locations. On the other hand, FuseSVG introduces more sophisticated mechanisms, such as using the const keyword for immutability, which helps preserve these relationships while still allowing for flexible updates. This ensures that changes to one element do not unintentionally disrupt the overall structure of the program.

5.2. Case Study

Let us use two benchmark examples to show how an SVG graphic is constructed step-by-step.

Battery Example: Initially, the canvas is empty, and the program assigns an empty list ([]) to a main variable. To draw a battery, we first draw a rectangle as shown in Figure 16a. This action automatically generates a piece of program, as shown on the right side, which is a main function that contains a list with the rectangle we just drew. The number 120 represents the color of the rectangle, while 119 and 146 are the x and y coordinates of the rectangle, respectively. The values 178 and 76 represent the width and height of the rectangle, respectively. By using direct manipulation, we eliminate the need to write the definition of the rectangle, especially the actual value. Sketching on the canvas is more intuitive than writing code. Subsequently, we draw another rectangle adjacent to the first one, as illustrated in Figure 16b. This action generates a delta, which is then fused into the program.

Although we placed the second rectangle next to the first one, the generated code does not inherently reflect this intention. The x value of the second rectangle (296) is very close to the sum of the x value and the width of the first rectangle (297). Therefore, we can refactor the code to establish relations by (1) abstracting values as variables and (2) constructing relations and marking them as non-modifiable with the const keyword. The final result is presented in Figure 16c. Now, regardless of whether we move or resize the first or second rectangle, the relations are consistently preserved.

Balance Scale Example: Figure 17 outlines the steps to construct a balanced scale (for more details, please refer to the video at the provided URL https://youtu.be/ImoOpGfubBw, accessed on 19 February 2025). After drawing the vertical red line in the second step, it is not possible to verify if the line is truly vertical just by looking at the SVG. Therefore, we need to check the code and adjust the y value to ensure it is truly vertical. Next, a refactoring step is needed to establish the relationship between the ellipse created in the first step and the vertical line. In fact, after each drawing step, a refactoring step is necessary to build the relationships, particularly the addition of the const marker, to the generated code. The construction of the balanced scale involves 20 steps and results in 26 lines of code. Since the relationships are well defined and marked as non-modifiable, any movement of an element in the figure will lead to changes in related elements.

6. Related Work

6.1. Bidirectional Transformations

Bidirectional transformations [10] serve as a mechanism for maintaining consistency between a source and a corresponding view, which is derived from that source. The programming language community has conducted extensive research on bidirectional transformations [11,12,13]. Existing approaches to bidirectional transformations have primarily focused on transformations over a variety of structured data [14]. This includes lists [15], trees [11,16,17], graphs [18,19], and relational databases [20]. Matsuda et al. [21] proposed FliPpr, a system designed to derive parsers from pretty-printers. None of the existing approaches permit modifications to the transformation program itself. In other words, updates to the view must remain within the scope of the forward transformation. However, our work, FuseSVG, supports updating the program to a new one. Cheney et al. [22] discussed the principle of least change for bidirectional transformations. They posited that a bidirectional transformation should only make changes to the artifact that are strictly necessary, which is a principle that can guide the design of backward transformations, but it does not allow for the updating of the bidirectional program.

6.2. Ambiguity and Updatability

Sketch-n-Sketch [4] employs heuristics to resolve ambiguities and introduces an approach based on trace-based program synthesis. This method tracks the origin of constants within the program and establishes value-trace equations with the manipulated output. These equations are subsequently solved using a constraint solver. As mentioned by the author:“Our approach utilizes heuristics to automatically resolve ambiguities, aiming to achieve a balance between interactivity and predictability.” For instance, consider the value-trace equation $n=(+x_0(\ast l_{0}sep))$. This equation can be solved by arbitrarily altering the value of either $x_0$ or $sep$.

Bidirectional Evaluation [7], Bidirectional Preview [23], and BiOOP [24], typically, introduce a backward update evaluation that propagates the updated output value to program constants by retracing the forward evaluation steps. Sketch-n-Sketch [6] provides semi-automated SVG programming via direct manipulation, and it is developed to further provide full output-directed programming for SVG [8]. During which, a freeze annotation, written !, is introduced to tell Sketch-n-Sketch not to change those constants marked by ! when shapes are moved on the canvas. The freeze operator is formally defined in the LITTLELEO language [7]. For a given expression marked by freeze ($freezee$), evaluation of this expression is the same as evaluating e, but for updating; if expression e evaluates to v, then only v can be propagated back, which means the value of the expression cannot be changed.

The motivation for introducing the keyword const differs from that of the freeze operation. While the freeze operation prohibits the changing of the value computed from the expression, our const keyword restricts the modification of the expression syntactically; however, the value can still be changed.

Let us assume that the output of the Sketch-n-Sketch program on the left, shown in Listing 3, represents the value of h. This output value is immutable, as the expression used to calculate h is marked with a freeze annotation(!). In contrast, within our system, FuseSVG, the const keyword is used to keep the expression 2*w static, while allowing for the update of the width w. The program is shown in Listing 4. The design goal of const is used to keep the form of the expression unchanged, which is useful for keeping the relation between two or more variables. The freeze totally prohibits any update to the variable in the expression. When const operates on constant values, it also does not allow modification of the constant value expression, which is, in fact the same as the freeze operation in Sketch-n-Sketch.

Listing 3. Sketch-n-Sketch.

w = 50                         h = !(2*w)

Listing 4. FuseSVG.

w = 50                         h = const (2*w)

FuseDM [9] adopts an operation-based approach that propagates deltas. This method can modify not only constant literals in programs but also alter program structures by integrating direct manipulations into the ‘proper position’. FuseDM has the same ambiguity issue as the state-based approaches. The application rules for fusing delta into addition expression A-Add and multiplication expression A-Mul defaults to update the left operand. Our work FuseSVG extends FuseDM by providing a const keyword, to describe user intention of which part in the program shall not be modified, to reduce the ambiguity and gain more control over the program when being updated.

6.3. Constraints on View

There are scenarios such that an output may include derived values that are calculated from other components of the output, necessitating that the output be consistently updated. For instance, if a rectangle’s width is twice as its height, the width is determined based on the height. The dependency problem on the output is discussed in the putback-based bidirectional transformation language BiGUL [25,26]; since in bidirectional transformation the output is the view, it is called view dependency. A constructor named $\mathit{Dep}$ accepts an additional function to describe the relation of two view elements. During backward transformation, the function is computed to check whether the two elements are consistent with the relation.

Several visual design systems [27,28,29,30] integrate constraint specification. Apparatus [30] is a hybrid graphics editor and programming environment, which is used for creating interactive diagrams. Upon drawing a rectangle, one can utilize arithmetic expressions to define the value of the attributes. For instance, Scale X is defined to be twice the value of Scale Y, as depicted in Figure 18. Consequently, stretching the rectangle either horizontally or vertically results in changes to Scale Y, while the constraint remains intact.

7. Conclusions

This paper presents a bidirectional live programming tool for SVG manipulation, which is capable of controlling the updatability of expressions by marking them with the const keyword. Moreover, by designating the relation expression as unmodifiable using the const keyword, it enables the update of other related elements to re-establish the relation on the output when a related element in the output is altered. From a theoretical perspective, we relaxed the PUTGET property to the WeakPUTGET property, facilitating the reconstruction of relations on the output. We ensure that both the evaluation semantics and the fusion algorithm comply with the well-behavedness properties. To demonstrate the adaptability of our approach, we have effectively implemented 14 benchmark examples that illustrate a variety of updatability scenarios.

Looking ahead, there are two potential areas for improvement. Firstly, although the relation can currently be constructed in the output, it is directly fused into the source program. A more efficient approach would be to store the relation in the output as well. This would allow for the corresponding update of a related element when one element changes, ensuring the consistency of relations in both the program and the output. Secondly, the task of inferring the inherent relations programmed in the source to the output needs to be addressed. Additionally, the delta language could potentially be extended to incorporate recursive drawing [31]. This technique, which employs recursive algorithms to generate complex, often fractal-like, graphics, could significantly enhance the versatility and expressiveness of the delta language in graphical representation.

Author Contributions

Conceptualization, T.Z., X.H. and Z.H.; methodology, T.Z., X.Z., X.H. and Z.H.; software, T.Z. and X.Z.; validation, T.Z., X.Z. and X.H.; investigation, T.Z., X.Z. and X.H.; writing–original draft preparation, T.Z., X.Z. and X.H.; writing–review and editing, T.Z., X.Z., X.H. and Z.H. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data are contained within the article.

Conflicts of Interest

The authors declare no conflicts of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Table 1

Detailed explanation of P-App3 with an example.

	Detail	Example
1	Evaluate $e_2$ to value.	$\{a\mapsto 0^{\mathtt{id}}\}⊢a\Rightarrow 0$
2	Evaluate $e_1$ to closure.	$\{a\mapsto 0^{\mathtt{id}}\}⊢(\lambda x.(1,x,a))\Rightarrow (\{a\mapsto 0^{\mathtt{id}}\},\lambda x.(1,x,a))$
3	Fuse the delta $\mathit{dv}$ into the function body $e_f$ and obtain $e_f^{\prime }$.	${({+}_{\Delta }2,{+}_{\Delta }3)}_{\Delta }▹(\{a\mapsto 0^{\mathtt{id}},x\mapsto 0^{\mathtt{id}}\},\{a\mapsto \mathit{false},x\mapsto \mathit{false}\})⊢(x,a)$ $⇝(\{a\mapsto 0^{{+}_{\Delta }3\circ \mathtt{id}},x\mapsto 0^{{+}_{\Delta }2\circ \mathtt{id}}\},\{a\mapsto \mathit{false},x\mapsto \mathit{false}\})⊢(x,a)$
4	Construct the function closure $(E_f^{\prime },E_{b}f,\lambda x.e^{\prime })$.	$(\{a\mapsto 0^{{+}_{\Delta }3\circ \mathtt{id}}\},\{a\mapsto \mathit{false}\},\lambda x.(x,a))$
5	Fuse the modified function closure into $e_1$.	$((\{a\mapsto 0^{{+}_{\Delta }3\circ \mathtt{id}}\},\{a\mapsto \mathit{false}\}),\lambda x.(x,a))▹(\{a\mapsto 0^{\mathtt{id}}\},\{a\mapsto \mathit{false}\})$ $⊢\lambda x.(x,a)⇝(\{a\mapsto 0^{{+}_{\Delta }3\circ \mathtt{id}}\},\{a\mapsto \mathit{false}\})⊢\lambda x.(x,a)$
6	Fuse the delta ${\mathit{dv}}_2$ into $e_2$.	${+}_{\Delta }2▹(\{a\mapsto 0^{\mathtt{id}}\},\{a\mapsto \mathit{false}\})⊢a⇝(\{a\mapsto 0^{{+}_{\Delta }2\circ \mathtt{id}}\},\{a\mapsto \mathit{false}\})⊢a$
7	Merge environments and solve conflicts.	${\{a\mapsto 0^{{+}_{\Delta }3\circ \mathtt{id}}\}}^{\lambda x.(x,a)}{\otimes }^a\{a\mapsto 0^{{+}_{\Delta }2\circ \mathtt{id}}\}$ $=(\{a\mapsto 0^{\mathtt{id}}\},\lambda x.(x,a+3)(a+2))$

Table 2

Analysis of benchmarks.

ID	Example	Multiple Update Choice	Immutability for Single Value	Number of Relations
1	Precision Floor Plan	✓	✓	3
2	Mondrian Arch	✓	✓	6
3	Balance Scale	✓	✓	15
4	Box Volume	✓	✓	21
5	Battery	✓	✓	2
6	Ladder	✓	✓	10
7	Logo	✓	✓	6
8	N Boxes	✓	✓	6
9	Ferris Wheel	✓	✓	54
10	Tree Branch	✓	✓	18
11	Target	✓	✓	6
12	Pencil Tip	✓	✓	11
13	Arrows	✓	✓	28
14	Rails	✓	✓	38

References

1. McDirmid, S. Living it up with a live programming language. Proceedings of the 22nd Annual ACM SIGPLAN Conference on Object-Oriented Programming Systems, Languages and Applications, OOPSLA ’07; Montreal, QC, Canada, 21–25 October 2007; pp. 623-638. [DOI: https://dx.doi.org/10.1145/1297027.1297073]

2. Burckhardt, S.; Fahndrich, M.; de Halleux, P.; McDirmid, S.; Moskal, M.; Tillmann, N.; Kato, J. It’s alive! continuous feedback in UI programming. SIGPLAN Not.; 2013; 48, pp. 95-104. [DOI: https://dx.doi.org/10.1145/2499370.2462170]

3. McDirmid, S. The promise of live programming. Proceedings of the 2nd International Workshop on Live Programming, LIVE; Rome, Italy, 2016; Volume 16.

4. Chugh, R.; Hempel, B.; Spradlin, M.; Albers, J. Programmatic and direct manipulation, together at last. Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation; Santa Barbara, CA, USA, 13–17 June 2016; [DOI: https://dx.doi.org/10.1145/2908080.2908103]

5. Shneiderman. Direct Manipulation: A Step Beyond Programming Languages. Computer; 1983; 16, pp. 57-69. [DOI: https://dx.doi.org/10.1109/MC.1983.1654471]

6. Hempel, B.; Chugh, R. Semi-Automated SVG Programming via Direct Manipulation. Proceedings of the 29th Annual Symposium on User Interface Software and Technology, UIST ’16; Tokyo, Japan, 16–19 October 2016; pp. 379-390. [DOI: https://dx.doi.org/10.1145/2984511.2984575]

7. Mayer, M.; Kuncak, V.; Chugh, R. Bidirectional Evaluation with Direct Manipulation. Proc. ACM Program. Lang.; 2018; 2, pp. 1-28. [DOI: https://dx.doi.org/10.1145/3276497]

8. Hempel, B.; Lubin, J.; Chugh, R. Sketch-n-Sketch: Output-Directed Programming for SVG. Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, UIST ’19; New Orleans, LA, USA, 20–23 October 2019; pp. 281-292. [DOI: https://dx.doi.org/10.1145/3332165.3347925]

9. Zhang, X.; Xie, R.; Guo, G.; He, X.; Zan, T.; Hu, Z. Fusing Direct Manipulations into Functional Programs. Proc. ACM Program. Lang.; 2024; 8, pp. 1211-1238. [DOI: https://dx.doi.org/10.1145/3632883]

10. Czarnecki, K.; Foster, J.N.; Hu, Z.; Lämmel, R.; Schürr, A.; Terwilliger, J.F. Bidirectional Transformations: A Cross-Discipline Perspective. Theory and Practice of Model Transformations; Paige, R.F. Springer: Berlin/Heidelberg, Germany, 2009; pp. 260-283.

11. Foster, J.N.; Greenwald, M.B.; Moore, J.T.; Pierce, B.C.; Schmitt, A. Combinators for Bi-Directional Tree Transformations: A Linguistic Approach to the View Update Problem. Proceedings of the 32nd ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL ’05; Long Beach, CA, USA, 12–14 January 2005; pp. 233-246.

12. Fischer, S.; Hu, Z.; Pacheco, H. The essence of bidirectional programming. Sci. China Inf. Sci.; 2015; 58, pp. 1-21. [DOI: https://dx.doi.org/10.1007/s11432-015-5316-8]

13. Hofmann, M.; Pierce, B.; Wagner, D. Symmetric Lenses. Proceedings of the 38th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL ’11; Austin, TX, USA, 26–28 January 2011; pp. 371-384.

14. Ko, H.S.; Hu, Z. An Axiomatic Basis for Bidirectional Programming. Proc. ACM Program. Lang.; 2017; 2, pp. 1-29. [DOI: https://dx.doi.org/10.1145/3158129]

15. Barbosa, D.M.; Cretin, J.; Foster, N.; Greenberg, M.; Pierce, B.C. Matching Lenses: Alignment and View Update. Proceedings of the 15th ACM SIGPLAN International Conference on Functional Programming, ICFP ’10; Baltimore, MD, USA, 27–29 September 2010; pp. 193-204.

16. Hu, Z.; Mu, S.C.; Takeichi, M. A programmable editor for developing structured documents based on bidirectional transformations. High.-Order Symb. Comput.; 2008; 21, pp. 89-118. [DOI: https://dx.doi.org/10.1007/s10990-008-9025-5]

17. Zhu, Z.; Ko, H.S.; Zhang, Y.; Martins, P.; Saraiva, J.; Hu, Z. Unifying Parsing and Reflective Printing for Fully Disambiguated Grammars. New Gener. Comput.; 2020; 38, pp. 423-476. [DOI: https://dx.doi.org/10.1007/s00354-019-00082-y]

18. Hidaka, S.; Hu, Z.; Inaba, K.; Kato, H.; Matsuda, K.; Nakano, K. Bidirectionalizing Graph Transformations. Proceedings of the 15th ACM SIGPLAN International Conference on Functional Programming, ICFP ’10; Baltimore, MD, USA, 27–29 September 2010; pp. 205-216.

19. Hidaka, S.; Asada, K.; Hu, Z.; Kato, H.; Nakano, K. Structural Recursion for Querying Ordered Graphs. Proceedings of the 18th ACM SIGPLAN International Conference on Functional Programming, ICFP ’13; Boston, MA, USA, 25–27 September 2013; pp. 305-318.

20. Tran, V.D.; Kato, H.; Hu, Z. Programmable View Update Strategies on Relations. Proc. VLDB Endow.; 2020; 13, pp. 726-739. [DOI: https://dx.doi.org/10.14778/3377369.3377380]

21. Matsuda, K.; Wang, M. FliPpr: A System for Deriving Parsers from Pretty-Printers. New Gener. Comput.; 2018; 36, pp. 173-202. [DOI: https://dx.doi.org/10.1007/s00354-018-0033-7]

22. Cheney, J.; Gibbons, J.; McKinna, J.; Stevens, P. On principles of Least Change and Least Surprise for bidirectional transformations. J. Object Technol.; 2017; 16, pp. 3:1-3:31. [DOI: https://dx.doi.org/10.5381/jot.2017.16.1.a3]

23. Zhang, X.; Hu, Z. Towards Bidirectional Live Programming for Incomplete Programs. Proceedings of the 44th International Conference on Software Engineering, ICSE ’22; Pittsburgh, PA, USA, 22–27 May 2022; pp. 2154-2164. [DOI: https://dx.doi.org/10.1145/3510003.3510195]

24. Zhang, X.; Guo, G.; He, X.; Hu, Z. Bidirectional Object-Oriented Programming: Towards Programmatic and Direct Manipulation of Objects. Proc. ACM Program. Lang.; 2023; 7, pp. 230-255. [DOI: https://dx.doi.org/10.1145/3586035]

25. Ko, H.S.; Zan, T.; Hu, Z. BiGUL: A formally verified core language for putback-based bidirectional programming. Proceedings of the 2016 ACM SIGPLAN Workshop on Partial Evaluation and Program Manipulation, PEPM ’16; St. Petersburg, FL, USA, 18–19 January 2016; pp. 61-72. [DOI: https://dx.doi.org/10.1145/2847538.2847544]

26. Hu, Z.; Ko, H.S. Principles and practice of bidirectional programming in BiGUL. Bidirectional Transformations: International Summer School, Oxford, UK, July 25–29, 2016, Tutorial Lectures; Springer: Cham, Switzerland, 2018; pp. 100-150.

27. Jackiw, R.N.; Finzer, W.F. The geometer’s sketchpad: Programming by geometry. Watch What I Do: Programming by Demonstration; MIT Press: Cambridge, MA, USA, 1993; pp. 293-307.

28. Xia, H.; Araujo, B.; Grossman, T.; Wigdor, D. Object-Oriented Drawing. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16; San Jose, CA, USA, 7–12 May 2016; pp. 4610-4621. [DOI: https://dx.doi.org/10.1145/2858036.2858075]

29. Jacobs, J.; Gogia, S.; Mundefinedch, R.; Brandt, J.R. Supporting Expressive Procedural Art Creation through Direct Manipulation. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI ’17; Denver, CO, USA, 6–11 May 2017; pp. 6330-6341. [DOI: https://dx.doi.org/10.1145/3025453.3025927]

30. Schachman, T. Apparatus. Available online: http://aprt.us (accessed on 17 February 2025).

31. Bricault, S. Recursive Drawing. Available online: http://bricault.mit.edu/recursive-drawing (accessed on 17 February 2025).