From Junk to Genius: Robotic Arms and AI Crafting Creative Designs from Scraps

Abstract

As sustainable architecture is increasingly emphasizing material reuse, this study proposes a novel, interactive workflow that integrates robotic arms and artificial intelligence to transform waste materials from architectural models into creative design components. Unlike existing recycling efforts, which focus on the construction phase, this research uniquely targeted discarded architectural model materials, particularly polystyrene foam, that are often overlooked, despite their environmental impact. The workflow combined computer vision and machine learning, utilizing the YOLOv5 model, which achieved a classification accuracy exceeding 83% for the polygon, rectangle, and circle categories, demonstrating a superior recognition performance. Robotic sorting demonstrated the ability to process up to six foam blocks per minute under controlled conditions. By integrating Stable Diffusion, we further generated speculative architectural renderings, enhancing creativity and design exploration. Participant testing revealed that human interaction reduced stacking errors by 57% and significantly improved user satisfaction. Moreover, human–robot collaboration not only corrected robotic errors, but also fostered innovative and collaborative solutions, demonstrating the system’s potential as a versatile tool for education and industry while promoting sustainability in design. Thus, this workflow offers a scalable approach to creative material reuse, promoting sustainable practices from the model-making stage of architectural design. While these initial results are promising, further research is needed to adapt this technique for larger-scale construction materials, addressing real-world constraints and broadening its applicability.

1. Introduction

In architectural design, physical models have long been essential tools for conveying design intentions and refining spatial concepts. Handmade models enable architects to engage with a tangible representation of architectural space, offering a physicality that digital models cannot fully replicate. While digital processes are faster and more precise, they remain visual simulations that lack the tactile qualities of real-world phenomena, which can hinder architects’ ability to fully perceive depth, scale, and spatial relationships. This limitation can restrict the holistic understanding needed for high-quality architectural design [1,2].

However, the creation of these physical models often generates substantial material waste, particularly during iterative design phases, when multiple versions are created. This waste contributes to the already significant environmental footprint of the construction industry, which is responsible for roughly 37% of global carbon emissions [3]. Polystyrene foam, widely used in architectural modeling due to its affordability and ease of manipulation, is one of the primary contributors to this waste. Its accumulation in landfills further exacerbates environmental degradation [4]. Recycling polystyrene and similar materials from architectural models is, therefore, critical, not only to mitigate environmental harm, but also to set a standard for sustainable practices within the profession.

Recent studies have highlighted AI-assisted sustainable design and construction, especially in reducing material waste during construction. Haeusler et al. (2021) utilized data-driven approaches for waste identification during the design phase, optimizing material usage by predicting waste patterns [5]. Baerlecken et al. (2012) showcased “Junk: Reuse of Waste Materials”, a series of studio projects that used waste as an aesthetic and structural resource in design and construction [6]. Similarly, Wang (2016) explored the use of unconventional materials in freeform structures, advocating for sustainable practices in construction [7]. Other works have focused on computational techniques for material reuse. For example, Marshall et al. (2020) presented methods for arranging demolition debris, while Gramazio Kohler Research showcased robotic systems for assembling irregular materials [8,9]. Additionally, Cornell’s Timber De-Standardized initiative provided an interactive mixed-reality environment that empowers users to design with non-standard elements [10,11].

Further advancements in human–robot collaboration have been documented. Lai et al. (2024) proposed a meta-model-based framework that seamlessly integrates robotic arms with traditional woodworking techniques [12]. Similarly, Shen (2024) presented a mixed-reality (MR) system, known as ROCOS, which facilitates human–robot collaboration in iterative design and construction processes [13]. Cousin et al. (2023) and Ali et al. (2020) focused on handling irregular materials and robotic assembly, emphasizing adaptability in complex construction workflows [14,15]. Gharbia et al. (2020) systematically reviewed on-site robotic applications, emphasizing the need for adaptable systems for complex construction tasks [16].

Further, Ma et al. (2022) investigated robotic substitution potential, highlighting human–machine collaboration’s role in complex tasks [17]. Wang et al. (2020) developed a vision-based robotic system for sorting and recycling construction waste, applying simultaneous localization and mapping (SLAM) and instance segmentation to improve sorting accuracy [18]. These studies underscore the potential of robotics and AI in material reuse, but predominantly address the construction phase rather than the design process.

While these studies offer valuable insights into material recycling and the role of AI in construction, they predominantly treat waste as a supplementary element, utilized during or after the design process. This approach limits the creative potential of waste materials, relegating them to secondary roles rather than recognizing their value as integral components of design ideation. A summary of related work is shown in Table 1.

To address this gap, our study introduces a workflow that places waste materials at the core of the design process. Rather than relegating waste to later stages, our method incorporates discarded model materials into the early phases of architectural design. Specifically, our workflow involves recycling, shredding, and classifying model waste, enabling collaborative design between robotic arms and human designers through machine learning. This approach repurposes waste to enhance spatial understanding and inspire creative ideas, restoring discarded materials to their initial role as tools for architectural exploration.

By embedding human–computer interaction within the design process, our study lowers barriers to sustainable design, making material reuse more accessible. In contrast to traditional methods that focus on waste during the construction phase, our approach integrates recycling with design ideation, supporting a circular economy model in architectural modeling. Ultimately, this study aims to redefine sustainable architectural practices by positioning waste as a dynamic element in innovative, eco-conscious design from the outset.

2. Materials and Methods

This section details a structured workflow for transforming discarded foam model scraps into new architectural design elements. The process is divided into the following five primary stages: (1) the recycling and preparation of foam blocks, (2) capturing and processing images for model training, (3) developing rules for robotic arm manipulation, (4) testing and refining human–robot interaction, and (5) enhancing visual outputs with Stable Diffusion. The proposed workflow for transforming discarded foam model scraps into new architectural design elements is shown in Figure 1. A complete workflow runs through three phases of data/model generation, human–robot collaboration, and architectural construction.

2.1. Recycling and Classification of Polystyrene Foam Blocks

Polystyrene foam is widely used in initial model drafts due to its affordability, light weight, and ease of manipulation. For this study, six discarded polystyrene foam models were collected with permission from students and manually disassembled. The sizes of the foam pieces ranged from 4 cm² to 20 cm², chosen for their compatibility with the robotic arm’s (shown in Figure 2) 3 cm² suction cup. A total of 42 foam pieces were used. The robotic arm’s suction cup could lift objects weighing up to 1 kg, with an operational radius of 290 mm. While weight-bearing limits were not explicitly tested during this study, these parameters provide a baseline for understanding the arm’s handling capabilities.

The foam fragments were then categorized into six shape types—irregular, polygon, quadrilateral, rectangle, triangle, and circle—based on geometric characteristics like edges, curvature, and angles. These categories were designed to encompass the majority of possible shapes while remaining simple enough for accurate machine recognition. The choice of a manageable shape taxonomy enhanced the classification reliability by limiting the complexity, ensuring that each shape could be clearly defined and distinguished by the model. Additionally, this categorization process will contribute to systematic handling, allowing for the more precise recycling of materials into architectural components.

To ensure result reliability, a manual verification process was conducted on all categorized shapes before proceeding with further experimentation. The size standardization and shape-based taxonomy were periodically re-evaluated to maintain consistency during the study, thus providing a solid foundation for subsequent automated processing and classification.

2.2. Image Data Processing and Neural Network Model Conversion

Using a built-in camera on the robotic arm, we captured over 1200 top-down images of foam fragments with a resolution of 640 × 480 pixels. Each shape category included a minimum of 200 labeled images to ensure balanced training data across classes. This number of images was selected to balance computational efficiency and model performance, as it provided sufficient data for accurate training without overwhelming processing capabilities.

The YOLOv5 model, pre-installed on the robotic arm’s Jetson Nano B01 controller, was selected for its compatibility, speed, accuracy, and low computational requirements, making it ideal for our experimental setup. The robotic arm functions as an intelligent computing device, enabling real-time data processing, shape recognition, and autonomous decision making. Training was conducted with a batch size of 4 over 10 epochs, and the trained YOLOv5 weights were converted into ONNX and TensorRT formats. This conversion facilitated faster inference times, critical for real-time applications, without relying on external computing resources.

To ensure the reliability of shape recognition, a validation process was implemented, tracking model performance metrics such as the precision, recall, and mAP (mean Average Precision) at each training epoch. Regular checks against validation metrics ensurd the model’s accuracy and generalizability across various shapes, thereby reinforcing the robustness of the recognition system before deployment. The entire workflow, from image capture to inference, was handled on the robotic arm, ensuring real-time shape recognition and immediate decision making during stacking tasks. This integration avoided the latency issues commonly associated with external processing platforms.

2.3. Rule Design for Recognition and Motion of the Robotic Arm

Our approach to rule design for the robotic arm began with identifying each shape’s center point based on color, followed by recognizing the full shape of each foam block. This two-step process simplified the 2D-to-3D transformation, creating a structured placement logic that aided in accurate stacking and sorting. This rule-based system essentially gave the robotic arm a set of consistent decision-making guidelines, enhancing stability and reproducibility in the stacking process, as shown in Figure 3.

Each shape was assigned a unique color—irregular shapes were red, polygons were orange, quadrilaterals were yellow, rectangles were cyan, triangles were purple, and circles were blue. This color-coded system allowed the robotic arm to quickly distinguish shapes, streamlining the process of sorting and stacking. Guided by these color cues, the robotic arm identified each shape’s center and perimeter, calculated the area, and then sorted the shapes by size to establish a stable stacking order. Larger shapes were placed as foundational elements, supporting smaller blocks above to maximize the arrangement’s structural integrity.

To verify the reliability of this rule design, we ran multiple trials, observed placement accuracy, and made adjustments as necessary. This iterative, rule-based approach ensured that the robotic arm achieved precise and stable configurations, improving the consistency and overall control of the experiment.

2.4. Robotic Arm Testing and Interaction

The experimental setup, including the environment and the robotic arm’s motion, is depicted in Figure 4. To address the challenges of color recognition under varying lighting conditions, the robotic arm was positioned within a studio measuring 80 × 80 × 80 cm, equipped with a controlled light source to maintain consistent illumination. One side of the studio was left open to facilitate human–robot interaction. A 15 cm cushion block was used to elevate the robotic arm, because the 3 cm high foam blocks necessitated a higher position for effective stacking. Since the arm itself was only 25 cm tall, this elevation was crucial for the arm to reach and stack the blocks effectively.

The robotic arm employs inverse kinematics for movement, with joint angles computed based on the position of the end effector. To simplify three-dimensional movement, the rotational joint of the lower pan-tilt head was removed, enabling kinematic analysis in a two-dimensional plane. The arm was modeled as a specialized 3-link mechanism, which reduced the inverse kinematics problem to solve for the angles ∠α and ∠β. Here, ∠β represents the rotation of the ID2 servo, while ∠α is derived from parallelogram geometry to determine the ID3 servo angle ∠c. The joint angle θ is computed as θ = arctan(x/y), with the quadrant determined by the signs of x and y.

During the experiment, the robotic arm first moved to a designated location for color recognition using OpenCV, then retureds to its initial position to perform shape recognition based on pre-trained YOLOv5 results. Participants placed various shapes randomly at the initial position, and the robotic arm attempted to transport them. The pseudocode and code flowchart are presented in Figure 5. The pseudocode outlines the following core functions:

Object Transportation: Controls the robotic arm’s movements, including height adjustments, suction activation, object transportation, and returning to the home position.

Color Center Detection: Processes images in the LAB color space to identify object contours and calculate their center positions for classification.

Shape Recognition: Utilizes the YOLOv5 model to classify foam block shapes into predefined categories (e.g., rectangle, circle, and polygon).

Image Processing: Runs in a continuous loop, processing queued images for recognition and transportation tasks while updating object counts.

Main Program and ROS Integration: Initializes the Robot Operating System (ROS), subscribes to the camera feed, and continuously triggers the image processing function.

The flowchart complements the pseudocode by visually depicting the operational logic, beginning with system initialization and ending with object transportation. Decision points, such as shape classification, guide subsequent actions, ensuring the accurate sorting and stacking of foam blocks. In the event of robotic arm failure, participants can intervene or experiment with different block placements, fostering human–robot interaction and improving the recognition process.

2.5. Enhancing Foam Block Model Representation with Stable Diffusion

To visually enrich the foam block designs, Stable Diffusion—a generative AI tool known for its quality in producing detailed images—was used to render enhanced architectural models. This tool was selected for its ability to retain essential model features while allowing for creative expansion of the design. Using specific prompt words related to architectural style and form, Stable Diffusion was guided to generate imaginative yet cohesive renderings based on the block shapes.

For generating architectural designs, we employed the ArchitectureRealMix_v1.ckpt model, which was freely downloaded from www.liblib.art (accessed on 10 August 2024) [19]. Key parameters such as redrawing amplitude (0.44–0.55) and CFG Scale (17–28) were adjusted to maintain fidelity to the original model while introducing creative variations. These settings allowed the generated images to incorporate stylistic elements like futurism and expressionism, adding depth and artistic value without altering the structural integrity of the model shapes.

To confirm the reliability of these generated renderings, the images were evaluated for consistency across multiple render attempts. This consistency was verified by comparing the output images and checking their alignment with the original model shapes, ensuring that the tool’s adjustments did not distort the design’s intended structure. Additionally, each generated image underwent a qualitative review to confirm its architectural coherence, providing a basis for using Stable Diffusion as a reliable tool in sustainable design visualization.

Figure 6 provides an overview of the complete system workflow, illustrating the collaboration between human designers, robotic operations, and AI-driven processes for recycling foam blocks into architectural elements. Details can be found in Appendix A.

3. Results

3.1. Effectiveness of YOLOv5 Model

Dataset: We collected a dataset of 1200 images, each with dimensions of 640 × 480 pixels, representing the six following categories: irregular shapes, polygons, quadrilaterals, rectangles, triangles, and circles, with 200 images per category. The dataset was split into training and validation sets using an 80:20 ratio. The distribution of images across these categories is shown in Figure 7a. We annotated each image with features, including the coordinates of the center point (x, y) and the entity’s width and height. Further analysis of these features is presented in Figure 7b. The variables x, y, width, and height follow a Gaussian distribution and exhibit nonlinear correlations with one another.

Training: We trained the YOLOv5 model for a total of 10 epochs. The location loss, objectness loss, and classification loss values for both the training and validation sets during the training process are shown in Figure 8. The overall training loss converged after 10 epochs.

Evaluation: We used Precision–Recall (PR) curves to evaluate the Intersection over Union (IoU) threshold, as shown in Figure 9a. Based on the overall class PR curve, we selected an IoU threshold of 0.5, at which precision equals recall. Overall, the polygon, rectangle, and circle categories exhibited better recognition performance, with PR curve areas of 0.850, 0.830, and 0.995, respectively. This is further supported by the model achieving a mean Average Precision (IoU = 0.5) of 64.6%, with a classification accuracy exceeding 83% for simpler shapes such as circles and polygons. We also verified this performance using confusion matrices, as shown in Figure 9b. In the matrix, the x-axis represents the predicted labels, the y-axis represents the actual label, and the numerical value indicate the proportion of samples predicted as class x. For example, the value of 0.14 in the upper-left corner represents the proportion of samples that were actually irregular and predicted as irregular, out of all the samples predicted as irregular. The diagonal line represents the proportion of correctly predicted samples, further confirming that the polygon, rectangle, and circle categories achieved a better recognition performance, with values of 0.71, 0.55, and 0.95, respectively.

3.2. Assessment of Robotic Arm and Human–Robot Collaboration

We assessed the robotic arm’s effectiveness in handling and stacking foam blocks through several key metrics, as follows: autonomous successful completions (labeled as “succeed”), human-assisted completions (“intervene”), human adjustments after the robotic arm’s errors (“remedy”), and human-only completions (“create”). As shown in Figure 10, the robotic arm performed adequately in simpler stacking scenarios, achieving a moderate success rate autonomously. However, more complex placements often required human intervention to complete the task accurately. Based on the task outcomes shown in Figure 10, we analyzed the effectiveness of human–robot collaboration across five participants using the following formula:

$\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\hspace{0.17em}\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{e}={\displaystyle \frac{Remedy}{Remedy+Intervene+Create}}\times 100\%$

The results are as follows:

Participant 1: ${\displaystyle \frac{4}{5}}\times 100\%=80.0\%$

Participant 2: ${\displaystyle \frac{5}{7}}\times 100\%=71.4\%$

Participant 3: ${\displaystyle \frac{6}{9}}\times 100\%=66.7\%$

Participant 4: ${\displaystyle \frac{2}{7}}\times 100\%=28.6\%$

Participant 5: ${\displaystyle \frac{2}{5}}\times 100\%=40.0\%$

The overall average correction scale was 57%, demonstrating the significant role of human intervention in reducing errors and improving task precision. Additionally, the robotic arm required approximately 10 s to complete a full cycle of operations, including recognizing the foam block, picking it up using the suction cup, moving it to the designated location, placing it, and returning to the home position. Under optimal conditions, this cycle time enabled the robotic arm to handle six objects per minute, providing a reliable baseline for efficiency in repetitive stacking tasks. Given the system’s robust performance with foam blocks, it is plausible that, with minor modifications, the workflow could be adapted to handle larger or heavier materials. For example, adjusting the suction cup pressure and enhancing the arm’s load-bearing capacity could enable the processing of materials such as medium-density timber or larger foam components. These potential improvements would allow the system to scale effectively for more diverse applications.

Our observations suggest that, while the robotic arm exhibited reliable recognition in the initial stages, repeated recognition attempts were often needed to maintain consistency in execution. In this study, the term ‘Human–Robotic Interaction’ describes the collaborative process between humans and the robotic arm. Unlike cobots, which are designed for seamless real-time collaboration, the robotic arm here operated under predefined rules and required human intervention for error correction and task optimization. This reflects a semi-autonomous system rather than a fully collaborative robotic environment.

To improve the system reliability, we implemented a rule requiring multiple confirmation attempts before a final decision on classification, which notably improved reliability and reduced misplacement errors. The participants rated their experience on a satisfaction scale from 1 to 10, with higher scores associated with tasks completed autonomously by the robotic arm and lower scores when significant human assistance was required. This feedback indicates that refining the robotic arm’s decision making and reducing reliance on human input could further enhance user experience and satisfaction.

In addition to observing operational metrics, we analyzed qualitative feedback to better understand areas where human–robot collaboration could be refined. The participants generally preferred tasks where the robotic arm demonstrated self-sufficiency, especially in cases requiring precision in shape alignment and consistent stacking. These insights underscore the importance of ongoing refinement to the robotic arm’s rule-based algorithm and recognition software, potentially allowing for the more advanced and autonomous handling of complex shapes in future iterations.

3.3. Effectiveness of Stable Diffusion in Generating Architectural Renderings

To assess the impact of Stable Diffusion in generating compelling architectural renderings from foam block shapes, we employed a series of architectural style prompts and adjusted key parameters to balance creativity with structural fidelity. Figure 11 shows a selection of generated images that incorporate stylistic elements of futurism and expressionism while maintaining the foam models’ fundamental shapes. These stylistic choices and structural consistencies provide the basis for evaluating Stable Diffusion’s effectiveness in creating imaginative yet coherent architectural visualizations.

Through experimentation, we found that setting the redrawing amplitude between 0.44 and 0.55 and the CFG Scale between 17 and 28 allowed for visual exploration without sacrificing the core structure of each shape. These parameter choices preserved the overall arrangement and provided creative variations that aligned with the study’s goals, showcasing the visual potential of recycled materials in architectural design. Notably, this configuration maintained stylistic and structural consistency across multiple generations, confirming Stable Diffusion as an effective tool for producing visually engaging and contextually relevant designs.

These results underscore Stable Diffusion’s capability to introduce a range of architectural styles and visual elements while respecting the initial design’s structure. This capability offers potential applications for exploring sustainable design options, allowing architects and designers to test a variety of aesthetic and functional adaptations based on core shapes derived from recycled materials. By enabling such flexibility in visual representation, Stable Diffusion contributes to a more adaptable approach to architectural design, fostering a blend of creativity and sustainability.

4. Discussion

This study presents a workflow that integrates AI-driven shape recognition, robotic manipulation, and human–computer interaction to creatively repurpose discarded materials for architectural design. Through the application of YOLOv5 for shape recognition, robotic arms for material handling, and Stable Diffusion for enhanced visual rendering, we explore a novel approach to sustainable architectural design, making use of waste materials in an innovative and functional way.

4.1. Key Findings and Contributions

The study demonstrates that AI and robotic systems can significantly contribute to sustainable design by enabling the reuse of materials in the early design phase. Our results with YOLOv5 showed a strong shape recognition performance in real-time, although certain shape categories, such as irregular polygons, were more challenging to classify. The decision to use YOLOv5 was informed by its compatibility with the robotic arm’s computational platform and its ability to balance accuracy and computational efficiency. While YOLOv5 performs well for real-time detection tasks, future studies will consider benchmarking its performance against newer YOLO versions or alternative models to validate its suitability for more complex scenarios. The color-coded rules allowed the robotic arm to perform tasks autonomously in most cases, showcasing the potential for an AI-robotic workflow to support efficient, semi-autonomous design processes.

Additionally, using Stable Diffusion to visualize foam blocks in complex architectural forms underscores the potential for generative AI to enhance design aesthetics while aligning with sustainability goals. These visuals illustrate how AI can augment human creativity by transforming simple, discarded shapes into imaginative architectural forms. This approach provides a valuable tool for exploring sustainable design aesthetics and generating creative possibilities with recycled materials.

To evaluate the performance of our proposed technique, a comparative analysis was conducted against the following two representative approaches: Technique A and Technique B. These methods were chosen for their relevance to robotics and material reuse in architectural and construction workflows.

Technique A: Vision-Based Sorting System (Wang et al., 2020 [18]). This method utilizes SLAM and instance segmentation algorithms (e.g., Mask R-CNN) for identifying and sorting construction and demolition waste (CDW). It demonstrates a robust performance in accurately classifying materials, even in complex environments. However, it operates autonomously, with limited human collaboration, and lacks adaptability for iterative design processes.

Technique B: MR-Based Human–Robot Collaboration (Shen, 2024 [13]). This approach integrates mixed reality (MR) for iterative design and robot-assisted assembly, enabling dynamic collaboration between humans and robots. While it is effective in fostering design creativity and resolving conflicts in real time, its scalability is limited by reliance on small-scale, controlled environments and pre-fabricated components.

Table 2 outlines the strengths and limitations of these techniques in comparison to our proposed workflow.

As shown in Table 2, the proposed technique bridges the gap between automation and collaboration, achieving a balance between efficiency and adaptability. Unlike Technique A, which prioritizes autonomous sorting during construction, our approach emphasizes early-stage material reuse and human–robot collaboration, significantly reducing stacking errors by 57%. Compared to Technique B, our workflow is better optimized for iterative design processes, although its scalability to larger materials remains an area for improvement. These insights underscore the potential of our method for integration into both educational and professional architectural practices.

4.2. Limitations and Areas for Future Improvement

While promising, this workflow has several limitations. First, the current reliance on color cues for shape recognition, while effective in controlled environments, limits the system’s adaptability to more varied and complex materials, such as wood, metal, or concrete. To improve its adaptability, future iterations could incorporate shape-to-shape recognition algorithms independent of color cues, leveraging multi-dimensional scanning to identify materials with diverse textures and patterns. Future iterations of this system could also aim to enhance the interaction level between humans and the robotic arm. By integrating real-time feedback mechanisms and adaptive learning algorithms, the system could evolve into a more collaborative framework resembling cobots. This would enable dynamic task sharing and reduce the need for human intervention, thereby improving efficiency and user experience in complex design tasks.

Another limitation lies in the classification framework itself. The shapes used in this study are basic geometries that do not always align with typical architectural elements. By shifting to a classification system based on architectural components—where elongated shapes in plan view could represent beams and points could represent columns—the workflow could enable a more intuitive 2D-to-3D translation process. This refinement would facilitate the workflow’s application in architectural settings, allowing the system to bridge the gap between design sketches and structural elements more effectively.

Scalability is a crucial aspect of this workflow. While the current study focuses on lightweight foam blocks, the modular design of the system indicates its potential adaptability to larger and heavier materials, such as timber or concrete. Handling such materials would require future enhancements, including improved suction technology, reinforced robotic arm components, and advanced algorithms capable of recognizing diverse textures and densities. These upgrades would address the challenges posed by increased weight and material complexity, paving the way for broader applications.

By incorporating multi-modal sensing and advanced machine learning techniques, the workflow could be extended to real-world architectural design tasks, further demonstrating its scalability and practical utility. Additionally, the 57% reduction in material stacking errors achieved through human–robot collaboration will significantly minimize resource wastage, enhancing the system’s overall efficiency. Future research will prioritize optimizing the energy efficiency of robotic components and assessing the workflow’s carbon footprint to ensure its comprehensive sustainability.

4.3. Educational and Practical Implications

This workflow holds significant potential for foundational design education. By integrating the system into early design courses, students could gain hands-on experience in spatial understanding and sustainable practices. Working directly with a robotic arm that translates 2D sketches into 3D forms could enhance spatial reasoning, while interaction with AI-generated renderings could inspire creativity. Additionally, this AI robotic workflow offers an accessible pathway to design for students from diverse backgrounds, reducing traditional entry barriers and fostering inclusive engagement in architectural exploration.

From a practical perspective, the workflow could be applied to small-scale construction projects or experimental installations where non-standard shapes and reclaimed materials are used. For example, an AI-guided robotic arm could transform architects’ sketches into physical structures using recycled materials for temporary pavilions, artistic installations, or custom architectural components. Such applications could streamline construction, reduce material waste, and potentially result in unique aesthetic expressions, bridging environmental responsibility with innovative design.

4.4. Future Directions

Looking forward, future research could focus on refining this workflow to eliminate color dependency and incorporate angle detection, allowing for the more precise handling of irregular or complex shapes. Future research will compare YOLOv5 with newer versions, such as YOLOv8 or YOLOv9, to identify models that balance speed and accuracy for handling complex architectural elements, advancing the workflow’s applicability. Additionally, integrating reinforcement learning could enable the robotic arm to adapt its actions based on real-time feedback, progressively reducing the need for human intervention and enhancing operational autonomy. This evolution would yield a more adaptable system capable of responding to varied tasks with minimal supervision.

Collaborative projects with architects, educators, and engineers could further enhance the workflow’s applicability. In educational settings, tailored modules could be developed to align with curriculum goals, while in practical construction scenarios, the system could be adapted to address specific industry challenges such as irregular construction materials and adaptive reuse. By continuing to refine this workflow, this study lays a foundation for sustainable, creative, and technologically integrated design practices that resonate with both environmental and academic objectives.

In conclusion, this study demonstrates how AI and robotics can drive sustainable design by reducing waste and expanding the creative potential of architectural exploration. By merging technological innovation with sustainable practice, this workflow offers a path forward for more adaptable, resource-efficient approaches in the built environment. With continued refinement, this system has the potential to inspire similar practices across other domains, contributing to a more sustainable future in architectural and construction fields.

5. Conclusions

This study introduces a novel workflow that integrates robotic arms, artificial intelligence, and generative design tools to creatively repurpose discarded materials within the design phase. By embedding material reuse directly into the early stages of architectural design, the workflow represents a significant innovation, bridging sustainability with an enhanced design efficiency. Unlike traditional methods that focus on material recycling during the construction phase, this approach aligns material reuse with creative exploration, reducing the time spent on iterative material cutting and enabling a more seamless design process.

The collaborative process between the robotic arm and human designers not only improves task precision, but also fosters creative problem solving, transforming discarded materials into tools for architectural inspiration. Generative AI further augments this process by providing speculative visualizations that expand the boundaries of conventional design thinking.

While the system is currently tailored for lightweight materials such as foam blocks, future research will address scalability challenges with heavier and more complex materials, including timber and concrete. Advancements in hardware, algorithm optimization, and energy efficiency will be essential for broadening the workflow’s applicability. This innovative system provides a foundation for resource-efficient, imaginative design practices, serving as a versatile tool for both education and industry while contributing to sustainable architectural innovation.

Author Contributions

Conceptualization, J.L. and X.C.; methodology, J.L.; software, S.Y. and J.L.; validation, J.L., X.C. and S.Y.; formal analysis, J.L.; investigation, J.L.; resources, J.L.; data curation, J.L.; writing—original draft preparation, J.L.; writing—review and editing, X.C.; visualization, J.L.; supervision, J.L.; project administration, J.L. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

The data presented in this study are available on request from the corresponding author.

Conflicts of Interest

The authors declare no conflicts of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. Workflow diagram for recycling foam blocks into architectural design.

Figure 2. Pictures of robotic arm and foam blocks.

Figure 3. Corresponding rule diagram of colors and shapes.

Figure 4. Design of human–computer interaction experiment.

Figure 5. Pseudocode and code flow diagrams.

Figure 6. Overall diagram of transforming foam model scraps into architectural designs.

Figure 7. (a) Labels and (b) label correlogram, which displays statistics for x, y, width, and height.

Figure 8. Location loss (upper left for the training set, lower left for the validation set), objectness loss (upper middle for the training set, lower middle for the validation set), and classification loss (upper right for the training set, lower right for the validation set) are displayed.

Figure 9. (a) Precision–recall curves and (b) confusion matrix.

Figure 10. Task distributions, recognition failures, and satisfaction scores for participant testing.

Figure 11. Built and generated architectural renderings.

Table 1

Summary of related work on AI and material recycling in architecture.

Author(s)	Focus Area	Methodology	Key Findings	Limitations
Haeusler et al. (2021)	Waste data analysis for construction	Data-driven waste identification	Identified waste patterns in design phase	No integration of waste into design phase
Baerlecken et al. (2012)	Aesthetic reuse of waste materials	Studio-based design projects	Demonstrated aesthetic use of waste	Limited scalability and application
Wang (2016)	Irregular material use in construction	Design build with unconventional materials	Advocated for unconventional materials	Focused on construction phase only
Shen (2024)	Human–robot collaboration in design	Mixed-reality systems for collaboration	Enabled efficient design iteration	Not applied to waste materials
This Study	Waste integration in architectural design	AI and robotic arm-based reuse workflow	Demonstrates early-stage waste reuse	Focused on foam materials

Table 2

Comparison with other techniques on AI and material recycling in architecture.

Metric	Proposed Technique	Technique A: Vision-Based Sorting System (Wang et al., 2020 [18])	Technique B: Mixed-Reality Human–Robot Collaboration (Shen, 2024 [13])
Accuracy	mAP: 64.6% (circle/polygon > 83%)	High (instance segmentation achieves >80% accuracy)	Moderate (dependent on human correction during collaboration)
Scalability	Foam blocks; potential for timber/concrete	Suitable for construction and demolition waste	Limited to specific MR-compatible design scenarios
Complexity	Medium (rule-based with YOLOv5 integration)	High (utilizes SLAM and segmentation algorithms)	Medium (complex mixed-reality setup but simpler interaction logic)
Decision Making	Semi-autonomous with human oversight	Fully autonomous	Semi-autonomous (relies on human for iterative design input)
Collaboration	Human–robot interaction reduces errors by 57%	Minimal collaboration (focus on automation)	High-level human interaction to refine outputs
Timing	Processes 6 blocks/minute	Variable based on segmentation and task complexity	Lower speed due to human interaction
Material Reuse	Early-stage reuse in design processes	Construction-phase waste reuse	Design-phase focus, no material-specific emphasis
Redundancy	Built-in verification for classification	Minimal redundancy	Moderate redundancy, human refinement of machine outputs
Self-Adaption	Fixed rules; potential for reinforcement learning	Adaptive algorithms (real-time decision making via SLAM)	No adaptability, relies on fixed mixed-reality rules

Appendix A

To provide a comprehensive understanding of the experimental workflow and the robotic arm’s operation, we uploaded a demonstration video. This video showcases key stages of the process, including foam block recognition, shape classification using the YOLOv5 model, robotic arm movements, and human–robot collaboration for stacking and sorting. Readers can access the video at the following link: https://youtu.be/5DTdS-f4tcU?si=3vyBa6G8I4bpFhf0 (accessed on 19 December 2024). This video supplements the descriptions provided in the manuscript and visually demonstrates the system’s functionality.

References

1. Chylinski, D.R. Dwelling in Model-Space: Experiments in Architectural Model Making. Master’s Thesis; Laurentian University of Sudbury: Sudbury, ON, Canada, 2022; Available online: https://zone.biblio.laurentian.ca/handle/10219/3886 (accessed on 19 November 2024).

2. Schilling, A. Architecture and Model Building: Concepts, Methods, Materials; Birkhäuser: Basel, Switzerland, 2018.

3. World Economic Forum. Towards Green Building Value Chains: China and Beyond. Available online: https://www3.weforum.org/docs/WEF_Towards_Green_Building_Value_Chains_2024.pdf (accessed on 25 June 2024).

4. Maafa, I.M. Pyrolysis of polystyrene waste: A review. Polymers; 2021; 13, 225. [DOI: https://dx.doi.org/10.3390/polym13020225] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33440822]

5. Haeusler, M.; Butler, A.; Gardner, N.; Sepasgozar, S.; Pan, S. Wasted… again: Or how to understand waste as a data problem and aiming to address the reduction of waste as a computational challenge. Proceedings of the International Conference on Computer-Aided Architectural Design Research in Asia (CAADRIA); Hong Kong, China, 29 March–1 April 2021; CAADRIA: Hong Kong, China, 2021; pp. 371-380. [DOI: https://dx.doi.org/10.52842/conf.caadria.2021.1.371]

6. Baerlecken, D.; Reitz, J.; Duncan, D. Junk: Reuse of waste materials. Proceedings of the 30th ECAADE Conference; Prague, Czech Republic, 12–14 September 2012; eCAADe: Brussels, Belgium, 2012; pp. 143-150. [DOI: https://dx.doi.org/10.52842/conf.ecaade.2012.2.143]

7. Wang, Y. Hooke park biomass boiler house. Advancing Wood Architecture; Taylor and Francis: London, UK, 2016; pp. 169-182. [DOI: https://dx.doi.org/10.4324/9781315678825-13]

8. Marshall, D.; Meuller, C.; Clifford, B.; Kennedy, S. Computational arrangement of demolition debris. Detritus; 2020; 11, pp. 3-18. [DOI: https://dx.doi.org/10.31025/2611-4135/2020.13967]

9. Johns, R.L.; Wermelinger, M.; Mascaro, R.; Jud, D.; Gramazio, F.; Kohler, M.; Hutter, M. Autonomous dry stone: On-site planning and assembly of stone walls with a robotic excavator. Constr. Robot.; 2020; 4, pp. 127-140. [DOI: https://dx.doi.org/10.1007/s41693-020-00037-6]

10. Lok, L.; Samaniego, A.; Spencer, L. Timber de-standardized. Proceedings of the 41st Annual Conference of the Association of Computer Aided Design in Architecture (ACADIA); New York, NY, USA, 25–30 October 2021; Acadia Publishing Company: Lafayette, LA, USA, 2022; Volume 13, pp. 222-231. [DOI: https://dx.doi.org/10.52842/conf.acadia.2021.222]

11. Lok, L.; Bae, J. Timber de-standardized 2.0: Mixed reality visualizations and user interface for processing irregular timber. Proceedings of the 27th CAADRIA Conference; Sydney, Australia, 5–8 April 2022; CAADRIA: Hong Kong, China, 2022; pp. 121-130. [DOI: https://dx.doi.org/10.52842/conf.caadria.2022.2.121]

12. Lai, Z.; Xiao, Y.; Chen, Z.; Li, H.; Huang, L. Preserving Woodcraft in the Digital Age: A Meta-Model-Based Robotic Approach for Sustainable Timber Construction. Buildings; 2024; 14, 2900. [DOI: https://dx.doi.org/10.3390/buildings14092900]

13. Shen, Y.-T. MR-Based Human–Robot Collaboration for Iterative Design and Construction Method. Buildings; 2024; 14, 2436. [DOI: https://dx.doi.org/10.3390/buildings14082436]

14. Cousin, T.; Marshall, D.; Pearl, N.; Alkhayat, L.; Mueller, C. Integrating irregular inventories: Accessible technologies to design and build with nonstandard materials in architecture. J. Phys. Conf. Ser.; 2023; 2600, 192004. [DOI: https://dx.doi.org/10.1088/1742-6596/2600/19/192004]

15. Ali, A.K.; Lee, O.J.; Song, H. Generic design aided robotically facade pick and place in construction site dataset. Data Brief; 2020; 31, 105933. [DOI: https://dx.doi.org/10.1016/j.dib.2020.105933]

16. Gharbia, M.; Chang-Richards, A.; Lu, Y.; Zhong, R.Y.; Li, H. Robotic technologies for on-site building construction: A systematic review. J. Build. Eng.; 2020; 32, 101584. [DOI: https://dx.doi.org/10.1016/j.jobe.2020.101584]

17. Ma, X.; Mao, C.; Liu, G. Can robots replace human beings?—Assessment on the developmental potential of construction robots. J. Build. Eng.; 2022; 56, 104727. [DOI: https://dx.doi.org/10.1016/j.jobe.2022.104727]

18. Wang, Z.; Li, H.; Yang, X. Vision-based robotic system for on-site construction and demolition waste sorting and recycling. J. Build. Eng.; 2020; 32, 101769. [DOI: https://dx.doi.org/10.1016/j.jobe.2020.101769]

19. LibLib Art. Architecturerealmix_v1.ckpt. Available online: https://www.liblib.art/ (accessed on 10 August 2024).