Essential for many industries ranging from Hollywood computer-generated imagery to product design, 3D modeling tools often use text or image prompts to dictate different aspects of visual appearance, like color and form. As much as this makes sense as a first point of contact, these systems are still limited in their realism due to their neglect of something central to the human experience: touch. Fundamental to the uniqueness of physical objects are their tactile properties, such as roughness, bumpiness, or the feel of materials like wood or stone. Existing modeling methods often require advanced computer-aided design expertise and rarely support tactile feedback that can be crucial for how we perceive and interact with the physical world. With that in mind, researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have created a new system for stylizing 3D models using image prompts, effectively replicating both visual appearance and tactile properties. The CSAIL team’s “TactStyle” tool allows creators to stylize 3D models based on images while also incorporating the expected tactile properties of the textures. TactStyle separates visual and geometric stylization, enabling the replication of both visual and tactile properties from a single image input. PhD student Faraz Faruqi, lead author of a new paper on the project, says that TactStyle could have far-reaching applications, extending from home decor and personal accessories to tactile learning tools. TactStyle enables users to download a base design — such as a headphone stand from Thingiverse — and customize it with the styles and textures they desire. In education, learners can explore diverse textures from around the world without leaving the classroom, while in product design, rapid prototyping becomes easier as designers quickly print multiple iterations to refine tactile qualities. “You could imagine using this sort of system for common objects, such as phone stands and earbud cases, to enable more complex textures and enhance tactile feedback in a variety of ways,” says Faruqi, who co-wrote the paper alongside MIT Associate Professor Stefanie Mueller, leader of the Human-Computer Interaction (HCI) Engineering Group at CSAIL. “You can create tactile educational tools to demonstrate a range of different concepts in fields such as biology, geometry, and topography.” Traditional methods for replicating textures involve using specialized tactile sensors — such as GelSight, developed at MIT — that physically touch an object to capture its surface microgeometry as a “heightfield.” But this requires having a physical object or its recorded surface for replication. TactStyle allows users to replicate the surface microgeometry by leveraging generative AI to generate a heightfield directly from an image of the texture. On top of that, for platforms like the 3D printing repository Thingiverse, it’s difficult to take individual designs and customize them. Indeed, if a user lacks sufficient technical background, changing a design manually runs the risk of actually “breaking” it so that it can’t be printed anymore. All of these factors spurred Faruqi to wonder about building a tool that enables customization of downloadable models on a high level, but that also preserves functionality. In experiments, TactStyle showed significant improvements over traditional stylization methods by generating accurate correlations between a texture’s visual image and its heightfield. This enables the replication of tactile properties directly from an image. One psychophysical experiment showed that users perceive TactStyle’s generated textures as similar to both the expected tactile properties from visual input and the tactile features of the original texture, leading to a unified tactile and visual experience. TactStyle leverages a preexisting method, called “Style2Fab,” to modify the model’s color channels to match the input image’s visual style. Users first provide an image of the desired texture, and then a fine-tuned variational autoencoder is used to translate the input image into a corresponding heightfield. This heightfield is then applied to modify the model’s geometry to create the tactile properties. The color and geometry stylization modules work in tandem, stylizing both the visual and tactile properties of the 3D model from a single image input. Faruqi says that the core innovation lies in the geometry stylization module, which uses a fine-tuned diffusion model to generate heightfields from texture images — something previous stylization frameworks do not accurately replicate. Looking ahead, Faruqi says the team aims to extend TactStyle to generate novel 3D models using generative AI with embedded textures. This requires exploring exactly the sort of pipeline needed to replicate both the form and function of the 3D models being fabricated. They also plan to investigate “visuo-haptic mismatches” to create novel experiences with materials that defy conventional expectations, like something that appears to be made of marble but feels like it’s made of wood. Faruqi and Mueller co-authored the new paper alongside PhD students Maxine Perroni-Scharf and Yunyi Zhu, visiting undergraduate student Jaskaran Singh Walia, visiting masters student Shuyue Feng, and assistant professor Donald Degraen of the Human Interface Technology (HIT) Lab NZ in New Zealand.

Ahead of Intelligent Health (13-14 September 2023, Basel, Switzerland), we asked Yurii Kryvoborodov, Head of AI & Data Consulting, Unicsoft, his thoughts on the future of AI in healthcare.   Do you think the increased usage of Generative AI and LLMs will have a dramatic impact on the healthcare industry and, if so, how?   Generative AI is just a part of the disruptive impact of all AI tech on the healthcare industry. It allows to dramatically reduce time efforts, costs and chances of mistakes. Generative AI and LLMs are applied to automating clinical documentation, drug discovery, tailoring of treatment plans to individual patients, real-time clinical decision support and health monitoring, extracting valuable insights from unstructured clinical records, streamlining administrative tasks like billing and claims processing, providing instant access to comprehensive medical knowledge. And this list continues.

We sat with Benjamin von Deschwanden, Co-Founder and CPO at Acodis AG, to ask him his thoughts on the future of AI in healthcare.   Do you think the increased usage of Generative AI and LLMs will have a dramatic impact on the healthcare industry and, if so, how?   I think that the strength of Generative AI lies in making huge amounts of information accessible without needing to manually sift through the source material. Being able to quickly answer any questions is going to be transformative for everyone working with increasingly bigger data sets.The challenge will be to ensure that the information we get by means of Generative AI is correct and complete – especially in healthcare – as the consequences of wrong data can be fatal. We at Acodis are actively working on practical applications of Generative AI inside our Intelligent Document Processing (IDP) Platform for Life Science and Pharma clients to drive efficiency and accelerate time to market, whilst controlling the risks.

Intelligent Health 2024 returns to Basel, Switzerland on 11th–12th September. We’ve got prominent speakers. An extensive programme. Groundbreaking advancements in #HealthTech. And much, much more. Our incredible 2024 programme will dive deeper than ever before. From sharing the latest innovation insights to exploring use cases of AI application in clinical settings from around the world. All through our industry-renowned talks, limitless networking opportunities, and much-loved, hands-on workshops. Read on to discover what themes await at the world’s largest AI and healthcare summit. 

We sat down with Margrietha H. (Greet) Vink, Erasmus MC’s Director of Research Development Office and Smart Health Tech Center, to ask her for her thoughts on the future of AI in healthcare.   Do you think the increased usage of Generative AI and LLMs will have a dramatic impact on the healthcare industry and, if so, how?   The integration of Generative AI and LLMs into the healthcare industry holds the potential to revolutionise various aspects of patient care, from diagnostics and treatment to administrative tasks and drug development. However, this transformation will require careful consideration of ethical, legal, and practical challenges to ensure that the benefits are realised in a responsible and equitable manner.