"use strict";
const {Fragment: _Fragment, jsx: _jsx, jsxs: _jsxs} = arguments[0];
function _createMdxContent(props) {
  const _components = {
    a: "a",
    h2: "h2",
    p: "p",
    span: "span",
    ...props.components
  };
  return _jsxs(_Fragment, {
    children: [_jsxs(_components.h2, {
      id: "the-interface---a-creators-playground",
      children: [_jsx(_components.a, {
        "aria-hidden": "true",
        tabIndex: "-1",
        href: "#the-interface---a-creators-playground",
        children: _jsx(_components.span, {
          className: "icon icon-link"
        })
      }), "The Interface - A Creator's Playground:"]
    }), "\n", _jsx(_components.p, {
      children: "Navigating VideoLeap is akin to a maestro conducting an orchestra. Each swipe and tap brings a crescendo of edits to life, intuitive yet powerful—turning complexity into simplicity."
    }), "\n", _jsxs(_components.h2, {
      id: "the-tools---painting-with-pixels",
      children: [_jsx(_components.a, {
        "aria-hidden": "true",
        tabIndex: "-1",
        href: "#the-tools---painting-with-pixels",
        children: _jsx(_components.span, {
          className: "icon icon-link"
        })
      }), "The Tools - Painting with Pixels:"]
    }), "\n", _jsx(_components.p, {
      children: "Here, AI isn't just a tool; it's a collaborator. Whether it's automagically color-correcting or smartly splicing scenes, VideoLeap's features are a playground for the imagination, enabling creators to push boundaries and tell stories that resonate."
    }), "\n", _jsxs(_components.h2, {
      id: "the-community---united-in-creation",
      children: [_jsx(_components.a, {
        "aria-hidden": "true",
        tabIndex: "-1",
        href: "#the-community---united-in-creation",
        children: _jsx(_components.span, {
          className: "icon icon-link"
        })
      }), "The Community - United in Creation:"]
    }), "\n", _jsx(_components.p, {
      children: "VideoLeap goes beyond software; it's a nexus where creators converge, share, and uplift. It's a canvas where collective wisdom transforms into stunning visual narratives."
    }), "\n", _jsxs(_components.h2, {
      id: "eyes-on-the-horizon",
      children: [_jsx(_components.a, {
        "aria-hidden": "true",
        tabIndex: "-1",
        href: "#eyes-on-the-horizon",
        children: _jsx(_components.span, {
          className: "icon icon-link"
        })
      }), "Eyes on the Horizon:"]
    }), "\n", _jsx(_components.p, {
      children: "This isn't just about editing videos. It's about sculpting the very fabric of digital storytelling. VideoLeap isn't just keeping pace with the future; it's defining it, preparing us for a world where AI and human ingenuity intertwine seamlessly in the dance of creation."
    }), "\n", _jsxs(_components.h2, {
      id: "the-verdict---join-the-vanguard",
      children: [_jsx(_components.a, {
        "aria-hidden": "true",
        tabIndex: "-1",
        href: "#the-verdict---join-the-vanguard",
        children: _jsx(_components.span, {
          className: "icon icon-link"
        })
      }), "The Verdict - Join the Vanguard:"]
    }), "\n", _jsx(_components.p, {
      children: "For those who dream in frames and think in cuts, VideoLeap is your digital studio. It's an open invitation to not just ride the wave of AI evolution but to steer it. Let's create, not just for today, but for the canvas of tomorrow."
    })]
  });
}
function MDXContent(props = {}) {
  const {wrapper: MDXLayout} = props.components || ({});
  return MDXLayout ? _jsx(MDXLayout, {
    ...props,
    children: _jsx(_createMdxContent, {
      ...props
    })
  }) : _createMdxContent(props);
}
return {
  default: MDXContent
};


AI Augmented


## Navigating the Ebb Boom and Flow of AI in 2023

2023 was marked by significant technological advancements, strategic business moves, and a deeper focus on the ethical implications of AI. This year-in-review aims to encapsulate the pivotal developments, month by month, that shaped the AI landscape.

## **Q1:** The Adventurous Start of AI in 2023

* **January 🚀**
    
    * OpenAI introduced DALL·E 2, an AI model for generating images from textual descriptions, showcasing the advancement in generative AI.
        
    * DeepMind's AlphaFold 2 was used to predict the structure of nearly every protein known to science.
        
    * Shutterstock integrated generative AI features, a big leap in digital media.
        
    * Getty Images took legal action against Stability AI for copyright issues.
        
    * 11 Labs Text-to-Speech: Unveiled a beta version of their realistic-sounding voice tool.
        
    * ChatGPT's Record MAUs: Surpassed 100 million monthly active users, a historic achievement.
        
    * Nvidia at CES 2023: Introduced new generative AI technologies for faster virtual world creation in its Omniverse platform
        
* **February 🚀**
    
    * Meta (formerly Facebook) unveiled Project CAIRaoke, an AI-powered karaoke feature, demonstrating the use of advanced audio and video processing in AI tools. They also released Llama-1 under a research license.
        
    * Microsoft announced the integration of OpenAI's language models into its Bing search engine.
        
    * Stability AI released Stable Diffusion, a generative AI model capable of generating photorealistic images.
        
    * Google introduced **Bard**, its own AI chatbot, shaking up the AI world. Also, they announced investment in Anthropic at the tune of $300 million deal, focusing on transparent and trustworthy AI.
        
    * Snapchat's "My AI": Debuted an AI chatbot powered by GPT technology for Snapchat Plus subscribers.
        
    * Spotify's AI DJ: Became a significant part of the listening experience for users.
        
* **March 🚀**
    
    * NVIDIA launched Omniverse Avatars, an AI-powered virtual collaboration system, highlighting the development of realistic virtual humans for interactive applications.
        
    * Amazon Bedrock, Google Vertex AI, and Salesforce Einstein GPT were among the top AI tools and products of 2023
        
    * OpenAI released GPT-4, a more advanced version of its language model with improved capabilities.
        
    * Google announced its own language model, Bard, as a competitor to OpenAI's ChatGPT.
        
    * Adobe and Canva rolled out AI-driven design tools, revolutionizing graphics.
        
    * Open Letter for AI Pause: Prominent scientists and tech leaders called for a halt in AI development.
        

## Q2: Midyear Marvels

* **April 🚀**
    
    * IBM released Watson 2.0, an enhanced AI platform with improved natural language understanding and advanced analytics capabilities, contributing to the evolution of AI tools.
        
    * OpenAI introduced DALL-E 2, a generative AI model capable of generating complex images from text prompts.
        
    * Microsoft announced the integration of GPT-4 into its Azure cloud computing platform.
        
    * Midjourney v5 amazed the world with its photorealistic AI images.
        
    * Snap extended "My AI" chatbot access to all Snapchat users.
        
    * Alibaba unveiled its AI chatbot, Tongyi Qianwen, a significant move in AI chatbots.
        
    * * Segment Anything: A tool for editing images and videos using AI.
            
    * AI-Generated Song Hits 20M Views: "Heart on My Sleeve" by Ghostwriter.
        
    * EU Regulatory Framework for AI: The European Commission proposed the first regulatory framework.
        
* **May 🚀**
    
    * DeepMind, a subsidiary of Alphabet Inc., revealed AlphaCode, an AI system proficient in programming tasks, signifying a significant milestone in AI research.
        
    * Businesses began to leverage AI tools like Synthesys and HitPaw for video and image enhancement capabilities
        
    * Nvidia announced the Isaac Sim, a platform for training and testing AI models for robotics applications.
        
    * OpenAI introduced Whisper, a powerful speech recognition model that can transcribe speech in multiple languages with high accuracy.
        
    * Databricks Funding: Raised $500 million in Series E.
        
    * Snapchat's AR Lens with AI: Introduced a new form of AR Lens incorporating generative AI.
        
    * Anthropic's 100k Token Context: Announced for AI models.
        
* **June 🚀**
    
    * Microsoft expanded Azure AI services, offering new capabilities for computer vision, speech recognition, and language understanding, empowering businesses with advanced AI tools.
        
    * Google announced the integration of its Bard language model into Google Search, providing users with more conversational search results. Google Sheets Launched an AI-powered feature for data analysis.
        
    * OpenAI released DALL-E 2 for commercial use, enabling businesses to generate custom images for marketing and other purposes.
        
    * AI-Generated QR Code Art: Became a popular form of AI-generated art.
        
    * LinkedIn Deepfake Detection: Began using AI to detect and remove deepfake content.
        

## Q3: AI in Full Flight

* **July 🚀**
    
    * Adobe unveiled Sensei 2.0, an updated AI platform integrating advanced machine learning and computer vision technologies, enhancing creative and marketing solutions.
        
    * Stability AI introduced DreamBooth, a tool that allows users to train AI models on custom datasets to generate personalized images. Stable Diffusion XL 1.0 released, enhancing AI-generated image quality.
        
    * Microsoft announced the integration of GPT-4 into its Office suite, providing users with AI-powered writing and analysis tools.
        
    * Meta released Llama 2, an open-source language model.
        
    * Elon Musk's xAI Funding: Announced plans to raise $1 billion.
        
* **August 🚀**
    
    * Tesla introduced Autopilot 4.0, featuring advanced neural network architectures for autonomous driving, representing a significant leap in AI-powered transportation systems.
        
    * OpenAI released a new version of ChatGPT with improved conversational abilities and the ability to remember past conversations.
        
    * Google announced the development of a new language model, LaMDA 2, with improved conversational abilities and a greater ability to understand context. Google's Duet AI for Workspace: Announced a $30 per month charge.
        
    * IBM and Salesforce partnered for enhanced AI tech integration.
        
    * Zoom AI Controversy: Faced accusations of misusing user content.
        
    * Adobe Firefly Global Launch: Made available worldwide.
        
* **September 🚀**
    
    * Huawei unveiled MindSpire 2.0, an upgraded AI computing framework, offering enhanced support for diverse AI workloads and reinforcing AI infrastructure.
        
    * OpenAI released a new version of DALL-E 2 with improved image generation capabilities and the ability to create animated images.
        
    * Google announced the development of a new natural language processing model, BERT 2, with improved language understanding capabilities.
        
    * Microsoft announced AI "copilots" and AI-powered Surface laptops.
        
    * Getty Images launched its AI image platform, a major step in digital media.
        
    * Anthropic Series C Funding: Raised $450 million.
        
    * Stack AV Funding: Raised $1 billion for self-driving truck technology.
        
    * Hollywood Writers Strike Resolution: Included terms on AI use.
        

## Q4: Year-End Antics and Surprises

* **October 🚀**
    
    * Samsung announced the development of NeonAI, an artificial human project aimed at creating lifelike digital avatars with advanced conversational abilities, advancing human-AI interaction.
        
    * Stability AI introduced a new generative AI model, Stable Diffusion 2, with improved image generation capabilities and the ability to create 3D images.
        
    * Microsoft announced the integration of GPT-4 into its Xbox gaming platform, providing gamers with AI-powered chatbots and personalization features.
        
    * Qualcomm showcased new AI capabilities at the Snapdragon Summit.
        
    * Universal Music Group vs. Anthropic AI: A lawsuit over copyright infringement.
        
    * LLaVA AI Model release by a Microsoft-affiliated research team
        
* **November 🚀**
    
    * OpenAI's board drama with Sam Altman's brief exit and return. They also released a new version of Whisper with improved speech recognition capabilities and the ability to transcribe multiple speakers simultaneously.
        
    * Salesforce introduced Einstein Vision 2.0, an enhanced computer vision platform with improved object detection and image recognition capabilities, empowering businesses with advanced visual AI tools.
        
    * Google announced the development of a new multimodal AI model, Multitask-2, capable of generating text, images, and other media.
        
    * Amazon introduced a new AI chatbot, Q, for enterprise solutions.
        
    * **DOD AI Adoption Strategy:** Released by the U.S. Department of Defense. Builds upon and supersedes the 2018 AI Strategy and the 2020 Data Strategy,
        
    * **Google DeepMind Material Prediction**: Used AI to predict the structure of new materials.
        
* **December 🚀**
    
    * Apple launched CoreML 4, a major update to its machine learning framework, offering improved performance and new features for developers building AI-powered applications on Apple's platforms.
        
    * Stability AI introduced a new AI model, DreamFusion, capable of generating 3D models from text prompts.
        
    * Microsoft announced the integration of GPT-4 into its Teams collaboration platform, providing users with AI-powered meeting summaries and other productivity features.
        
    * Google revealed its AI model Gemini, a significant AI milestone.
        
    * EU AI Act Agreement: Reached a provisional agreement on AI legislation.
        
    * Hugging Face and AMD Partnership: Announced a collaboration to accelerate AI models.
        

[![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704946066237/35abf924-bcc0-470f-af9f-542ec6ba11ed.png align="center")](https://journal.everypixel.com/2023-the-year-of-ai)

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704943903043/65384bdc-0ca2-4311-8fb3-3c5eb31a695e.jpeg align="center")

# Funding

In 2023, the AI industry saw significant investments from various sources, including capital, venture capital (VC), and private equity firms. Some of the most notable investments in AI during that year were: (Based in US)

### Notable Investments

![Image](https://pbs.twimg.com/media/GDj3smCbgAAYQ6k?format=png&name=small align="center")

| Company | Amount | Lead investors | Valuation | Industry | Date announced |
| --- | --- | --- | --- | --- | --- |
| OpenAI | $10,000,000,000 | Microsoft | n/a | Artificial intelligence | 1/23/23 |
| Anthropic | $4,000,000,000 | Amazon |  | Artificial intelligence | 9/25/23 |
| Anthropic | $2,000,000,000 | Google | n/a | Artificial intelligence | 10/27/23 |
| Inflection AI | $1,300,000,000 | Microsoft, Reid Hoffman, Bill Gates, Eric Schmidt, Nvidia | $4,000,000,000 | Artificial intelligence | 6/29/23 |
| SandboxAQ | $500,000,000 | n/a | n/a | Artifical intelligence | 2/14/23 |
| Anthropic | $450,000,000 | Spark Capital | $4,100,000,000 | Artificial intelligence | 5/23/23 |
| Adept AI | $350,000,000 | General Catalyst, Spark Capital | $1,000,000,000 | Artificial intelligence | 3/14/23 |
| Anthropic | $300,000,000 | Google |  | Artificial intelligence | 02/03/2023 |
| OpenAI | $300,000,000 | n/a | $2,800,000,000 | Artificial intelligence | 4/28/23 |
| Hugging Face | $235,000,000 | Salesforce Ventures | $4,500,000,000 | Artificial intelligence | 8/23/23 |
| Imbue | $200,000,000 | n/a | $1,000,000,000 | Artificial intelligence | 09/07/2023 |
| [Character.AI](http://Character.AI) | $150,000,000 | Andreessen Horowitz | $1,000,000,000 | Artificial intelligence | 3/21/23 |
| AlphaSense | $150,000,000 | Bond Capital | $2,500,000,000 | Artificial intelligence | 9/28/23 |
| Runway | $141,000,000 | n/a | $1,500,000,000 | Artificial intelligence | 6/29/23 |
| [Pony.ai](http://Pony.ai) | $139,000,000 | GAC Toyota Motor | n/a | Artificial intelligence | 08/07/2023 |
| [X.AI](http://X.AI) | $135,000,000 | n/a | n/a | Artificial intelligence | 12/05/2023 |
| D-Matrix | $110,000,000 | Temasek | n/a | Artificial intelligence | 09/06/2023 |
| Insider | $105,000,000 | Qatar Investment Authority, Esas Private Equity. | $1,900,000,000 | Artificial intelligence | 5/24/23 |
| Together | $102,500,000 | Kleiner Perkins | n/a | Artificial intelligence | 11/30/23 |
| Humane | $100,000,000 | Kindred Ventures | n/a | Artifical intelligence | 03/08/2023 |
| Celestial AI | $100,000,000 | IAG Capital Partners, Koch Disruptive Technologies, Temasek’s Xora Innovation | n/a | Artificial intelligence | 6/28/23 |
| Typeface | $100,000,000 | Salesforce Ventures | n/a | Artificial intelligence | 6/29/23 |
| Anthropic | $100,000,000 | SK Telecom | n/a | Artificial intelligence | 8/13/23 |
| Modular | $100,000,000 | General Catalyst | $600,000,000 | Artificial intelligence | 8/24/23 |
| Writer | $100,000,000 | Iconiq Growth | $500,000,000 | Artificial intelligence | 9/18/23 |
| Pryon | $100,000,000 | US Innovative Technology Fund | $625,000,000 | Artificial intelligence | 9/19/23 |
| Prins AI | $100,000,000 | AAB VC | n/a | Artificial intelligence | 10/03/2023 |

These investments highlight the growing interest and confidence in the AI industry in 2023, with major players like Microsoft, Google, and other VC firms pouring billions of dollars into promising AI startups.

### Top AI Startup Funding Recap

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704939126159/56d8823a-7b46-4e9b-b7c2-3c78edb6f010.png align="center")

### References:

[https://twitter.com/MoshehP/status/1737981628859740456](https://twitter.com/MoshehP/status/1737981628859740456)

[https://news.crunchbase.com/biggest-us-vc-startup-funding-deals-2023/](https://news.crunchbase.com/biggest-us-vc-startup-funding-deals-2023/)

[2023 Recap of AI developments | Digital Watch Observatory](https://dig.watch/updates/2023-recap-of-ai-developments)

# Join the Conversation: Your Insights into AI's Future

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1704943962665/6cc82523-566a-44f8-8bf0-5307057d3298.jpeg align="center")

As we reflect on a year of unparalleled innovation and challenge in the AI domain, we invite you, the professionals, enthusiasts, and observers, to join this ongoing conversation. What were the most impactful developments for you in 2023? How do you see these advancements influencing the future of AI in 2024 and beyond? Share your insights, experiences, and predictions with us. Your perspectives are not just valued; they are essential for navigating and shaping the continually evolving AI landscape. Join us in this dialogue to ponder over the past and envisage the future of artificial intelligence.


2023 AI Boom Highlights


3D imaging and machine learning algorithms are quickly improving their ability to capture the unique characteristics of physical objects. The digital twin industry stands at the cusp of a significant transformation with a novel method to extract a highly detailed mesh from any 3D Gaussian Splatting scene within minutes on a single GPU.

Introducing SuGaR (Surface-Aligned Gaussian Splatting), which promises to impact 3D graphics and rendering by offering precise and fast mesh extraction from 3D Gaussian splatting.

SugaR's extracted meshes can be imported into traditional 3D programs for further composition and animation, allowing for editing, sculpting, rigging, animating, or relighting. This integration into conventional 3D workflows underscores the flexibility and practical utility of the SuGaR pipeline.

%[https://youtu.be/QkAGvAHfctI] 

## Understanding SuGaR

Gaussian Splatting has recently become very popular as it yields realistic rendering while being significantly faster to train than NeRFs. It is however challenging to extract a mesh from the millions of tiny 3D Gaussians as these Gaussians tend to be unorganized after optimization. Leveraging Gaussian Splatting, SuGaR allows for the creation of highly detailed, editable meshes, a feat that eludes many traditional methods.

### Technical Innovations

The genius of SuGaR lies in its approach to mesh extraction. An approach is introduced to derive a volume density from new Gaussians that can be assumed to be flat and well distributed over the scene surface. Then, with a newly introduced method to very efficiently sample points on the visible part of a level set of a density function, a Poisson reconstruction algorithm is run to ultimately result in the creation of detailed (triangle) meshes.

The SuGaR pipeline addresses the complexities and limitations of previous methods like Neural Radiance Fields (NeRFs) and Marching Cubes algorithm.

The key contributions of the SuGaR project include:

**Aligning the Gaussians with the Surface**: Introducing a regularization term that encourages the alignment of 3D Gaussians with the surface of the scene. It encourages the Gaussians to be well distributed over the scene surface, ensuring a more accurate representation of the scene geometry.

![Extracting a mesh from Gaussians. Without regularization, the Gaussians have no special arrangement after optimization, which makes extracting a mesh very difficult. Without our regularization term, Marching Cubes fail to extract an acceptable mesh. With our regularization term, Marching Cubes recover an extremely noisy mesh even with a very fine 3D grid. Our scalable extraction method obtains a mesh even without our regularization term. Still, the mesh is noisy. By contrast, our full method succeeds in reconstructing an accurate mesh very efficiently.](https://cdn.hashnode.com/res/hashnode/image/upload/v1703636836201/bb47850d-baee-4484-99f8-197ee88d3e3e.png align="center")

**Efficient Mesh Extraction**: Utilizing the alignment of 3D Gaussians to sample points on the real surface of the scene and extracting a mesh using Poisson reconstruction, a method that is fast, scalable, and detail-preserving.

![Examples of (a) renderings and (b) reconstructed meshes with SuGaR. The (c) normal maps help visualize the geometry](https://cdn.hashnode.com/res/hashnode/image/upload/v1703636991059/2715c49b-391d-4482-995f-45971c8509d1.png align="center")

**Refinement Strategy**: An optional refinement strategy that binds Gaussians to the surface of the mesh and jointly optimizes these Gaussians and the mesh through Gaussian splatting rendering. New Gaussians are bound to the mesh triangles and optimized jointly with the mesh using the Gaussian Splatting rasterizer. Configs for time to spend on refinement can be "short" (2k iterations), "medium" (7k iterations) or "long" (15k iterations).

![Joint refinement of mesh and Gaussians. Left: We bind Gaussians to the triangles of the mesh. Depending on the number of triangles in the scene, we bind a different number of Gaussians per triangle, with predefined barycentric coordinates. Right: Mesh before and after joint refinement.](https://cdn.hashnode.com/res/hashnode/image/upload/v1703634170575/74fe99d4-83c7-4e37-956a-83d0d9542809.png align="center")

**Improved Efficiency and Quality**: The method enables quick retrieval of an editable mesh for realistic rendering, with significant improvements in rendering quality and speed compared to state-of-the-art methods on neural SDFs​​.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703637204877/0b646205-3f1d-49ad-9ffa-8ef93df8a988.gif align="center")

![](https://github.com/Anttwo/SuGaR/raw/main/media/examples/walk.gif align="center")

This pipeline enables high-quality rendering of the mesh using Gaussian splatting rendering rather than traditional textured mesh rendering. This allows for easy editing, sculpting, rigging, animating, or relighting of the Gaussians using traditional software like Blender, Unity, Unreal Engine, etc. by manipulating the mesh instead of the Gaussians themselves.

### Requirements

Before starting the main pipeline steps of the SuGaR project, a vanilla 3D Gaussian Splatting model optimized for 7,000 iterations must be provided. This pre-optimized model serves as the foundation for the subsequent steps in the SuGaR pipeline.

You can obtain a pre-optimized model by setting up the Gaussian Splatting environment on a suitable machine and running the provided training script on an appropriate dataset (like COLMAP or NeRF Synthetic). This foundational model is crucial for the subsequent SuGaR optimization steps.

### DIY: Tips for using SuGaR on your own data and obtain better reconstructions

The SuGaR repository provides several tips aimed at helping users obtain the best possible results when using SuGaR for 3D mesh reconstruction from their own captured data.

* **Capturing Images or Videos:** Use a smartphone or camera to capture images or a video covering the entire surface of the 3D scene you wish to reconstruct.
    
    * Move around the scene slowly and smoothly to avoid motion blur.
        
    * Maintain a uniform focal length and constant exposure time for consistent reconstruction and easier camera pose estimation with COLMAP.
        
    * Disable auto-focus to ensure constant focal length.
        
* **Estimating Camera Poses with COLMAP**: Install a recent version of COLMAP, ideally CUDA-powered. Put the images in a directory `/input`, then use the `gaussian_splatting/`[`convert.py`](http://convert.py) script from the Gaussian Splatting implementation to compute camera poses from the images using COLMAP.
    
* **Choosing Regularization Method**: Decide between density regularization and SDF regularization based on your scene. Density regularization works well for objects centered in the scene, while SDF regularization provides stronger regularization, especially in background regions.
    

* **Adapting Scale and Bounding Box**: If you're reconstructing a large scene, you may need to adjust the scale and bounding box. By default, the bounding box is computed as the bounding box of all camera centers, but you can provide a custom bounding box to the [`train.py`](http://train.py) script using the `--bboxmin` and `--bboxmax` parameters​​.
    

## Implications for the Digital Twin Industry

The digital twin industry, which thrives on creating virtual replicas of physical entities, stands to gain immensely from SuGaR. The ability to quickly generate detailed and editable 3D models can accelerate the development of digital twins, leading to more accurate simulations and analyses. Retrieving an editable mesh for realistic rendering is done within minutes with SuGaR, compared to hours with the state-of-the-art method on neural SDFs, while providing a better rendering quality in terms of PSNR, SSIM and LPIPS.

![This image illustrates the workflow of the SuGaR pipeline for 3D mesh reconstruction and composition using traditional 3D tools. On the left, there are two sets of input data, each showing a set of images or a video of a scene with different objects—a robotic action figure and a toy bulldozer. In the middle, these inputs are processed by the SuGaR pipeline, resulting in a hybrid representation of each object, composed of a mesh overlaid with 3D Gaussians, which are visual approximations for rendering.  On the right, the images show the results of further processing. The action figure and the bulldozer have been enhanced with traditional color textures for the mesh. These textures give the objects a more lifelike appearance, closer to the original input images. Additionally, the image in the top right corner shows a 3D editing software interface (like Blender), suggesting that the reconstructed objects can be imported into such a program for further composition and animation, allowing for editing, sculpting, rigging, animating, or relighting with traditional software tools. This integration into conventional 3D workflows underscores the flexibility and practical utility of the SuGaR pipeline.](https://cdn.hashnode.com/res/hashnode/image/upload/v1703632190289/11c35040-d11a-4627-8e40-b4b9b0e0c00a.png align="center")

![In this image, the workflow is similar to the previous one, focusing on the SuGaR pipeline's process for mesh reconstruction and subsequent editing in 3D software. The input data on the left shows images or videos of different scenes: an action figure in one and a child's playroom in another. These are processed by SuGaR to create a hybrid representation consisting of a mesh combined with 3D Gaussians.  The second set of images showcases the outcome after further processing, where the objects and the room have traditional color textures applied to the mesh, giving a realistic look to the 3D models. Finally, the image in the top right corner displays a 3D editing software interface (like Blender), indicating that these reconstructed models can be imported into such programs for additional composition and animation, leveraging traditional 3D modeling and animation tools. The workflow exemplifies the integration from raw image data to a fully textured and editable 3D model.](https://cdn.hashnode.com/res/hashnode/image/upload/v1703632122341/16c81833-866a-4e4a-956b-b58d4e929f35.png align="center")

### Evaluation and Analysis

While SuGaR represents a significant advancement, scalability and performance in diverse scenarios are still to be achieved. Its reliance on Gaussian alignment and Poisson reconstruction, though innovative, may present challenges in extremely complex environments.

Gaussian Splatting representations of Real scenes typically end up with one or several millions of 3D Gaussians with different scales and rotations, the majority of them being extremely small in order to reproduce texture and details in the scene.

2,000 iterations are usually enough to obtain high quality rendering since the extracted mesh “textured” with surface Gaussians is already an excellent initialization for optimizing the model. However, further refinement helps the Gaussians to capture texturing details and reconstruct extremely thin geometry that is finer that the resolution of the mesh, such as the spokes of the bicycle.

![Refined SuGaR renderings with different numbers of refinement iterations. ](https://cdn.hashnode.com/res/hashnode/image/upload/v1703633875570/d609af26-2da3-4534-9617-f75f97a161a0.png align="center")

SuGaR extracts a mesh in 30~35 minutes on average on a single GPU. After mesh extraction, the refinement time can take from a few minutes up to an hour. A short refinement time is enough to produce a good-looking hybrid representation in most cases but in complex environments longer refinement may be necessary.

The optimization time may vary (from 20 to 45 minutes) depending on the complexity of the scene and the GPU used. Moreover, the current implementation splits the optimization into 3 scripts that can be run separately (SuGaR optimization, mesh extraction, model refinement) so it reloads the data at each part, which is not optimal and takes several minutes. There are plans in the near future to optimize this.

### Hopeful Outlook

As SuGaR matures, its integration into mainstream 3D modeling and rendering workflows could revolutionize how we interact with digital twins. It opens a door to recreating real-world scenes into detailed virtual environments, which could be pivotal in fields like urban planning, healthcare, and manufacturing.

The latest updates on the SuGaR GitHub page as of December 2023 are:

* **December 20, 2023**: A short notebook was added to demonstrate how to render images with the hybrid representation using the Gaussian Splatting rasterizer.
    
* **December 18, 2023**: The initial release of the code.
    

Additionally, the to-do list includes:

* Adapting the code for compatibility with Windows, as the current code is not compatible due to path-writing conventions.
    
* Adding the capability to use the NeRF synthetic dataset, which differs in format from COLMAP scenes.
    
* Completing and cleaning the code for composition and animation, and incorporating it into the sugar\_scene/sugar\_[compositor.py](http://compositor.py) script.
    
* Creating a tutorial on using scripts in the blender directory and the `sugar_scene/sugar_`[`compositor.py`](http://compositor.py) class for importing composition and animation data into PyTorch and applying it to the SuGaR hybrid representation
    

## Conclusion

SuGaR sweetens the prospects for the digital twin industry, offering a blend of precision, efficiency, and versatility. Its role in the future of AI-driven 3D graphics is not just promising but potentially game-changing. As we embrace these advancements, it's crucial to continue critically evaluating and contributing to the evolution of such technologies.

## References

[https://imagine.enpc.fr/~guedona/sugar/](https://imagine.enpc.fr/~guedona/sugar/)

[https://arxiv.org/abs/2311.12775](https://arxiv.org/abs/2311.12775)

[https://github.com/Anttwo/SuGaR?tab=readme-ov-file](https://github.com/Anttwo/SuGaR?tab=readme-ov-file)

[https://github.com/graphdeco-inria/gaussian-splatting](https://github.com/graphdeco-inria/gaussian-splatting)


SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction


DEGA is a system that creates **real-time photorealistic full-body avatars** using 3D Gaussian splatting. It's designed to handle dynamic scenarios, including challenging cases like sliding garments over the body.

Consider a realm where digital personas are sculpted not by polygons or voxels, but by the probabilistic grace of Gaussian distributions. This is the domain we step into, where avatars reside as three-dimensional ensembles of Gaussian components—a veritable cornucopia of statistical shapes melding to form human likenesses.

%[https://www.youtube.com/watch?v=C4IT1gnkaF0&t=15s] 

Embark on a journey through the intricate process that transforms a plethora of captured poses into a flexible model, akin to preparing a gourmet dish with meticulously gathered ingredients. We witness how traditional tetrahedral cages evolve, embracing a transformation under the influence of linear blend skinning, setting the stage for more nuanced, pose-dependent refinements.

The core of this system is a neural network ensemble, where each network—an artisan in its own right—carves out precise body deformations, ensuring that each pose retains natural form and fluidity. This team of networks weaves together the final form of our avatars, endowing them with not just shape, but also texture, color, and the subtleties that make them almost indistinguishable from reality.

But the endeavor does not halt here. An empirical odyssey unfolds as these Gaussian protagonists are scrutinized alongside their kin—other methodologies that vie for realism and accuracy in the portrayal of virtual humans. Side by side, they reveal their strengths in capturing the essence of motion, each with its own unique blend of computational alchemy and aesthetic charm.

And so, we find in this exploration a harmony of mathematics, computer science, and artistry

### Key Features

* **3D Gaussian Splatting**: Extends surface splatting to animate characters, a technique initially for static scenes.
    
* **Input and Composition**: Utilizes multiview video captures from 200 synchronized cameras. Inputs include character pose, 3D facial keypoints, RGB ground truth, and segmentation masks.
    

### Technical Aspects

1. **Gaussian Deformations**:
    
    * Uses tetrahedral cages for 3D Gaussian deformations.
        
    * Each Gaussian's 3D mean is a combination of tetrahedron vertices and barycentric coordinates.
        
    * Incorporates scaling and rotation into the Gaussian covariance matrix.
        
2. **Gaussian Nets**:
    
    * Employs small MLPs for independently modeling avatar parts (face, upper garment, lower garment, body).
        
    * Predicts corrections to cage node positions, Gaussian positions, rotations, and scales.
        
    * Determines per-Gaussian color and opacity.
        
3. **Volumetric Rendering**:
    
    * Uses forward mapping from canonical space with motion vectors, face embeddings, color features, and view direction.
        
    * The final image is achieved through alpha blending-based volumetric rendering.
        

### Comparisons and Advantages

* **Compared with Other Methods**:
    
    * Shows better garment modeling and motion capture than mesh-based body decoders, **MVP**, and **DVA**.
        
    * Excels in handling view-dependent effects.
        
* **Advantages**:
    
    * Effective in modeling sliding garments and dynamic motions.
        
    * Allows decomposition into independent garments for flexible animation.
        
* **Limitations**:
    
    * Challenges in capturing fine garment details like wrinkles and self-shadows.
        

### Conclusion

DEGA marks a significant advancement in creating photorealistic, controllable avatars, suitable for applications in virtual reality, gaming, and digital media.


DEGA: Real-Time Photorealistic Full-Body Avatars


Looking for tools that not only simplify tasks but also open avenues for innovation? Today, we're excited to share our discovery of Roboflow Supervision, a toolkit poised to revolutionize how we approach computer vision projects.

### What's Roboflow Supervision?

Roboflow Supervision is a versatile toolkit designed for building reusable computer vision applications. It's a boon for both beginners and seasoned professionals, offering a range of functionalities from loading datasets to processing detections.

### Why It's a Game Changer

* **Detections Processing:** Seamlessly integrate with models like YOLO for object detection and employ advanced filtering to refine your results.
    
* **Datasets Handling:** Supervision supports various dataset formats, including YOLO, Pascal VOC, and COCO, facilitating easy dataset manipulation and processing.
    
* **Model Evaluation:** Tools for generating confusion matrices and calculating mean average precision (mAP) enhance the model evaluation process, ensuring your applications are accurate and reliable.
    

### Roboflow Supervision and SAM Integration

Recently, Roboflow released a tutorial video demonstrating the integration of a remarkable model called 'Segment Anything' (SAM) by Meta AI, showcasing its versatility in segmentation tasks.

The addition of SAM, which segments any visible object in images, further enhances its capabilities. This integration offers comprehensive solutions for automated mask generation and complex segmentation tasks.

%[https://www.youtube.com/watch?v=D-D6ZmadzPE] 

#### Key Points from the SAM Tutorial:

* **Introduction to SAM:** Capable of detailed segmentation tasks, SAM can handle both single-point extraction and two-stage segmentation solutions.
    
* **Setting Up:** The tutorial in a Roboflow notebook guides through SAM's installation and setup, ensuring readiness for various tasks.
    
* **Automated Mask Generation:** Utilizing SAM, one can automatically create segmentation masks, annotated directly on images using Supervision.
    
* **Advanced Segmentation Techniques:** The tutorial explores sorting masks by size, handling duplicates, and segmentation using points or bounding boxes.
    
* **Real-Time Applications:** SAM's speed is apt for real-time applications, and it's being integrated into Roboflow's annotation tool for enhanced data annotation capabilities.
    
* **Practical Application Example:** Demonstrating SAM’s versatility, the tutorial shows its application in converting bounding box annotations in MRI images into segmentation masks.
    

### The Big Picture

Tools like Supervision simplify access to complex technologies. They empower developers to create more efficient and innovative computer vision applications without getting bogged down in the intricacies of data handling and model tuning.

Roboflow Supervision stands out as a significant contribution to the AI community, offering an accessible yet powerful tool for computer vision applications. It's a testament to the spirit of exploration and preparedness that defines the AI landscape, making it an exciting find for our readers at AI Boom.

Explore more about Roboflow Supervision and how it can enhance your AI projects by visiting [Roboflow Supervision Documentation](#).


Discovering Roboflow Supervision: Reusable Computer Vision tools


## Welcoming the Wave of Change

We're surfing on the crest of the AI wave, charged with possibilities that could redefine our digital existence. It’s an exhilarating ride, but one that demands we keep a watchful eye on the data privacy landscape evolving beneath our feet.

## Balancing Innovation with Integrity

The need for rich data to train AI is undeniable. As builders and creators, we understand the need for rich data to train AI – it’s the bedrock of personalized experiences. But with lawsuits like the one from LinkedIn challenging OpenAI over data scraping, we're reminded of the delicate dance between innovation and privacy. These challenges are a reminder of our responsibility to push the envelope while also safeguarding personal data—a balancing act of the utmost importance. We're in an era where every step forward must be measured against the yardstick of individual privacy.

## Starting Conversations and Raising Questions

Recent discussions have brought to light concerns over the capabilities of AI models to not just process but potentially disclose user data. When members of the tech community express surprise at the ability to download extensive datasets from the recently released GPTs, it prompts a necessary examination of the protocols and safeguards in place. Such conversations are invaluable, sparking collective scrutiny and accountability.

## Proactive Steps and Mindful Solutions

OpenAI's move to introduce Copyright Shield is a step in the right direction. It speaks to the proactive measures needed in our community – recognizing creators' rights while exploring AI's potential. It's a proactive approach that should be mirrored across the tech landscape, ensuring that innovation does not come at the expense of creators' rights.

## The Democratization of AI: A Double-Edged Sword

The drive towards making AI more accessible brings with it the challenge of ensuring that data rights are not compromised. The debate around OpenAI's approach highlights a broader conversation taking place across the tech community. It's a dialogue about how we can democratize AI without diluting the ethical standards that should govern data usage and privacy.

## Crafting the Future Together

The Twitter thread underlines an emerging truth: the journey into the AI future is a collective endeavor. The way we handle these early challenges will set the precedent for future interactions with AI technologies. We must work together—developers, users, and industry watchdogs—to ensure that the path we're carving out is responsible, ethical, and forward-thinking.

## A Vision for a Collaborative Tomorrow

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1699666671543/ca0ed93e-685a-429e-80fb-66640efbefcc.png align="center")

As we chart our course through the uncharted waters of the AI boom, let's proceed with a shared commitment to upholding both the innovative spirit and the privacy rights that will define this era. By embracing vigilance and fostering cooperation, we can ensure that the progress we make respects the principles we hold dear. Together, we can build a future that honors our aspirations for both technology and society, where integrity in innovation is not just an ideal but a reality.

%[https://twitter.com/kanateven/status/1722762002475475426] 

%[https://twitter.com/BuckedUnicorn/status/1723015827249578136]


Data Freedom and Security in the AI Boom


**Hey Creators,**

Luma AI is calling all digital artists, meme creators, and tech enthusiasts to participate in the Make-A-Meme Challenge! This is your chance to transform the flat world of memes into dynamic 3D creations using the innovative Genie text-to-3D technology.

## How This Works:

1. **Join the Family**: Dive into Luma AI's community by hopping onto their [Discord](https://lumalabs.ai/discord). Start playing with Genie, which is like your 3D canvas, and guess what? It’s free.
    
2. **Share Your Vision**: Make a snappy video of your 3D meme creation, share it on social media, tag Luma AI, and slap on the #MadeWithGenie hashtag. Video must be under 45 seconds
    
3. **Get Recognized**: Post your video link in the discord channel [🏆￨make-a-meme-challenge](https://discord.com/channels/991613372439212142/1171876690327240765) by November 20th.
    

## The Loot:

* 🥇 First Place gets AirPods Pro (2nd Gen). Imagine editing your next creation with these bad boys in.
    
* 🥈 Second Place snags a JBL Charge 5. Music and creativity go hand in hand, right?
    
* 🥉 Third Place scores a $100 Amazon Gift Card. More gear, more art!
    

## Tick-Tock, the Clock’s Ticking:

November 20th, 2023, by the stroke of midnight PST. That’s Luma AI's deadline. Make every second count!

## The Blueprint:

Luma AI is searching for the stuff that makes people stop scrolling and start smiling. Get wild with Genie’s Text-to-3D tools. Luma AI's challenge requires your entry to be original, meme-tastic, and inspire a wave of creators. Add your own spin with music, graphics—whatever tells the story of your meme. Keep it Genie-centric and original.

## Rules & Creative Guidelines:

Luma encourages you to push the boundaries of creativity with Genie’s Text-to-3D capabilities. Whether it’s laugh-out-loud funny or thought-provoking, your meme should inspire and incite others to join the creation wave. Feel free to mix in music, graphics, or any artistic flair that you think will make your meme stand out. Just remember, your submission must be a unique creation that includes at least one Genie-generated element and not promote other generative AI platforms.

## The Judges' Table:

You’re in good hands with Luma AI's judges:

* Ian Curtis, the XR wizard from Niantic 8th Wall,
    
* Amit Jain, the chief dreamer and doer at Luma AI,
    
* Barkley Dai, who’s growing Luma AI faster than a viral meme.
    

## Your Move:

Creators, this is more than a contest. It's a chance to shape the meme culture in 3D! Start bending reality with your creativity and who knows, your meme might just be the next big thing. Luma AI is here for it. Are you?

## Join the Conversation:

If you’ve got a meme idea that’s lighting up your mind or you've already drafted your thoughts, let’s talk about it down below. Inspire and get inspired by Luma AI's community!

%[https://youtu.be/JiKGnA85k8c]


Unleash Your Meme Magic in 3D with Luma's Make-A-Meme Challenge


## The Interface - A Creator's Playground:

Navigating VideoLeap is akin to a maestro conducting an orchestra. Each swipe and tap brings a crescendo of edits to life, intuitive yet powerful—turning complexity into simplicity.

## The Tools - Painting with Pixels:

Here, AI isn't just a tool; it's a collaborator. Whether it's automagically color-correcting or smartly splicing scenes, VideoLeap's features are a playground for the imagination, enabling creators to push boundaries and tell stories that resonate.

## The Community - United in Creation:

VideoLeap goes beyond software; it's a nexus where creators converge, share, and uplift. It's a canvas where collective wisdom transforms into stunning visual narratives.

## Eyes on the Horizon:

This isn't just about editing videos. It's about sculpting the very fabric of digital storytelling. VideoLeap isn't just keeping pace with the future; it's defining it, preparing us for a world where AI and human ingenuity intertwine seamlessly in the dance of creation.

## The Verdict - Join the Vanguard:

For those who dream in frames and think in cuts, VideoLeap is your digital studio. It's an open invitation to not just ride the wave of AI evolution but to steer it. Let's create, not just for today, but for the canvas of tomorrow.


VideoLeap: Crafting the Future Frame by Frame

The Interface - A Creator's Playground: Navigating VideoLeap is akin to a maestro conducting an orchestra. Each swipe and tap brings a crescendo of edits to life, intuitive yet powerful—turning complexity into simplicity. The Tools - Painting with Pi...

The Interface - A Creator's Playground:

The Tools - Painting with Pixels:

The Community - United in Creation:

Eyes on the Horizon:

The Verdict - Join the Vanguard:

Read more about
AI Augmented

2023 AI Boom Highlights

SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction

DEGA: Real-Time Photorealistic Full-Body Avatars

Discovering Roboflow Supervision: Reusable Computer Vision tools

Data Freedom and Security in the AI Boom

Unleash Your Meme Magic in 3D with Luma's Make-A-Meme Challenge

VideoLeap: Crafting the Future Frame by Frame

The Interface - A Creator's Playground: Navigating VideoLeap is akin to a maestro conducting an orchestra. Each swipe and tap brings a crescendo of edits to life, intuitive yet powerful—turning complexity into simplicity. The Tools - Painting with Pi...

The Interface - A Creator's Playground:

The Tools - Painting with Pixels:

The Community - United in Creation:

Eyes on the Horizon:

The Verdict - Join the Vanguard:

Read more about AI Augmented

2023 AI Boom Highlights

SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction

DEGA: Real-Time Photorealistic Full-Body Avatars

Discovering Roboflow Supervision: Reusable Computer Vision tools

Data Freedom and Security in the AI Boom

Unleash Your Meme Magic in 3D with Luma's Make-A-Meme Challenge

Read more about
AI Augmented