Enhancing Rendering Flexibility in Optical Flow Research with Kubric

The Flow Dataset project enhances the rendering capabilities within optical flow research by addressing existing limitations in camera and object control. Leveraging the Kubric framework, this project improves how dynamic, multi-object scenes are generated for optical flow algorithms, making significant advancements in simulating complex, realistic environments.

Key Objectives

The goal was to overcome the challenges posed by limited camera paths, object motion, and environmental effects that hinder sophisticated optical flow simulations. To achieve this, we integrated:

• Advanced Camera Control: Customizable camera paths that support smooth Bezier curve-based transitions. This introduces dynamic camera movement to capture intricate scene details.
• Procedural Rain and Lens Effects: Utilizing Blender’s Python API, custom shaders simulate realistic raindrops and lens distortions like fisheye effects. These features are crucial for understanding domain shifts in optical flow models under various environmental conditions.

Tools and Technologies

The project extensively uses Docker to containerize the development environment, making it reproducible across different systems. Additionally, the following tools were critical to the project’s success:

• Blender + PyBullet: For handling 3D scene creation and physics simulation.
• PyTorch and TensorFlow: These frameworks were employed to evaluate the optical flow models, training neural networks under various conditions generated by Kubric.
• Kubric and HDRI: For generating synthetic multi-object video datasets, complete with depth maps, segmentation masks, and optical flow annotations.

Major Contributions to Kubric

As part of this project, significant contributions were made to the open-source Kubric framework:

1. Custom Motion and Environmental Effects: Extended the Kubric’s scene generator to handle randomized object motions (sliding, rotating) and procedural weather effects.
2. FBX Model Integration: Enabled the import of animated characters from FBX files, simulating non-rigid body motions, essential for complex optical flow testing.
3. Advanced Rendering Pipeline: Implemented a modular pipeline that integrates camera control, weather simulation, and lens distortions into a cohesive framework.

Here's an example of code that generates procedural rain effects in Blender:

def generate_raindrops(num_drops, area_size):
    drops = []
    for _ in range(num_drops):
        x = np.random.uniform(-area_size, area_size)
        y = np.random.uniform(-area_size, area_size)
        size = np.random.uniform(0.01, 0.15)  # adjust raindrop size
        drops.append((x, y, size))
    return drops

This improvement allowed for more realistic weather conditions, which in turn enabled neural networks to perform better under adverse conditions.

Evaluation and Results

The enhancements introduced in the Flow Dataset were tested using PyTorch models, yielding comprehensive insights into the robustness of optical flow algorithms under different environmental and object motion conditions. The addition of lens effects like fisheye distortion and the procedural generation of weather elements provided significant improvements in terms of dataset variability and realism.

Summary of Key Results:

• Increased camera control and motion flexibility
• Robust optical flow simulations with diverse environmental effects
• Procedural generation of weather conditions (rain) improving model resilience

By contributing to the Kubric project and enhancing its dataset generation capabilities, this work pushes the boundaries of what synthetic data can achieve in the domain of optical flow research.

Conclusion

The Flow Dataset’s improvements in rendering flexibility, camera control, and environmental effects demonstrate a significant step forward in optical flow research. By integrating these features into the Kubric framework, we have provided researchers with the tools they need to develop and test their models under more realistic and challenging conditions.

The source code for this project is not available because of copyright.