Extension

We will walk through the process of extending PhyFu to support the fuzzing of a new physical scenario. We will use the example of an Atlas scene. The Atlas scene is a human-like robot standing on the ground. The robot is able to perform complex actions like doing Yoga. The robot is controlled by dozens of controllers on its joints. The Atlas scene is a good example of a complex physical scenario.

First, make sure you are at the top directory of the phyfu repository (the directory with setup.py inside). Then download the necessary skeleton files for the Atlas body and ground. For simplicity, we directly clone the original repository of nimblephysics, which contains the skeleton files for the Atlas scene. The skeleton files are located at nimblephysics/data/sdf/atlas. To clone the repository, run the following command:

git clone https://github.com/keenon/nimblephysics.git

Next, create a new file atlas.py inside the directory phyfu/nimble_mutate. Import the necessary modules:

import os

import nimblephysics as nimble
import numpy as np
import torch

from phyfu.array_utils.array_interface import ArrayUtils
from phyfu.nimble_mutate.model_loader import NimbleModel
from phyfu.utils.path_utils import ModulePath

Next, create a new class Atlas that inherits from NimbleModel:

class Atlas(NimbleModel):
    def __init__(self, module_path_utils: ModulePath, au: ArrayUtils,
                override_options=None):
        super().__init__(module_path_utils, au, override_options)

In the __init__ function, we need to load the skeletons that we just downloaded, and define necessary properties, such as the gravity and simulation time step, for the simulation:

class Atlas(NimbleModel):
    def __init__(self, module_path_utils: ModulePath, au: ArrayUtils,
                override_options=None):
        super().__init__(module_path_utils, au, override_options)

        # Load Skeleton and define necessary properties
        self.dt = self._cfg.model_def.dt
        self.substeps = self._cfg.model_def.substeps

        world = nimble.simulation.World()
        world.setGravity([0, -9.81, 0])

        atlas: nimble.dynamics.Skeleton = world.loadSkeleton("atlas_v3_no_head.urdf")
        atlas.setPosition(0, -0.5 * 3.14159)
        ground: nimble.dynamics.Skeleton = world.loadSkeleton("ground.urdf")
        floorBody: nimble.dynamics.BodyNode = ground.getBodyNode(0)
        floorBody.getShapeNode(0).getVisualAspect().setCastShadows(False)

        forceLimits = np.ones([atlas.getNumDofs()]) * 500
        forceLimits[0:6] = 0
        atlas.setControlForceUpperLimits(forceLimits)
        atlas.setControlForceLowerLimits(forceLimits * -1)

        self._default_state = torch.from_numpy(np.copy(world.getState()))
        self.world = world

Note that variable self._cfg actually stores customizable parameters for the simulation and is loaded from a dedicated yaml file for easier management of all the parameters. We will see how to customize these parameters later. But first, let’s move on to implement other necessary functions. We need to implement functions related to action so that the robot can move:

@property
def act_std(self):
    # Variation level of the control
    return torch.tensor([self._cfg.act_std for _ in range(27)])

@property
def dof_shape(self) -> tuple:
    return 27,

@staticmethod
def expand_act(action_list):
    return torch.concatenate([torch.zeros(len(action_list), 6), action_list], dim=1)

In the code above, we define that the robot has 27 degrees of freedom, and the first 6 degrees of freedom are fixed. We also define the variation level of the control (act_std) and the function expand_act to expand the action list to the correct shape.

Besides, we need to implement some methods for the seed scheduling algorithm to work. The seed scheduling algorithm requires a numpy vector as input, so we need to implement a function to convert the state of the robot to a numpy vector:

def state_embedding(self, state):
    return state.flatten().numpy()

@staticmethod
def state_relevant_part(state):
    return state.flatten().numpy()

We also need some additional functions:

@property
def name(self):
    return 'atlas'

def is_valid_state(self, state):
    return True

We set the is_valid_state to always return True because we do not have any constraints on the state of the robot (the is_valid_state only function as a safeguard if the PSE outputs an invalid state (true positive in this case) even if the simulation is started from a valid meta state.).

Now, we come to the parameter configuration file, which will be loaded and used to populate the self._cfg variable:

model_def:
    name: atlas
    dt: 1e-3
    substeps: 1
disable_logging: False
num_steps: 100
test_times: 100
lr: 5e-3
loss_func: linear
opt: Adam
seed_getter:
    type: art
    art_params:
        init_pop_size: 10
        cand_size: 10
        refresh_prob: 0.1
    min_steps: 50
    max_steps: 300
    reset_freq: 20
act_std: 1
max_non_zero_act_steps: 20
mut_dev: 0.05
mut_steps: 10
loss_utils:
    max_epochs: 500
    threshold_sigma: 3
    max_len: 100
    display_freq: 10
    converge_threshold: 1.0e-3
use_gui: False

The content above should be stored in the yaml file of phyfu/configs/fuzzing/nimble/atlas/mutate.yaml. Notably, the model_def.dt and model_def.substeps define the simulation time step and the number of substeps per simulation step, respectively. The seed_getter defines the parameters related to our Simulate-Then-Collect (STC) scheme. The min_steps and max_steps define the minimum and maximum number of simulation steps between the previous collected seed state and the next collected one. The reset_freq defines the frequency of resetting the STC process. The max_non_zero_act_steps defines the maximum number of simulation steps that the robot can perform without any action. The mut_dev and mut_steps define the standard deviation and the number of steps for the mutation algorithm. The loss_utils defines the parameters for the loss function. The use_gui defines whether to use the GUI for visualization (actually this option is not used in the current version of the code).

Also, we need to add the file analysis.yaml into the same directory as the mutate.yaml file. This file defines the parameters for oracle checking. The content of the file is as follows:

min_loss_threshold: 1e-1
diff_tolerance: 1e-3
sigma: 3
write_to_file: True

The file contains the parameters for the oracle checking algorithm. The min_loss_threshold defines the minimum loss value; if the loss value after optimization is greater than the threshold, then we deem it as a backward error. sigma defines maximum threshold for the forward oracle. Denote difference between the seed and mutant’s initial state before the optimization as d0, and the difference between the seed and mutant’s initial state after the optimization as d1. If d1 > d0 * sigma, then we deem it as a forward error. In reality, sigma is usually set to 3 (the intuition comes from three-sigma rule). Also, to filter numerical noise, we set the elements in d1 that are less than diff_tolerance to 0. The write_to_file defines whether to write the bug information to a file.

The full code for the atlas.py file is provided in the phyfu/nimble_mutate folder; the yaml configuration files are in phyfu/configs/fuzzing/nimble/atlas/mutate.yaml and phyfu/configs/fuzzing/nimble/atlas/analysis.yaml.

Before we can run the simulation of Atlas, we need to register it so that we can use it from the command line. To do so, we first need to append an entry in the factory dict of the phyfu.nimble_mutate.registry.NimbleRegistry class:

factory = {
    "two_balls": {
        "fuzz": model_loader.TwoBalls,
        "find_errors": bug_oracle.NimbleOracle
    },
    "catapult": {
        "fuzz": model_loader.Catapult,
        "find_errors": bug_oracle.NimbleOracle
    },
    "atlas": {
        "fuzz": Atlas,
        "find_errors": bug_oracle.NimbleOracle
    }
}

We also need to add an option to the command line parser in phyfu/utils/cli_utils.py:

MODEL_NAME_CHOICES = ["two_balls", "ur5e", "catapult", "snake", "mpm", "atlas"]

Now, we can run the simulation with the following command:

phyfu.fuzz nimble atlas --test_times 10

After waiting for around 2 minutes, the simulation will finish and output the following information:

#loss_too_large: 5
#deviated_init_state: 1

The full code after all the extension steps are provided a separate branch in the repository. The branch can be checked out with the following command:

git checkout atlas

And you can directly run the fuzzing campaign with the same command as above.