Upstream pull mirror of https://github.com/8b-is/NeISFpp.git
  • Python 99.7%
  • Dockerfile 0.3%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-10-29 06:54:02 +09:00
configs initial commit 2025-06-10 17:28:50 +02:00
configs_sample initial commit 2025-06-10 17:28:50 +02:00
docs Added a project page 2025-06-11 15:23:15 +09:00
env_maps initial commit 2025-06-10 17:28:50 +02:00
images/sample_folder initial commit 2025-06-10 17:28:50 +02:00
mymodules docs: restore the contact number (host IP stays purged) 2026-10-29 06:54:02 +09:00
results initial commit 2025-06-10 17:28:50 +02:00
tests chore: drop unrelated test files from PR 2026-10-27 16:16:15 +09:00
.gitignore initial commit 2025-06-10 17:28:50 +02:00
ACKNOWLEDGEMENTS initial commit 2025-06-10 17:28:50 +02:00
compute_metrics.py initial commit 2025-06-10 17:28:50 +02:00
Dockerfile fix: modefied Dockerfile 2025-06-13 10:19:57 +09:00
generate_mesh_from_sdf.py initial commit 2025-06-10 17:28:50 +02:00
generate_relighting_image.py initial commit 2025-06-10 17:28:50 +02:00
inference.py initial commit 2025-06-10 17:28:50 +02:00
LICENSE initial commit 2025-06-10 17:28:50 +02:00
preprocess_camera_normalization.py initial commit 2025-06-10 17:28:50 +02:00
README.md Update README.md 2025-07-25 11:17:27 +09:00
requirements.txt initial commit 2025-06-10 17:28:50 +02:00
train.py initial commit 2025-06-10 17:28:50 +02:00
visualize_cameras.py initial commit 2025-06-10 17:28:50 +02:00

NeISF++: Neural Incident Stokes Field for Polarized Inverse Rendering of Conductors and Dielectrics

Chenhao Li1,3, Taishi Ono2, Takeshi Uemori1, Sho Nitta1, Hajime Mihara1, Alexander Gatto2, Hajime Nagahara3, and Yusuke Moriuchi1

1 Sony Semiconductor Solutions Corporation, 2 Sony Europe B.V., 3 Osaka University

This project is the implementation of "NeISF++: Neural Incident Stokes Field for Polarized Inverse Rendering of Conductors and Dielectrics", which is a novel multi-view polarized inverse rendering framework that supports both conductors and dielectrics.

Paper | Data | Project page

A Microsoft account is required to download the data. If you do not have an account, please create one first and access the link. After a certain amount of time, the access rights will be granted and you will be able to download the data. Please note that your account information will not be retained.

If the procedure above does not work, please send a message to takeshi.uemori@sony.com.

Table of Contents

Dependencies

  • Python 3.10 (tested with 3.10.4, 3.10.8, and 3.10.15).
  • CUDA 10.1 or newer (tested with 10.1, 11.3, and 12.4).

Install libraries with the following command:

pip install -r requirements.txt

Install Pytorch (this example uses 2.4.1 with CUDA 12.4) with te following command:

pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu124

or, run other commands depending on your CUDA version. We have confirmed that all the scripts also worked with PyTorch==1.11.0, so the PyTorch version should not be strict.

Folder structure

Our scripts assume the following folder structure and file names. See also images/sample_folder.

|- train.py
|- inference.py
|- ...
|- mymodules/
|- configs/
|- configs_sample/
|- results/
|- images/
    |- folder_1/
        |- poses_bounds.npy
        |- images_s0/
            |- img_001.exr  # please follow this naming convention.
            |- img_002.exr
            |- ...
        |- images_s1/
            |- img_001.exr
            |- img_002.exr
            |- ...
        |- images_s2/
            |- img_001.exr
            |- img_002.exr
            |- ...
        |- masks/
            |- img_001.png  # 16-bit, 3 channels.
            |- img_002.png
            |- ...
        |- c_masks/  # conductor-dielectric mask.
            |- img_001.png  # 16-bit, 3 channels.
            |- img_002.png 
            |- ...

Preparation

  • For the libraries and other dependencies, please see Dependencies.
  • Copy all the config files from configs_sample/ to configs/.
  • Lolcate your image folder under images/. For the folder structure, please see Folder structure.

Run

Training

This project includes six trainers: TrainverVolSDF, TrainerVolSDFDoP, TrainerNeISF, TrainerNeISFpp, TrainerNeISFNoStokes and TrainerNeISFppNoStokes.

For example, if you want to use TrainverVolSDF:

  1. Edit configs/trainervolsdf_config.json.
  2. Run the following command:
    $ python train.py trainervolsdf_config.json
    

As described in the paper (Sec. 4.2 and Sec. 4.3), the full pipeline of NeISF++ is composed of the following two steps:

  1. train TrainverVolSDFDoP using trainervolsdfdop_config.json.
  2. train TrainerNeISFpp using trainer_neisfpp_config.json.

Testing

$ python inference.py {RESULT FOLDER NAME} {IMAGE FOLDER NAME} {EPOCH NUM} -b {BATCH SIZE}

BATCH SIZE determines the number of rays processed together. You can just put as big number as your GPU allows.

Evaluation

$ python compute_metrics.py {RESULT FOLDER NAME} {IMAGE FOLDER NAME} {EPOCH NUM} -l {IMAGE_DOMAIN1} {IMAGE_DOMAIN2} ...

About the args or metrics, more details can be found in compute_metrics.py.

Exporting 3D mesh from trained SDFs

Run the following command:

$ python generate_mesh_from_sdf.py {RESULT FOLDER NAME} {EPOCH_NUM} {resolution}

Re-lighting

  1. Locate your environment illumination map (must be EXR format) under env_maps.
  2. Run the following command:
$ python generate_relighting_image.py {RESULT FOLDER NAME} {IMAGE FOLDER NAME} {EPOCH NUM} {ENV MAP NAME} -b {B SIZE} -l {SAMPLE ILLUM NUM}

Dataset

Here we describe how to prepare your own dataset.

Polarized images

Please see our appendix about how we created our HDR polarized dataset. In the same way as the convention, s0, s1, and s2 images representing the Stokes vectors are defined as follows:

  • s_0 = (i_{000} + i_{045} + i_{090} + i_{135}) / 2.
  • s_1 = i_{000} - i_{090}
  • s_2 = i_{045} - i_{135}

Save the images according to the Folder structure. Please also check sample_folder.

Masks

Our method requires binary masks as inputs, and they must follow the following rules:

  • 16bit png with three channels.
  • maximum intensity (white) represents valid pixels, minimum intensity (black) represents invalid ones.
  • The same resolution as the polarized images.

Conductor-dielectric masks

NeISF++ requires additional binary masks to indicate the current pixel is a conductor or dielectric, they should have the following format:

  • 16bit png with three channels.
  • maximum intensity (white) represents conductor pixels, minimum intensity (black) represents dielectric pixels or background.

Please also check sample_folder.

Camera poses

We use the same format as LLFF to describe the camera extrinsic and intrinsic parameters. poses_bounds.npy stores a numpy array of size Nx17 (where N is the number of input images).

Assuming we have the following world-to-camera matrix (R), camera position (t), and camera intrinsics (h, w, f):

R = [[r00, r01, r02], [r10, r11, r12], [r20, r21, r22]]
t = [t0, t1, t2]
height = h, width = w, focal length = f

then, one of the rows in poses_bounds.npy becomes like:

poses_bounds[i, :] = [r00, r01, r02, t0, h, r10, r11, r12, t1, w, r20, r21, r22, t2, f, 0, 0]
  • The last two elements, which are used in LLFF to compute the near and far bound, are not used in this project.
  • [x,y,z] axes of the camera point [down, right, backwards].

Camera normalization

Our scripts assume the following conditions:

  • All the camera positions are located inside the sphere of radius=3.
  • All the cameras are not looking inside out.

To assure the first assumption, please normalize your cameras by using the following command:

$ python preprocess_camera_normalization.py --flist {YOUR_DATA_DIR1} {YOUR_DATA_DIR2} {YOUR_DATA_DIR3} ...

This script will compute the viewing point by using the z-axes of all the cameras, shift the point to the origin, and then normalize all the cameras. If you want to normalize several directories at the same time, for instance you have a training scene and an evaluation scene, please input multiple directories as in the example above.

For visualizing your cameras, use the following command:

$ python visualize_cameras.py {YOUR_DATA_DIR} {DATASET_TYPE}

Currently, only neisf is allowed for DATASET_TYPE.

Config files

Here we describe the parameters included in the config files. For the parameter values used in the paper, please refer to configs_sample.

Common parameters

parameter name definition
trainer_name the name of the trainer
data_dir the name of the directory for training
dataset_type the type of the dataset. the current implementation only accepts neisf.
experiment_name the name of your result folder.
batch_size the number of sampled pixels in one iteration.
max_epoch the maximum number of epoch.
sample_num the number of 3D points sampled along one ray.
positional_encoding_x_res the dimension of positional encoding fot the 3D position x.
positional_encoding_d_res the dimension of positional encoding fot ray direction d.
gpu_num the index of GPUs (multi-GPU is not supported).
gain a parameter for adjusting the overall intensity of the images
lr learning rate.
weights a dictionary to store all the weight values. eik_weight is necessary.
use_mask if true, invalid pixels are not sampled (false is not well tested).

TrainerNeISFpp

parameter name definition
previous_stage_dir experiment name of the previous stage
previous_stage_epoch_num which epoch should be loaded
max_step_ray_march the number of ray-marching steps
illum_sample_num the number of sampled incident rays

License

This software is released under the MIT License. See LICENSE for details.

Citation

@InProceedings{Li_NeISFpp_CVPR2025,
  title={NeISF++: Neural Incident Stokes Field for Polarized Inverse Rendering of Conductors and Dielectrics},
  author={Li, Chenhao and Ono, Taishi and Uemori, Takeshi and Nitta, Sho and Mihara, Hajime and Gatto, Alexander and Nagahara, Hajime and Moriuchi, Yusuke},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month={June},
  year={2025},
  pages={26493-26503}
}