First, thank you for the great work, really inspiring!

To the point: I'm trying to use EPE on my own data (Carla as source/fake domain, A set of real images as real domain). I created fake_gbuffers, created patches, matched them, and all is working correctly.

For some reason, at iteration a little above 5000, the function clip_gradient_norm throws Error/Warning, and from that point on the reconstructed images are black, and all outputs are 0/nan. I checked, and clip_gradient_norm results in a NAN value, hence the error.

Looking at the tensor itself, it seems that most values(weights) are indeed very close to 0.

My question is what do you think can cause this? a few notes that might be relevant:

1. source domain is RGB, target is grayscale (I don't see why would that be a problem actually)
2. I have (currently, just as a test) 100 images from each domain. In general, I have a total of 100k images from each domain so that won't be a problem...

Thanks.

Does anyone either know an alternative to mseg or how one might get mseg to work on windows 10? I tried installing it but it is giving me an error saying that it can only be installed on linux or mac osx. Thanks!

Hello,

You have done really a great work! I really appreciate your contribution! And, I want to know if there is any promotion in code and data sharing?

Thanks

I am using my data to train the discriminator, I kept the code as it is and my data images are 960x540, I used your code to simulate the Gbuffers and your code to generate the crops and matching. When training, the data is loaded correctly and starts showing values, but soon after (about 1 hour), all values were replaced by nan, can you help me?

2022-05-08 16:27:05,927 17940 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:06,367 17941 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:07,162 17942 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:07,604 17943 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:08,380 17944 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:08,827 17945 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:09,623 17946 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:10,044 17947 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:10,852 17948 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:11,263 17949 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:12,092 17950 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:12,505 17951 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:13,318 17952 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:13,728 17953 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:14,549 17954 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:14,972 17955 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:15,841 17956 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:16,254 17957 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:17,068 17958 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:17,484 17959 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:18,317
2022-05-08 16:27:18,318 17960 rdf0 ds0  rdf1 ds1  rdf2 ds2  rdf3 ds3  rdf4 ds4  rdf5 ds5  rdf6 ds6  rdf7 ds7  rdf8 ds8  rdf9 ds9  rdr0 rdr1 rdr2 rdr3 rdr4 rdr5 rdr6 rdr7 rdr8 rdr9 reg  gs0  gs1  gs2  gs3  gs4  gs5  gs6  gs7  gs8  gs9  vgg
2022-05-08 16:27:18,319 17960 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:18,755 17961 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:19,542 17962 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:19,970 17963 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:20,772 17964 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:21,219 17965 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:22,026 17966 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:22,444 17967 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:23,269 17968 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:23,728 17969 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:24,528 17970 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:24,976 17971 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
^[[B^[[B^[[B^[[B^[[B^[[B2022-05-08 16:27:25,746 17972 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:26,180 17973 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:26,992 17974 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:27,433 17975 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:28,269 17976 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:28,729 17977 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan
2022-05-08 16:27:29,522 17978 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2022-05-08 16:27:29,948 17979 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- nan nan nan nan nan nan nan nan nan nan nan

Hi, Below is the snippet of the training log from training on a similar dataset I put together. It featues a real dataset from Houzz and an artificial dataset of GTAV buildings. Could you explain why I get a get loss update at each iteration for the generator but only occasional updates for the discriminator? I can't tell if this is intentional or if I made a mistake along the way.

I've also noticed that for the .mat files saved in /out store ['i_fake'] and ['i_real] correctly, but ['i_rec_fake] is filled with NaN. Testing the network produces entirely black images.

Sorry, this is probably too wide to format correctly... 2022-04-19 22:53:09,737 346880 rdf0 ds0 rdf1 ds1 rdf2 ds2 rdf3 ds3 rdf4 ds4 rdf5 ds5 rdf6 ds6 rdf7 ds7 rdf8 ds8 rdf9 ds9 rdr0 rdr1 rdr2 rdr3 rdr4 rdr5 rdr6 rdr7 rdr8 rdr9 reg gs0 gs1 gs2 gs3 gs4 gs5 gs6 gs7 gs8 gs9 vgg 2022-04-19 22:53:09,737 346880 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:10,071 346881 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.02 0.99 0.99 1.04 0.99 0.99 1.03 0.98 0.97 1.05 0.72 2022-04-19 22:53:11,022 346882 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:11,034 346883 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.01 1.02 1.00 1.06 1.03 1.01 1.08 0.98 1.00 1.04 0.64 2022-04-19 22:53:11,983 346884 0.00 0.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- 0.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:12,328 346885 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.04 1.03 0.99 1.04 1.02 1.02 1.07 0.97 0.97 1.03 0.24 2022-04-19 22:53:13,278 346886 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:13,625 346887 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.03 1.02 0.99 1.05 1.03 1.01 1.07 0.97 0.98 1.03 0.50 2022-04-19 22:53:14,576 346888 ---- ---- ---- ---- ---- ---- ---- ---- 0.00 0.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.00 ---- ---- ---- ---- ---- 0.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:14,588 346889 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.00 1.01 0.99 1.06 1.04 1.02 1.07 0.98 1.00 1.04 0.70 2022-04-19 22:53:15,540 346890 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 0.00 0.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.00 ---- ---- ---- ---- 0.00 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:15,551 346891 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.01 1.02 1.00 1.06 1.05 0.93 1.09 0.97 1.00 1.04 0.79 2022-04-19 22:53:16,497 346892 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:16,509 346893 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 0.99 1.02 1.00 1.06 1.05 0.92 1.09 0.98 1.01 1.04 0.80 2022-04-19 22:53:17,461 346894 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:17,472 346895 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.00 1.02 1.00 1.06 1.04 0.93 1.09 0.98 1.01 1.04 0.73 2022-04-19 22:53:18,424 346896 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:18,436 346897 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.02 1.02 1.00 1.05 1.03 0.94 1.08 0.98 1.00 1.03 0.58 2022-04-19 22:53:19,387 346898 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 2022-04-19 22:53:19,399 346899 ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 1.00 1.02 1.00 1.06 1.05 0.93 1.09 0.98 1.01 1.04 0.73

I could not find how to contact you direclty, and issues are not the place for this, but I was really curious if we could reverse the process. What I mean by that is if we used photorealistic images as the input, could convert them into much cartoony or fictional(?) ones?

I have no affiliation with Luke Ross, I'm just a GTA player.

But the reason I suggest getting in contact with him, is because he has managed to get deep in to the GTA engine, to create a VR mod.

I read a load of information regarding how he was able to achieve this, and from my limited understanding, the things he understands about the game engine, could help you get deeper into the game's engine, allowing faster rendering with those g-buffers. (Which I believe Luke Ross has experience with, to get the VR system to work).

His GitHub name is @LukeRoss00 (ok looks like I cant tag him)... https://github.com/LukeRoss00/ (I hope I'm allowed to to that, forgive me if links aren't allowed.

But yeah, I love the video demonstration, its got me excited to see how this system can be implemented in loads of games.

Like, imagine flight simulator games, paired with a satalite imagery dataset... Slow moving gameplay allows for plenty of time to render stuff...

Truck Simulator games, where the driving style is realistic, and less busy than gta, I'd love that myself since I love just driving around in VR, with a wheel.

Please, I know it's hard, but...... Keep VR in mind..... Too many developers are brushing off VR, but.... if this became stable, and then VR on top, that would literally be a break through in technological development.... possibly Nobel prise winning, since it really would be a milestone in realtime computer generated photorealism....

Also, can we keep multi gpu support in mind? Because I would quite happily fork out for x4 3090 cards, if the software could benefit from multiple cards... Would multiple cards be able to get that half a second render time, down to a quarter, or an 8th?

Hi, very interesting work! I have a question. In epe/dataset/pfd.py, it says that number of channels G-buffers contain is {'fake':32, 'all':26, 'img':0, 'no_light':17, 'geometry':8}. Where do these numbers come from? I looked Playing for Data but couldn't find them. Thank you for any help.

Hello, I've noticed that when using the hr_new network the rec_fake (output of the generator) seems to have an unbounded value range. Compared to the loaded fake and real image which have pixels values bounded between [0, 1]

https://github.com/isl-org/PhotorealismEnhancement/blob/6936e62a96fc2d913cdd66463d113f7d591d09aa/code/epe/EPEExperiment.py#L204-L209

The 'hr' option uses the ResidualGenerator which uses a sigmoid on the output to bound the output between [0, 1] https://github.com/isl-org/PhotorealismEnhancement/blob/6936e62a96fc2d913cdd66463d113f7d591d09aa/code/epe/network/generator.py#L4-L6

However the PassthruGenerator which 'hr_new' uses doesn't have any sigmoid on the output.

Is this a bug? since it seems that the output image should be bounded [0,1] just as the real image is

Hi, my name is Luan I would like to know how I can use this AI Code to turn CGI Video or Photo to Photorealism Enhancement with subjects like Humans and Animals with and without fur, to reach a more realistic video.Thank You Sir,Best, Luan Dal Orto Vaz.

Hi Guys,

Firstly love your great work, I'm fascinated by this subject . I would like to experiment with this using some of my own data and curious to see what results I can get. But task1 is to just get this working. I'm a CGI artist and Python developer but I use Windows. To get this working with all the prerequisites is Linux required? I am willing to switch over to Lunix to work on this but would like to know if that's necessary.

Thank you for your time. Mark.

RuntimeError: Input and output sizes should be greater than 0, but got input (H: 196, W: 0) output (H: 13, W: 13)

any idea what could be the possible reason ?

I followed through the whole workflow with little trouble and after setting everything up I'm getting this error upon saving the model before training starts.

2022-11-09 17:15:04,408 Saving model to savegames/pfd2cs_ie2-0-break.
2022-11-09 17:15:07,098 Unexpected error: <class 'ValueError'>
Traceback (most recent call last):
  File "code/epe/EPEExperiment.py", line 390, in <module>
    experiment.run()
  File "/mnt/b/Dev/cv-gta/code/epe/experiment/BaseExperiment.py", line 636, in run
    self.__getattribute__(self.action)()
  File "/mnt/b/Dev/cv-gta/code/epe/experiment/BaseExperiment.py", line 543, in train
    for batch in self.loader:
  File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
    return self._process_data(data)
  File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
    data.reraise()
  File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/_utils.py", line 543, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/mnt/b/Dev/cv-gta/code/epe/matching/paired.py", line 108, in __getitem__
    idx = np.min(np.nonzero(p<self._cumsum)[0])
  File "<__array_function__ internals>", line 180, in amin
  File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 2918, in amin
    return _wrapreduction(a, np.minimum, 'min', axis, None, out,
  File "/home/chanka/anaconda3/envs/torch/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation minimum which has no identity

Help would be appreciated.

Hello,

Thanks for the great work. Inside matching/feature_based/sample_matches.py the images are loaded by calling get_by_path method of the datasets.

However, the ImageDataset class does not have such an attribute, so this ends up with an error. Replacing that with the _load_img method and applying ToTensor() on its output solved the issue for me.

For your notice. Cheers!