A new machine learning project from Intel Labs has managed to make footage from Grand Theft Auto V look even more realistic.
The 'Enhancing Photorealism Enhancement' project was carried out by Intel Labs' Stephan R. Richter, Hassan Abu AlHaija and Vladlen Koltun, who used a special method to modify images from the game, in turn making them appear more realistic.
Their model was trained to 'translate' GTA V to the Cityscapes Dataset, which contains images mostly of German cities, recorded with a car's built-in camera.
In a video about the project, the researchers explain: "With the model of reality induced by Cityscapes, our method paves the roads with smoother asphalt, increases the glossiness of car paint, and greens the sun-dried hills of GTA's California.
Advert
"The results are geometrically and semantically consistent with the input images and temporally stable.
"It is a convolutional network, which produces images frame-by-frame, and can be run at interactive rates."
While game footage sometimes 'hallucinates' features like trees, which might randomly pop up in the middle of gameplay before disappearing, the altered version appears more stable and 'produces less artefacts'.
Advert
It also removes haze and rebuilds the roads, makes grass 'more voluminous', adds reflections to the windows and increases the fresnel effect (the amount of reflectance you can see on a shiny surface from different angles, e.g. on the roof of cars).
In their paper's abstract, the authors explain: "The images are enhanced by a convolutional network that leverages intermediate representations produced by conventional rendering pipelines.
Advert
"The network is trained via a novel adversarial objective, which provides strong supervision at multiple perceptual levels.
"We analyse scene layout distributions in commonly used datasets and find that they differ in important ways. We hypothesise that this is one of the causes of strong artifacts that can be observed in the results of many prior methods.
"To address this we propose a new strategy for sampling image patches during training. We also introduce multiple architectural improvements in the deep network modules used for photorealism enhancement.
Advert
"We confirm the benefits of our contributions in controlled experiments and report substantial gains in stability and realism in comparison to recent image-to-image translation methods and a variety of other baselines."
Featured Image Credit: Intel Labs
Topics: Science, GAMING, News, Interesting, Technology, Grand Theft Auto