Animation Technology VFX

Face2Face: Real-time Face Capture

I have already blogged about software that allows actors facial expressions to be edited in post. Now take a look at Face2Face: Real-time Face Capture. It can map new facial expressions real time over video. While very interesting from a technological viewpoint, the idea of ‘photoshopping” video will certainly affect journalistic ethics and the trustworthiness of video evidence.

From Michael Zhang Petapixel.

Face swap camera apps are all the rage these days, and Facebook even acquired one this month to get into the game. But the technology is getting more and more creepy: you can now hijack someone else’s face in real-time video.

A team of researchers at the University of Erlangen-Nuremberg, Max Planck Institute for Informatics, and Stanford University are working on a project called Face2Face, which is described as “real-time face capture and reenactment of RGB videos.”


Basically, they’re working on technology that lets you take over the face of anyone in a video clip. By sitting in front of an ordinary webcam, you can, in real-time, manipulate the face of someone in a target video. The result is convincing and photo-realistic.


The face swap is done by tracking the facial expressions of both the subject and the target, doing a super fast “deformation transfer” between the two, warping the mouth to produce an accurate fit, and rerendering the synthesized face and blending it with real-world illumination.


To test the system, the researchers invited subjects to puppeteer the faces of famous people (e.g. George W. Bush, Vladimir Putin, and Arnold Schwarzenegger) in video clips found on YouTube. You can see the results (and an explanation of the technology) in this 6.5-minute video:

Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, June 2016.


We present a novel approach for real-time facial reenactment of a monocular target video sequence (e.g., Youtube video). The source sequence is also a monocular video stream, captured live with a commodity webcam. Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion. To this end, we first address the under-constrained problem of facial identity recovery from monocular video by non-rigid model-based bundling. At run time, we track facial expressions of both source and target video using a dense photometric consistency measure. Reenactment is then achieved by fast and efficient deformation transfer between source and target. The mouth interior that best matches the re-targeted expression is retrieved from the target sequence and warped to produce an accurate fit. Finally, we convincingly re-render the synthesized target face on top of the corresponding video stream such that it seamlessly blends with the real-world illumination. We demonstrate our method in a live setup, where Youtube videos are reenacted in real time.

See Matthias Nießner for more info.


It can also be done with 2 live cameras.

ACM Transactions on Graphics 2015 (TOG)


We present a method for the real-time transfer of facial expressions from an actor in a source video to an actor in a target video, thus enabling the ad-hoc control of the facial expressions of the target actor. The novelty of our approach lies in the transfer and photo-realistic re-rendering of facial deformations and detail into the target video in a way that the newly-synthesized expressions are virtually indistinguishable from a real video. To achieve this, we accurately capture the facial performances of the source and target subjects in real-time using a commodity RGB-D sensor. For each frame, we jointly fit a parametric model for identity, expression, and skin reflectance to the input color and depth data, and also reconstruct the scene lighting. For expression transfer, we compute the difference between the source and target expressions in parameter space, and modify the target parameters to match the source expressions. A major challenge is the convincing re-rendering of the synthesized target face into the corresponding video stream. This requires a careful consideration of the lighting and shading design, which both must correspond to the real-world environment. We demonstrate our method in a live setup, where we modify a video conference feed such that the facial expressions of a different person (e.g., translator) are matched in real-time.

Animation Disney Technology VFX

Disney’s Augmented Reality Characters from Colored Drawings

Photo from the Verge.

A Disney Research team has developed technology that projects coloring book characters in 3D while you’re still working on coloring them. The process was detailed in a new paper called “Live Texturing of Augmented Reality Characters from Colored Drawings,” and it was presented at the IEEE International Symposium on Mixed and Augmented Reality on September 29th. That title’s a mouthful, but it’s descriptive: the live texturing technology allows users to watch as their characters stand and wobble on the page and take on color as they’re being colored in. You can see an example in the video above: the elephant’s pants are turning blue on the tablet screen just as they’re being filled on the page itself.

Coloring books capture the imagination of children and provide them with one of their earliest opportunities for creative expression. However, given the proliferation and popularity of digital devices, real-world activities like coloring can seem unexciting, and children become less engaged in them. Augmented reality holds unique potential to impact this situation by providing a bridge between real-world activities and digital enhancements. In this paper, we present an augmented reality coloring book App in which children color characters in a printed coloring book and inspect their work using a mobile device. The drawing is detected and tracked, and the video stream is augmented with an animated 3-D version of the character that is textured according to the child’s coloring. This is possible thanks to several novel technical contributions. We present a texturing process that applies the captured texture from a 2-D colored drawing to both the visible and occluded regions of a 3-D character in real time. We develop a deformable surface tracking method designed for colored drawings that uses a new outlier rejection algorithm for real-time tracking and surface deformation recovery. We present a content creation pipeline to efficiently create the 2-D and 3-D content. And, finally, we validate our work with two user studies that examine the quality of our texturing algorithm and the overall App experience.

Download File “Live Texturing of Augmented Reality Characters from Colored Drawings-Paper”
[PDF, 1.72 MB]

Copyright Notice
The documents contained in these directories are included by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author’s copyright. These works may not be reposted without the explicit permission of the copyright holder.

Animation Art Design Technology

Watch: How Andy Warhol’s Lost Computer Art Was Finally Found

Andy Warhol with Debbie Harry at the 1985 Amiga Launch at Lincoln Center. Photo Computer History Museum

From Warhol & The Computer.

In Walter Isaacson’s biography of Steve Jobs, he describes Sean Lennon’s ninth birthday party at the Dakota Apartments in New York. Dozens of significant figures in the arts and entertainment industry were there to attend the birthday of the son of the late John Lennon, including artists Louise Nevelson and Keith Haring. Warhol was in attendance, as he was at a great many New York society parties, but a slightly surprising guest was Steve Jobs, co-founder of Apple Computer. Jobs brought with him a Macintosh computer. At first, Warhol and Haring watched as Jobs showed Sean how to work with the machine. After a while, Warhol took Sean’s place in front of the Mac while Jobs tried to explain how to use a mouse. It took a while, but finally Warhol used the Pencil tool to draw.

“Look! Keith! I drew a circle!” Warhol said to Haring.

This was Warhol’s introduction to creating art with computers, though Jobs had called him previously trying to give him a Macintosh. Warhol had never returned his calls, seeing no reason to consider computers.

From  Lost and Found: Andy Warhol’s Amiga Artworks at the Computer History Museum.

Scattered on floppy disks and hard drives around the world, there may be millions of works of art created on now-archaic computer systems. Most of these are personal treasures, perhaps of value only to their creators or to their immediate circle of friends and family. Some computer art is of interest to researchers as an example of the progression of artistic tools that were used at a given time. Still others are just bad art, best forgotten.

What happens when a major figure in the art world has created works on a system that is decades-old and whose images are stored on potentially unreadable media? That was exactly the problem that faced a team of student researchers, curators, and perhaps most significantly, artist Cory Arcangel, in relation to the files created by the legendary Pop Artist Andy Warhol on the Amiga.

One of the forty-one disks disks containing Andy Warhol's Amiga files.
One of the forty-one disks disks containing Andy Warhol’s Amiga files.

Cory Arcangel describes himself as an ‘Acolyte of Andy Warhol.’ “Between DuChamp and Warhol, those are the two big influences for the kind of art I make,” says Arcangel.  Arcangel’s works span both physical and digital realms, as well as static and moving pieces, much like Warhol’s own works. Videos such asA couple of thousand short films about Glenn Gould and Sans Simon, as well as web-based works like Punk Rock 101, and even Nintendo cartridge hacks like Super Mario Clouds, make arcangel a multi-dimensional artist whose works can be seen as inhabiting areas that Warhol himself likely would have waded into had he not died in 1987.

Arcangel noted Warhol’s influence, “…when I first moved to New York and started becoming an artist, you know, I read his diaries, and for me, they were really helpful because they taught me, in a way, how to be an artist. Not in the poetic sense, but in the day-to-day sense, like ‘I went to work today. I made three paintings. Then I went to an opening.’ like a real kind of ‘this is what you do if you’re an artist. The life of an artist’”

Warhol’s interaction with the Amiga started with the introduction of the Amiga 1000 computer in 1985. He took part in a launch event at New York’s Lincoln Center, creating a piece of very Warholian art, using a scanned image of Debbie Harry of the New Wave act Blondie. “I’m going to do this ‘How to Paint’ thing that this computer company wants me to do,” Warhol noted in his diaries. It was a video of the event uploaded to YouTube, along with the mention in the Warhol diaries, that caught the attention of arcangel, who has used obsolete technology in many of his works. He became interested in Warhol’s work with the Amiga, in particular, whether any of Warhol’s floppy disks had survived. He contacted the Andy Warhol Museum in Pittsburgh in a quest to find out more about the condition of Warhol’s Amiga works and equipment.

Commodore Amiga computer equipment used by Andy Warhol 1985-86
Commodore Amiga computer equipment used by Andy Warhol 1985-86.

Warhol was given at least one full Amiga 1000 system with a drawing pad, camera, and various pieces of drawing software. He created several works on the Amiga, often using familiar subjects such as Marilyn Monroe and Campbell’s soup cans, but giving them a new sensibility harnessing the techniques that the Amiga provided. Using the GraphiCraft painting system and other software, for example, Warhol created images that emulated techniques similar to his silkscreen creations of the 1960s, such as bright colors layered over black and white photos of celebrities and newspaper photos.

After his death, Warhol’s works, as well as a great deal of other material, eventually went to the Andy Warhol Museum in Pittsburgh. Commenting on this massive amount of material, Arcangel told Themed for Your Pleasure podcast host Vanessa Applegate, “Warhol was a hoarder. A maximum, a hundred and twenty percent hoarder, so the amount of material that the Warhol Museum has is mind-boggling.” This led to much of the material being little examined; not lost, but over-looked among hundreds and thousands of other items relating to Warhol’s life and art.

“The (Warhol) museum has so much stuff, they’re still going through it.” Arcangel noted. Among the items were 41 floppy disks, Warhol used to store the works he created on the Amiga. These disks contained many pieces, and while Museum archivist Matt Wrbican knew of their existence, no attempt had been made to read the disks or recover the images.

Wrbican noted, “Really, we didn’t have any clue what was on the disks. At that point, I only knew of two images that Warhol had made (on the Amiga). I assumed there were lots of other images because Andy’s assistant Jay (Shriver) told me that he had played around with the images for a while, but exactly what those images were I had no idea.”

The two images Wrbican had seen, the Debby Harry photo manipulation done at the Amiga Launch and a self-portrait created for the cover of Amiga World Magazine, were widely-known, but no one was certain about the contents of the disks the museum held.”They were as well taken-care of as they could be. They were in incredible shape,” arcangel noted.

In 2011, Arcangel connected with the Carnegie Mellon University Computer Club, known for a great many retro-technology feats. Working with Arcangel, Wrbican and others, members of  the club began reading the disks, using the best digital archival practices. The Kryoflux, an interface that allowed them to read disks from the Amiga and bring the files into a modern PC, way key. One rule was that each disk should only be handled and read only once. “They were like ‘we need to do it with the Kryoflux. We need to do it at the lowest level. They were the people pushing, making sure everything was at a very high level, [an] archival preservation project. I can’t say enough nice things about the Carnegie Computer Club,” arcangel said, “there is nothing cooler than a retro-computing hacking club in my mind! Especially at a place like Carnegie Mellon.”

By the end of 2013, the club had read all of the disks and began working on recovering the individual files using emulators running on modern PCs. The recovered images were definitely Warhol creations, and while rather small by today’s standards (200 x 300 pixels), they show Warhol’s famous style. Using techniques and tools like the paint bucket and fill, he created images that felt like the silkscreened photograph-based works that have become beloved and highly collectable in the years since. A self-portrait, called Andy 2, features a photo of Warhol that has had fields of textured color layered over it. The effect is right in line with his self-portraits of the 1960s and 70s. “…They were really amazing,” arcangel says. “There’s one of Marilyn Monroe that he had sort of splashed all over… Fantastic!” and “he couldn’t have done any better.”

age: Andy Warhol, Campbell’s, 1985, ©The Andy Warhol Foundation for the Visual Arts, Inc.
Andy Warhol, Andy 2, 1985, ©The Andy Warhol Foundation for the Visual Arts, Inc.

Not all of the pieces recovered were intended to be artworks. Some were test pieces used to discover the capabilities of the Amiga. In much the same way a painter may hone brush technique through practice on works never intended to see the light of day, Warhol apparently created similar images to allow himself to experiment with the Amiga environment.

Arcangel commented, “what is amazing to me is that he seemed to get it really quick, really quick… how to work with the computer, and how to work with the paint bucket. You can see a couple of them where he’s trying to learn the copy/paste [function] and he’s learning to use the paint bucket, and then all of a sudden, there are a couple that are just amazing.”

He also created images from scratch, including a Campbell’s soup can–the iconic image of his days as Pop Art’s leading practitioner and art world Bad Boy.  The mainstream art establishment had been slowly embracing works done with the aid of computers since the 1960s, but these works were created with large computers—including mainframes and supercomputers. Steve Jobs had made it something of a mission to bring the Macintosh to artists. The artist Keith Haring had been introduced to the Apple Macintosh computer at the same time as Warhol, but integrated it into his work much more. Graphic artists had also gravitated towards the Mac, but the Amiga was billed as a ‘Multi-Media Computer,” and that may have drawn Warhol towards it.“The thing that I like most about doing art on the computer is that it looks like my work.” Warhol once said.

Andy Warhol, Campbell’s, 1985, ©The Andy Warhol Foundation for the Visual Arts, Inc.
Andy Warhol, Campbell’s, 1985, ©The Andy Warhol Foundation for the Visual Arts, Inc.

Arcangel’s passion for the project, the skill and technological know-how of the CMU Computer Club, and the background of the archivists and curators at the Andy Warhol Museum, all led to the extraction of twenty images. The disks, along with Andy’s Amiga 1000 computer, his software, graphics tablet, camera–as well as the images that the Carnegie Mellon Computer Club extracted from the disks–are in the permanent collection of the Andy Warhol Museum in Pittsburgh. A documentary, The Invisible Photograph: Part II (Trapped), chronicles the interactions between the Warhol Museum, the CMU Computer Club, and Arcangel, as well as the Warhol-Harry portion of the Amiga product launch. These materials show how one of the most significant artists of the 20th century expressed his artistic vision using one of the most important tools ever created, the computer. They also represent some of Warhol’s final art pieces, created over the last eighteen months of his life.

Click to enlarge.

Animation Disney Film Editing Film Sound People Technology

Veteran ADR Mixer Doc Kane of Walt Disney Studios

Stage B – the birthplace of the creative voice track of many animated films including Cars 2, Toy Story 3, Tangled, Rango and many more, is also the ADR home of top industry filmmakers

In this exclusive SoundWorks Collection profile we talk with veteran ADR mixer Doc Kane at Walt Disney Studios in Burbank, CA to explore his extensive project credits and unique approach to capturing some of Hollywood’s most talented voices.

The Art of ADR

The art of ADR is much more than having a collection of microphones and knowing how to use them, although Doc’s mic cabinet is pretty impressive. It’s also more than having the latest and greatest hardware and software, but rest assured, Doc has all of the most modern bells and whistles.Perhaps even more important and some might even argue that it qualifies ADR as an art, is the sensitivity to the client.

What has three letters, many aliases and is of major significance to the sound community? You guessed it: ADR aka Automated Dialog Replacement aka Additional Dialog Recording aka Dubbing aka Looping. All of these monikers are understood as the process of re-recording dialog that cannot be salvaged from a production. To make one thing clear, there is nothing automated about it. ADR is an art. And here to tell us more about the art is an artist whose name also has only three letters and many aliases but nonetheless has made a significant impact on the sound community.

His name is Doc Kane but most just call him Doc. He has over 300 projects under his belt and a slew of awards and nominations, including four Academy Award nominations.

Tom Hanks talks about the fact that the voice of Woody for toys and games is sometimes actually the voice of his brother, Jim. He tells a story about what it is like working on Stage B when he is recording the voice of Woody for the Toy Story films.

Animation Disney Technology

Glen Keane ‘drawing sculptures’ in virtual reality with Tilt Brush

Photo from Wired

“I can put goggles on and I just step into the paper and now I’m drawing in it,” Keane says. “Today, all the rules have changed.”

“All directions are open now, just immersing myself in space is more like a dance. What is this amazing new world I just stepped into? When I draw in VR I draw all the characters real life size. They are that size in my imagination. The character can turn […] and even if you take the goggles off, I’m still remembering — she’s right there, she’s real.”

Over nearly four decades at Disney, Glen Keane animated some the most compelling characters of our time: Ariel from The Little Mermaid, the titular beast in Beauty and the Beast, and Disney’s Tarzan, to name just a few. Keane has spent his career embracing new tools, from digital environments to 3D animation to today’s virtual reality, which finally enables him to step into his drawings and wander freely through his imagination. At FoST, he’ll explore how to tap into your own creativity, connecting to emotion and character more directly than ever before.

More on the FoST Summit here. website.

More about Tilt Brush


Animation Filmmaking Technology VFX

“What Lives Inside” Episodes

I previously wrote about What Lives Inside here.

This is the future folks, computer companies producing a movie to be shown on a streaming service.

Get ready to be taken to a world beyond your imagination. From Academy Award Winner Robert Stromberg, Dell presents What Lives Inside. Starring Academy Award Winner J.K. Simmons, Colin Hanks and Catherine O’Hara. Premiering March 25th only on Hulu.

This is the Episode playlist and some behind the scenes.

Here is the Making of video.

Intel Dell What Lives Inside – Behind the Scenes by CGMeetup

Intel Dell What Lives Inside – VFX Breakdown by CGMeetup

Animation Filmmaking Technology VFX

“What Lives Inside” Official Trailer and behind the scenes

This is the future folks, computer companies producing a movie to be shown on a streaming service.

Get ready to be taken to a world beyond your imagination. From Academy Award Winner Robert Stromberg, Dell presents What Lives Inside. Starring Academy Award Winner J.K. Simmons, Colin Hanks and Catherine O’Hara. Premiering March 25th only on Hulu. Find out more at here.

From Fast Co.Create.

What Lives Inside is the fourth installment of Intel’s “Inside Films” series, dating back to a partnership with Toshiba, and agency Pereira & O’Dell, that started in 2011 with Inside, starring Emmy Rossum and directed by D.J. Caruso. It was followed by 2012’s The Beauty Inside starring Topher Grace, and 2013’s The Power Inside starring Harvey Keitel.

This year’s film, divided into four episodes, is about the son of an absentee father (Hanks) who finds himself on a journey of self-discovery after the death of his father (Simmons), a well-known and acclaimed children’s puppeteer who was widely celebrated for his creativity. The son discovers a mysterious world of his dad’s creation and finds himself on an adventure that will soon unlock his own creativity.


Pereira & O’Dell chief creative officer PJ Pereira says one of the biggest challenges this year was finding another fresh way of bringing the same premise—Intel tagline “It’s what’s inside that counts”—to life. “We had to find a role to make the product not the subject of the story we are telling, but a character,” says Pereira. “Because characters are what the audience will remember and love months after the campaign is gone.”

In addition to Oscar-winning talent, each Inside Films series featured a social element, soliciting submissions from people for the chance to see their photo and videos for the film, or even audition for a part. This year, it worked a bit differently. “This time, because the central theme is their creativity, that’s what is on display. Their drawings, as if they were all kids that have submitted ideas to the character played by J.K. Simmons,” says Pereira.

Just six weeks after Stromberg issued a challenge online, the film received thousands of creature submissions and more user-generated content than the two previous films combined. “This project always felt more like a film than an ad, with its longer format, incredible cast and extensive visual effects,” says Stromberg, who won art direction Oscars for Alice in Wonderland and Avatar. “The whole interactive angle is also super interesting to me. We’ve had over 6,000 submissions of art work, which is crazy! I just think that’s a much better indicator of engagement than throwing a project into testing. I love how it lets people be an active part of the final product. Any time I can be a part of inspiring others to get in touch with their creative side, only inspires me more as an artist.”

The film debuts on Hulu, with new episodes weekly for four weeks, then starting May 6 the full series will be available on and YouTube.


1970 Rod Serling Ad for Radio Shack (video)


Radio Shack is closing. For those who were electronics hobbyists, like myself, Radio Shack was the likely introduction to the trends and fads in consumer electronics. AM and FM radio, stereo, turntables, open reel tape recording, cassettes, 8 track, CB radio, satellite tv, video tape, camcorders, phones, and cell phones.

They also sold a computer called the TRS 80, pejoratively called the “Trash 80.” Of course, you could get accessories and discrete components.

The above website has many old catalogs as well as a thorough history. My Dad liked Radio Shack because when he was young, he went to the large store in Boston.