Video Sync Issues with Flash AS3

It’s been almost a year exactly since my last update. I guess I’ve been pretty preoccupied. I wanted to do a post about an age old problem very often encountered in Flash – compositing video so that overlaying elements sync perfectly with a video. In essence, getting the true current frame of an FLV or F4V. Specifically the problem I want to talk about tackling is having an image be composited as if it were in the video. I think a great example of this is a site we (as in Firstborn, not me) did for 5 React a year or so ago. There are a couple of solutions to solve this problem that work almost perfectly 99% of the time. In general, they usually involve an XML file that contains corner points for each frame of how an image should be transformed to match the video. For live video (no CG) tracking is set up in After Effects and then the cue points are exported. While After Effects doesn’t have a native export cue points as XML option, the keyframe data can be copied & pasted out as plain text, and then you can just run a simple RegEx to convert it to XML. This is all pretty simple and isn’t the problem, the issue lies with not being able to get the exact frame that a video is displaying.

after-effects-cue-points
Simply guessing the frame based on the position property doesn’t give an accurate enough result. Another option that Zeh had discovered was using decodedFrames and droppedFrames property to get the current frame. This works pretty well, but has several drawbacks as it is a total value, so looping or seeking can become a issue, and switching tabs or right clicking can throw these values out of of sync. We had also tried using cue points on every frame that would contain the frame number, or better yet as Simon had pointed out, you can embed the tracking values directly as values for the cue points. This works pretty well, and it eliminates the need for an XML file. However, on some Macs this doesn’t work, and the video slips and becomes out of sync with the cue point data. I’ve pretty much come to the conclusion that there is no true way to get the true current frame of a video using any properties of the stream or metadata from the video. Whatever information you receive about the video you can never be sure that’s actually what pixels are being displayed. Even if you draw the video to a bitmap, you can never be sure what frame the video object is currently displaying when you copy the pixels. That is, the only way to know what frame is displayed is to actually look at that frame. The only true thing you can trust about a displayed frame is the frame itself. Therefore, the solution I came up with is to embed the frame value into the video itself. This can be done as a small bar at the bottom of the video that is the binary value of the current frame displayed as a barcode. Then, every frame you capture the video frame, check the pixels of the barcode, and convert the binary value back to base-10 and voilà, you have the actual current frame. Basically your video has an extra 10 pixels at the bottom. The first 5 pixels are the bottom 5 pixels of the video mirrored. This is to ensure that during the compression bleeding and artifacts don’t appear due to the stark contrast between the video and the barcode. The last 5 rows of pixels below the mirrored strip are the barcode. There are many options to make the barcode at the bottom of the video. Zeh wrote an After Effects expression that generates it as the video renders. I made mine externally, via Photoshop JSX. A better option than Photoshop scripting would have been processing or AIR app, but at the time I was too lazy to set either of those up and honestly I wanted to explore using JavaScript in Photoshop some more.

video-mirrored-and-barcode

I’m not going to pretend that this solution is by any means perfect – in fact, it is far from it. There are possible performance problems  from drawing the video every frame, and the video is slightly larger so you have to crop it. The video file might be larger in size due to less compression because of the drastic difference between frames due to the barcode. The biggest drawback is that this method requires a lot more prep work. However, the payoff is perfectly synced video to frame and you can still use FLVs and F4Vs without having to embed it into the timeline in Flash or use a PNG sequence, which would increase file size greatly. If you want to read some more about video frame stuff, check out Zeh’s post from a few weeks ago. You can see a demo of the final result here.

4 thoughts on “Video Sync Issues with Flash AS3”

  1. Thank You Decker For This Brillant Solution ! I’d be thankfull if you would please attach your project files here as open-source.
    Thanks Again :)

Leave a Reply

Your email address will not be published. Required fields are marked *