There are a couple of solutions to solve this problem that work almost perfectly 99% of the time. In general, they usually involve an XML file that contains corner points for each frame of how an image should be transformed to match the video. For live video (no CG) tracking is set up in After Effects and then the cue points are exported. While After Effects doesn’t have a native export cue points as XML option, the keyframe data can be copied & pasted out as plain text, and then you can just run a simple RegEx to convert it to XML. This is all pretty simple and isn’t the problem, the issue lies with not being able to get the exact frame that a video is displaying.

Simply guessing the frame based on the position property doesn’t give an accurate enough result. Another option that Zeh had discovered was using decodedFrames and droppedFrames property to get the current frame. This works pretty well, but has several drawbacks as it is a total value, so looping or seeking can become a issue, and switching tabs or right clicking can throw these values out of of sync. We had also tried using cue points on every frame that would contain the frame number, or better yet as Simon had pointed out, you can embed the tracking values directly as values for the cue points. This works pretty well, and it eliminates the need for an XML file. However, on some Macs this doesn’t work, and the video slips and becomes out of sync with the cue point data.

I’ve pretty much come to the conclusion that there is no true way to get the true current frame of a video using any properties of the stream or metadata from the video. Whatever information you receive about the video you can never be sure that’s actually what pixels are being displayed. Even if you draw the video to a bitmap, you can never be sure what frame the video object is currently displaying when you copy the pixels. That is, the only way to know what frame is displayed is to actually look at that frame.

The only true thing you can trust about a displayed frame is the frame itself. Therefore, the solution I came up with is to embed the frame value into the video itself. This can be done as a small bar at the bottom of the video that is the binary value of the current frame displayed as a barcode. Then, every frame you capture the video frame, check the pixels of the barcode, and convert the binary value back to base-10 and voilà, you have the actual current frame.

Basically your video has an extra 10 pixels at the bottom. The first 5 pixels are the bottom 5 pixels of the video mirrored. This is to ensure that during the compression bleeding and artifacts don’t appear due to the stark contrast between the video and the barcode. The last 5 rows of pixels below the mirrored strip are the barcode. There are many options to make the barcode at the bottom of the video. Zeh wrote an After Effects expression that generates it as the video renders. I made mine externally, via Photoshop JSX. A better option than Photoshop scripting would have been processing or AIR app, but at the time I was too lazy to set either of those up and honestly I wanted to explore using JavaScript in Photoshop some more.

I’m not going to pretend that this solution is by any means perfect – in fact, it is far from it. There are possible performance problems  from drawing the video every frame, and the video is slightly larger so you have to crop it. The video file might be larger in size due to less compression because of the drastic difference between frames due to the barcode. The biggest drawback is that this method requires a lot more prep work. However, the payoff is perfectly synced video to frame and you can still use FLVs and F4Vs without having to embed it into the timeline in Flash or use a PNG sequence, which would increase file size greatly.

If you want to read some more about video frame stuff, check out Zeh’s post from a few weeks ago.

You can see a demo of the final result here or below.

(Please open the article to see the flash file or player.)

On a side note, I also brewed a special beer for this project. It was a Doritos style ale, and you can read all about it on my other blog, beer.eric-decker.com. I’ll be brewing a special beer per project from now on.

And on another side note, today was my girlfriend’s birthday. She’s a painter and has a blog which she updates regularly with her work and progress, and you should defiantly check it out at eringregory.com.

NanoTween is only one class. Each tween is created as it’s own more or less independent object. They are however synchronized to a central enter frame event dispatcher, which is kept as a static instance. Instead of providing callback functions and parameters for events such as onComplete and onUpdate (as is the case with some tween engines) NanoTween instead dispatches native Event.COMPLETE and Event.ENTER_FRAME. Additionally, the Robert Penner easing functions are externalized into separate classes to keep filesize down. Special properties such as brightness are not included, but can easily be achieved though an external proxy object.

The syntax looks like this:
var tween:NanoTween = new NanoTween(myMovieClip, 1.5, {x:100, y:200}, Quad.easeOut);
tween.addEventListener(Event.COMPLETE, hndlTweenComplete);
tween.start(1);

The previous example will tween myMovieClip to the position (100,200) over the period of 1.5 seconds using the ease out Quad function, and is delayed by 1 second. Upon completion of the tween the hndlTweenComplete will be called. Likewise, the syntax can also be ‘condensed’ for simple tweens:
new NanoTween(myMovieClip, 1, {alpha:0}).start();

Anyhow, I thought it might be useful to other people, and I wanted to try out Google Code so you can grab it from the project page at: https://code.google.com/p/nanotween/. Once again, the purpose of this is not to replace mammoth tween engines such as gTween, TweenMax, or Tweener. Rather, it is intended for special cases where size is the main importance such as preloaders and banner ads.

(Please open the article to see the flash file or player.)

I probably should have mentioned this way earlier, but I’ll be speaking along with Mathieu Badimon at OFFF Paris on behalf of Firstborn. Our presentation will be on Friday, June 25th at 12:30. I have to say, this is quite an honor to speak at this conference. If you’re interested, there should be more information on the OFFF Paris 2010 website.

http://www.fitc.ca/events/presentations/presentation.cfm?event=102&presentation_id=1227

Nowadays, there are numerous ways to create websites and a large amount of tools, frameworks, packages and other reusable pieces available online.

Some of these icon libraries, 3D engines, Augmented Reality demos and diverse experiments can often give good inspiration during the conception phase And can sometimes be requested by a client (to see a “demo” of how something would work). Sometimes using these pieces in a website or other digital experience can make you feel like you’re stitching a giant patchwork blanket without really getting into the detail of each of the parts; this is sometimes a safe approach but not necessarily an innovative one.

There is a alternative approach, which gives freedom to generate new ideas and where everyone can let their imagination flow before some of these ideas eventually make the cut of the technical feasibility. Implementing such ideas is sometimes more difficult but also a more exciting and challenging path into crafting something unique.

This presentation is about clients asking questions such as: “Can I have a headphone wire as realistic as the one in the TV spot…and for it to be interactive?”, “Is it possible to colorize the car in any color I want?”, “Can the user deform the 3D face to look like himself?” and internal brainstorm sessions punctuated with: “Could a pen write whatever request I make?”, “Can we fake this 3D character into a 2D landscape?” or “What about we sew the text onto the screen itself?”

More importantly, we will go into the process and the techniques we had to come up with to answer these questions through the creation of prototypes and unique tools.

experiments.eric-decker.com/time_displacement

I basically make a huge sprite sheet of all the frames I have saved. Since Flash 10 lets you have bigger bitmaps (16,777,215 pixels total, max dimension at 8,191 src) I can create a large image that tiles all the frames I currently have saved. I can have a max of 12 frames across (at 640) and 4 frames down (at 480). So after I create/update this giant bitmap, I send it to a pixel bender filter that is currently applied to a gray scale image. The Pixel Bender filter then looks at each pixel, evaluates the brightness and interpolates it to what frame it needs to update from, then replaces the current pixel with the appropriate one from the selected frame.

These experiments are still a little rough, but I’ll post ‘em anyways. No videos yet, I need to find some time to do screen recordings. The ticker is obviously by Mr. Doob. You’ll need a webcam for all of these, sorry.

• brightness: Displacement based off of brightness of current frame.
• movement: Displacement based off of movement (difference between frames)
• painting: Displacement based off area pained by user, as in previous post (inspired by Smudge City).
• noise: Displacement based off of perlin noise.
• filmstrip: Displacement based off of titled blocks.

In any of these experiments, you can right-click to show the source that is controlling the displacement.

I had remember seeing a video (I think it was at FITC ’07) of some really cool video Processing experiments by James Seo (see Asynchrony Smudge City). He had made an experiment where you could paint a time remap on video. This experiment has always stuck with me, as I always thought it was really well done. So after doing the previous experiment, I thought I would like to try what he had done too, except I choose to do it in Flash (which, I imagine, would make it perform worse than Processing).

It ended up being a fun little project. First, I just paint into a hidden canvas and capture that bitmap data. Then, depending on the number of frames recorded, I break out multiple version of the painting each with a different threshold. I then use this image as an alpha mask via the Layer blend mode to mask out the frame. Lastly, I draw each one to the resulting bitmap, which is displayed. Additionally, since the threshold method is 0 and 1 there are hard edges around each threshold, so I also apply a blur filter (16 x 16, quality 1) to help them blend a little more. This ends up being very taxing, and causes for a lot of slow down. In the demo, you can right-click and toggle this on and off to see the difference for yourself. I also did a version with PixelBender which I need to tweak a little bit, but the performance doesn’t seem to be that great (although I might be able to optimize it still).

Play with it here: Webcam Time Painting. There are also some video recordings below.

(Please open the article to see the flash file or player.)

(Please open the article to see the flash file or player.)

So, applied to a webcam video on an interval, a bitmap data snapshot is taken of the current video frame. Then it is encoded as a JPEG and returned as a ByteArray. The ByteArray is then loaded back into Flash though the loader, which takes a one frame delay, which I imagine is because the loader object probably needs to initialize. Then, the bitmap is extracted from the loader the same way any image is, and the bitmapdata is copied to a display bitmap on the stage, and the cycle loops again.

You’ll notice a pretty significant delay on the demo. The delay comes from the encoding of the image, not the loading. You can play with the example here: JPEG Video Experiment. No webcam? There’s a prerecorded video below.

(Please open the article to see the flash file or player.)

So I decided to make a really simple webcam capture utility using Flash. Even if there was some really simple app out there, as I am sure there is, the developer urge inside me just wanted to create one anyways. I figure this way is pretty cool, since all you need is the webcam driver (so that the webcam actually works) and Flash Player 10 installed (which is current at 94% penetration). It’s just a simple little tool, you can re-size the image and save it as a jpeg. That’s really all the functionality that I needed from it.

Anyways, you can check it out here:
experiments.eric-decker.com/webcam_snapshot

I’ve also used it to take some macro snapshot, as apparently my webcam can focus insanely close on things, and I’ve uploaded them to my flickr account:
flickr.com/photos/ericdecker/sets/72157623403965642