decker labs

I decided to try using Pixel Bender to do the time displacement in the recent experiments I’ve been doing, as I was hoping it would perform better. It actually seems to, but there still is a significant lag. However, I’m pretty sure this is because of having to make a huge sprite sheet first.

I basically make a huge sprite sheet of all the frames I have saved. Since Flash 10 lets you have bigger bitmaps (16,777,215 pixels total, max dimension at 8,191 src) I can create a large image that tiles all the frames I currently have saved. I can have a max of 12 frames across (at 640) and 4 frames down (at 480). So after I create/update this giant bitmap, I send it to a pixel bender filter that is currently applied to a gray scale image. The Pixel Bender filter then looks at each pixel, evaluates the brightness and interpolates it to what frame it needs to update from, then replaces the current pixel with the appropriate one from the selected frame.

These experiments are still a little rough, but I’ll post ‘em anyways. No videos yet, I need to find some time to do screen recordings. The ticker is obviously by Mr. Doob. You’ll need a webcam for all of these, sorry.

• brightness: Displacement based off of brightness of current frame.
• movement: Displacement based off of movement (difference between frames)
• painting: Displacement based off area pained by user, as in previous post (inspired by Smudge City).
• noise: Displacement based off of perlin noise.
• filmstrip: Displacement based off of titled blocks.

In any of these experiments, you can right-click to show the source that is controlling the displacement.

I started playing around with doing some experiments with video and time delay. When showing the previously mentioned JPEG video experiment to my coworker Roushey, he mentioned how just the effect of the time delay was interesting. I was thinking about neat things to do with, and the first idea I had was to offset each frame as one row or column of pixels. So in a 640 x 480 video, the left-most column of pixels would be from the current frame, and the last rightmost column would be from 639 frames ago. The experiment is live here, and there’s a video of it on YouTube.

I had remember seeing a video (I think it was at FITC ‘07) of some really cool video Processing experiments by James Seo (see Asynchrony Smudge City). He had made an experiment where you could paint a time remap on video. This experiment has always stuck with me, as I always thought it was really well done. So after doing the previous experiment, I thought I would like to try what he had done too, except I choose to do it in Flash (which, I imagine, would make it perform worse than Processing).

It ended up being a fun little project. First, I just paint into a hidden canvas and capture that bitmap data. Then, depending on the number of frames recorded, I break out multiple version of the painting each with a different threshold. I then use this image as an alpha mask via the Layer blend mode to mask out the frame. Lastly, I draw each one to the resulting bitmap, which is displayed. Additionally, since the threshold method is 0 and 1 there are hard edges around each threshold, so I also apply a blur filter (16 x 16, quality 1) to help them blend a little more. This ends up being very taxing, and causes for a lot of slow down. In the demo, you can right-click and toggle this on and off to see the difference for yourself. I also did a version with PixelBender which I need to tweak a little bit, but the performance doesn’t seem to be that great (although I might be able to optimize it still).

Play with it here: Webcam Time Painting. There are also some video recordings below.

(Either JavaScript is not active or you are using an old version of Adobe Flash Player. Please install the newest Flash Player.)

(Either JavaScript is not active or you are using an old version of Adobe Flash Player. Please install the newest Flash Player.)

A few days ago I was thinking if there was a way to reproduce the effect of JPEG compression real time in flash. I figured the best way would be to literally encode the image with the native AS3 JPEGEncoder class. The problem there was that the returned ByteArray could not be interpreted as an image anymore. However, the Loader class does allow for loading a ByteArray and converting it to a Bitmap. And luckily, the loadBytes method let us load the ByteArray right from Flash (as opposed to having to save out a file and load it externally).

So, applied to a webcam video on an interval, a bitmap data snapshot is taken of the current video frame. Then it is encoded as a JPEG and returned as a ByteArray. The ByteArray is then loaded back into Flash though the loader, which takes a one frame delay, which I imagine is because the loader object probably needs to initialize. Then, the bitmap is extracted from the loader the same way any image is, and the bitmapdata is copied to a display bitmap on the stage, and the cycle loops again.

You’ll notice a pretty significant delay on the demo. The delay comes from the encoding of the image, not the loading. You can play with the example here: JPEG Video Experiment. No webcam? There’s a prerecorded video below.

(Either JavaScript is not active or you are using an old version of Adobe Flash Player. Please install the newest Flash Player.)

Every once and a while, I find that I need to just take a quick snapshot with my webcam at work. Since I’m on a PC, so I don’t have Apple’s pre-installed Photobooth app, or any other simple program (that I’m at least aware of). I also didn’t want to install any bulky 3rd party software that might come with the webcam that I’m using, since all I need in order for it to work is the driver, which more or less Windows installed all by itself. Photoshop has an option to capture from webcam, but the util is pretty limited as it only lets me capture a very small thumbnail size image.

So I decided to make a really simple webcam capture utility using Flash. Even if there was some really simple app out there, as I am sure there is, the developer urge inside me just wanted to create one anyways. I figure this way is pretty cool, since all you need is the webcam driver (so that the webcam actually works) and Flash Player 10 installed (which is current at 94% penetration). It’s just a simple little tool, you can re-size the image and save it as a jpeg. That’s really all the functionality that I needed from it.

Anyways, you can check it out here:
experiments.eric-decker.com/webcam_snapshot

I’ve also used it to take some macro snapshot, as apparently my webcam can focus insanely close on things, and I’ve uploaded them to my flickr account:
flickr.com/photos/ericdecker/sets/72157623403965642

Damn, it’s been a long time since I’ve had a site a launch. But I’m happy to announce my latest project at Firstborn has gone live: SoBe.com.  I was the lead flash developer for this project. The current SoBe campaign also featured a facebook app with contest and a mobile version of the site as well (which were not developed by me, but Phil and Miller respectively).

There are two sides to the site. The product ‘flavors’ section features videos of real people stopped and interviewed on street about what they think of SoBe lifewater (which we, firstborn, filmed). We used streaming video from FMS provided by Akamai in order to provide responsive seekable video. This was actually a challenge, as streamed video has some slight programmatic behavior differences than progressive download. For instance, streamed video will not dispatch any Net Events for when it has reached the end of the stream. Another issue I ran into was that due to security restrictions, the  BitmapData.draw function is prohibited unless specifically enabled on the FMS side. The other side of the site, ‘world’, was pretty straight forward, except that since we will be doing updates for the rest of the year the site needed to be extremely dynamic. This section of the site also features actress Ashley Greene in the nude, painted in two skinsuits to promote SoBe lifewater’s two new flavors.

Recently I did some playing around with video and thresholds, mostly just for fun. I first experimented with just changing the threshold of the video, essentially converting it to black and white. Then I tried comparing the current frame threshold and the previous frame threshold to get the difference. The result looks like an outline/edge effect. It’s an interesting way to visualize movement. Then, just for kicks, I decided to display the previous comparison and the current one, colorizing the current difference as red and the previous as cyan. The result is a very pseudo-3D outline effect. This fake 3D technique can also be applied to to straight up video, showing the previous frame as cyan and the current as red and screening the two. Since this all based on movement and the idea that objects closer to the camera appear to move greater distances due to their perspective, it’s very easy for it to display incorrectly.

Check out the first demo. You’ll need a webcam. Click on the video to change the mode, and the two scroll bars control the threshold level and amount of blur applied to the video before any image processing.

I then noticed that while changing the threshold, it almost seemed like it was displaying different depth passes on the video. Of course, this has nothing to do with depth, but rather brightness. Still, I wanted to see what it would look like if I broke out multiple thresholds of the same frame and layered them in 3D. The result, although unpractical, still looks pretty cool.

Check out the second demo. There’s different options on the left and two scroll bars at the bottom that control layer transparency and amount of blur applied to the video before any image processing.

Recently I’ve been working on making a front end profanity filter. It makes more sense to have a filter be on the server side and return an accepted or rejected response, but since my strength is in AS3 I’ve decided to do it this way. There’s two parts I want to explain, the first being the reason for a profanity filter, and the second being the actual code. You can check out the demo here.

Why

If you have a site that lets users submit text and you want to control what is displayed, you’re going to need some sort of moderation system. There are two main options. The first is a profanity filter, which is a script that looks for certain words or word patterns, and allows or blocks them based on it’s parameters. The strength here is that it is automatic, but the weakness is that it can’t always catch everything, due to new slang, misspellings, context, and creative punctuation. Additionally, a filter that tries to block too much might end up blocking actually appropriate submissions. For example, for a website that lets users submit their favorite books, the filter might block “My Big Hairy Dick” but it might also block “Dick Tracy” just because it picked up on the word “dick.” Context is a major weakness, which brings us to the second option, which is a live moderator. The strengths of having a live person allow and reject is that they can easily see past misspellings, have a better idea on new slang, and most of all, they can interpret context. The downside is that the content cannot go live until it is approved, and if there are lots of submissions, that means more work for the moderator, meaning more time and more money.

My solution is to use both. First, a profanity filter is not enough, unless you don’t really care what gets by. But in most cases, clients want to make sure nothing offensive is posted to their site, both in terms of vulgarities and  brand defacement. The trick with writing a profanity filter is not necessarily blocking offensive words, instead it’s allowing appropriate words. For example, when to allow the word “ass.” You can block the word ass alone, but what about ass tied with other words, such as asshole, assface, asshat, etc. You can combine ass with many nouns to create something sounding [absurdly] offensive. And you cannot just block all instances of ass+word, because then you block assistance, assets, assist, etc. When writing  a profanity filter, the filter should fire little to no false positives.

But if there’s a person moderating content, then why bother at all with a front end profanity filter? The reason is to make that person’s job easier. The profanity filter’s job is to cut down on the amount of submissions by rejecting the without a doubt non-acceptable ones. If a site gets thousands of submissions and one person needs to go though them one by one and approve, that can take a long time. But if we can reject a handful from the start that will make that person’s job much easier. The profanity filter should never “assume” that something is offensive, – if something might be offensive (such as the “Dick” example above), then let it pass and the moderator will decide based on it’s context. I worked on a site that used this technique (in this case it was just a word list and a php script that matched literals), and we tracked the number of times users clicked submit and the number of times the profanity filter returned true (meaning the submission was rejected). The result was that over one third of submissions were blocked by a very basic filter, which gives me reason to believe that having both systems is defiantly a good idea. Now, as with the previous case, the filter could just be a gigantic word list congaing every variation of certain words, but it’s more fun to try and make a more “smart filter” that can take care of leetspeak and word variations on it’s own.

How

Rather than looking for exact words, the filter I wrote looks more or less for word patterns. This was a good case to dive into some regular expressions. The filter has three word lists (which can be set by the user, as this content should be external). The first word list are words to flag no matter what – words such as “fuck” that in the terms of the site are never appropriate or a part of any word. The second set is a list of words that are only to be flagged if written on their own, so we detect “ass” but not “assist” or “assface.” I also test for plurals and other suffixes automatically, so not only would “shit” be matched, but “shits,” “shitting,” etc. The third list is an allow list, words that might be exceptions to the previous list. For instance, block “crap” and “crappy” but don’t block “craps,” as craps is a casino game.

The first step is converting the word lists into regular expression patterns. The first step is that each word from the profanity list is split up so that various characters can be replaced. For instance, asshole can be written as a$$hole. So rather than match /asshole/ we replace letters with boolean values.  Instead we have something that looks like this: /a(s|$)(s|$)hole/.  When the regular expression is run, the ignore case flag is included so that capitals and lowercase are matched automatically. However, this does not work for accented characters (such as É and Ü), which I will mention some more about later.

In order to test for plurals and other suffixes, we add an optional ending to the pattern. This looks something like ((s|ing|er)?) and would test for [word], [word]s, [word]ing, and [word]er. We can also try and look for [word]y but first we must make sure that only y can be appended on. In some cases, usually when the last character is a certain letter (b,d,f,g,l,m,n,p,r,t,v,z)  and the character before that is a vowel, we duplicate that last letter. So “shit” becomes “shitty” instead of just “shity.” Of course, this is a very broad rule and won’t work in every case, which again is why there is a person moderating. Additionally, in rare cases by adding on the suffix will create a valid word. This is where that third list comes into play, were we can make sure we don’t match “craps” for crap.

Once our word list is build, it’s simply running the different RegEx patterns on the given text. Even with large bodies of text, this is still relatively fast, although I’m sure it would be tons faster on the server side. However, I don’t think there would be many cases where users would be submitting huge paragraphs of text. The class I wrote has two options: a normal validate function and a quickValidate function. The quick one simply keeps running patterns, and if it gets a match it terminates and returns true. The other validate function runs though each word and keeps track of the matches. It then returns a result object, that contains the validation status (true or false), as well as three arrays: a list of matches as they appear in the submitted text, the index start and end values for those matches, and a list of matches as defined in the filter list. This is useful if you need to do any syntax highlighting or anything other than just simply validating.

One issue I ran into building my demo was involving accented characters. Whenever there was a non-regularly used character, it would throw off the index position of that character. It seems for one reason or another, certain characters are considered to take up more than one index position according to RegEx. For instance, the em dash character is interpreted to take up three index positions. So if there is an em dash at index position of 5, and the letter “a” at 6, RegEx will think that the letter “a” is at index position of  8 instead. The TextField class does not think that these characters take up more positions than they do, so as in the previous example, if you tried to highlight the  letter a based on the returned index position, it would be three characters off (or if it were at the end it would throw an error). One workaround is right before validation run another RegEx that matches all these accented characters and replaces them with similar looking letters or just with an asterisk or something else – this is just for running the RegEx test, as the displayed text is left alone. This allows you to at least get the correct index values, and also will match É for E since the ignore case flag doesn’t ignore accents.

I created a demo that shows the matching though highlighting, as well as lets you edit the word lists. Keep in mind, this probably could have more uses than just for profanity.

Quick Example:

(Either JavaScript is not active or you are using an old version of Adobe Flash Player. Please install the newest Flash Player.)

Did a quick test this weekend. I wanted to see if you have a player 9 swf and load it into a player 10 swf, if there could be any issues. Mainly, I wanted to see if there could be problems with the new 3D properties in player 10. So I just created  a new player 9 swf that had some five3D elements in it, so essentially I had an extension of the Sprite class with properties such as z, rotationX, rotationY, and rotationZ. Since player 10’s Sprite class contains these properties nativity,  my theory was that importing the player 9 swf would throw errors because now those 3D properties of the five3D elements would not be properly overridden. This is exactly what happened.  Good to keep in mind in case you ever run into a situation where you might have to load an older flash 9 swf into a newer flash 10 shell.

I don’t know how this escaped me for so long, but the other day I realized that Flash doesn’t understand HTML character entity reference codes. Codes other than & and a few others won’t render out to their appropriate symbol. So for example, if I need to display a special character in an HTML page, I would normally use the &+code+; in order to display it. So “copyright ©2009 eric decker” would be “copyright © 2009 eric decker”. So it would make sense that if I bring in this copy into flash, set it to a textfield with the set htmlText method, that it would have the same result. However, it doesn’t. &copy will read exactly as &copy.  Flash does, however, understand numeric reference codes, which look like ©. Personally, I find the name-based entity codes easier to use and seem like they are more widely used (although I could be wrong about that, I’m not a HTML guy).

Anyways, determined to be able to use © instead of the more ambiguous © I wrote a class that is able to understand the entity codes. The reasons for this were 1. as I just mentioned, the name based codes are easier to read and 2. sometime you’re not in charge of your source copy, and it might contain entity based symbols instead of numeric and of course 3. developers are a little neurotic like that and sometimes ‘do it for the sake of doing it.’ (That could be  a T-Shirt.) I was able to write a pretty simple converter using a regular expression and a lookup dictionary. You can download the class or view it as text.

The class contains one accessible static function that you pass your raw text to and it returns the new formatted text: myTextField.htmlText = HTMLTextUtils.formatHTMLTokens(myString);  Better yet, if you have a custom TextField class that you’ve extended, you can override the set htmlText function to have it always use this function.

Grant Skinner’s RegExr is a huge help when doing anything with RegEx.

So over the last week or so, I restructured my 3D engine. Nothing that would be visibly noticeable, but I feel better about where it is at right now. What I like about it, though, is that it is still not what would be probably considered “the right way” to do it. The reason I am happy about this is that it makes it my own, and I understand exactly why and how it works. Once I have a really solid understanding, then I can move towards doing it the “correct way,” but for now, my little hacked together experiment makes me happy.

One of the changes I did make is to the shaders. I added a PseudoGouraudShader, which uses gradients to draw each polygon face to try and smooth out the hard lines created by using a flat shader. I doubt this method would ever produce 100% accurate results that are identical to a gouraud shader, but it makes for interesting results. The method I am using is pretty slow, as it needs to draw each face four times, three of them being gradient fills. Each vertex’s normal is calculated from the sum of all the faces it is a part of, and that brightness value is used to draw a radial gradient on that face, the center point being the vertex. (Two examples: RGB example and dot example, the circle areas are ‘painted’ on via gradient fill) This can produce some fun effects too when played with.

I had an example of the PseudoGouraudShader last week, but it was off, it looked like alternating faces had different brightness values. Just playing around last night I found the issue, it lies with how I was calculating my normal.  When getting the cross product of the vectors, it is better to normalize the result instead of normalizing the vector before. The example below shows the two results and code snippets:

Still not perfect, but I’m not sure how close I can get using this method. I think there might still be some slight errors in the core math, which could be producing the results that look similar to the image on the left (although less intense). You can check out the live demo here.

On a last note, calculating normals and centroid over and over again for rendering can get pretty intense. In the case like this, the same normal gets calculated multiple times a render even though it hasn’t changed (because each vertex needs to get the normal of each face it is connect to). In order to boost speed, I created a caching system. When the normal is calculated, it saves it to a “stale normal” and updates an object (which I called the cached face) that tracks the position of each vertex in the face. Then, the next time the normal is called, if that cached face is not different then the real one, it just uses the stale normal instead of re-calculating.