(Please open the article to see the flash file or player.)

There are more images on flickr.

Initial Idea
The idea for this project originated about six months ago during the summer. I was thinking of things I could do with Teleo devices, and one thing I wanted to explore was using a guitar. I first wanted to attach a vibration sensor to the strings on an acoustic guitar and make a ‘Folk Guitar Hero’ game in Flash. That would be too awkward too, with all those wires hanging off of the guitar. So the next step was just to hook an electric guitar up to a tuner, and do some rewiring to the tuner so it can be plugged into a Teleo device. The tuner will output a value, which would be the note or pitch, and then Flash could interpret that to be whatever it needed to be. However, after getting a tuner I soon realized a problem with this – the tuner can only recognize one string at a time. So an open strum would confuse it, or even if two notes were played separate but back-to-back, enough time has to pass for it to ‘clear out.’ The result would be a pretty lame game.
So, as it usually happens with me, sometime the most obvious solution comes later. The electric guitar has an output to the amp, and the amp has an output to headphones, so why not just get a male-to-male end headphone cable and plug it into the microphone line-in on the computer? So that’s what I ended up doing. This is the setup I’m using: The electric guitar cable plugs into a Y splitter. One end plugs into an amp, so that the guitar can actually be heard. The other end of the splitter plugs into a headphone amplifier. Using a ¼ to 1/8 adapter, a male-to-male end headphone cable is plugged into the line-in port on a computer. And that’s all there is too it, the electric tuner doesn’t need to be used at all, although it does help me tune is since I actually don’t play the guitar ¬– or any instrument at all.

Gathering the Data
The next step was to take whatever the computer takes in and turn that into some sort of note. Flash doesn’t really have anything for real time audio analysis, even with the extended capabilities with Action Script 3. So the next logical step was Processing. “Processing is an open source programming language and environment for people who want to program images, animation, and interactions. It is used by students, artists, designers, researchers, and hobbyists for learning, prototyping, and production .” In order to read the live input sound data, I had to use the external library Sonia. So the way to analyze sound in Sonia is by using and FFT array. The Fast Fourier Transform array is a series of bin samples of frequency ranges and their levels. What I needed is the pitch, so basically pitch detection. I did a little research and apparently many people want to do pitch detection, but unfortunately it hasn’t been done yet with Java. It seems that in order to do pitch detection you need to do something a little more serious, like C++. I looked up other people doing note detection from instruments, and it looks like they are all computer science majors working on their thesis, so it might be a little over my head. I looked for ways to convert the FFT to a pitch or try to get some harmonics or anything from it, but I’m a designer who’s maybe decent in math, I took pre calc in high school but really don’t remember much from it. But I kept finding equations that all looked like this:

The best way I can describe this is as “it looks like the quadratic formula exploded .” So this pretty much killed my idea for a Guitar Hero game with a real guitar.
The best I could do with the FFT data is look at the patterns it makes as certain notes are played. When a note is played, certain frequencies respond much stronger than others, and when graphed out these form peaks. By looking at just these peaks we can kind of determine what note is played. And since the guitar is plugged directly into the input, there is reduced background noise to worry about. What I ended up creating is a program that looks for certain peaks, and when all those peaks show up it determines that a certain note is played. I call this pseudo-note detection. It’s not really authentic because it doesn’t work half of the time, and it gets confused – a lot. This is because a many notes share similar peaks, so when whole bunch get played the program see more peaks, and thus more notes. This makes it way too inaccurate for a game where a mistake on the game’s part would result in the player becoming very annoyed. But for making generative art, it is actually kind of cool because you get more data to visualize, and plus you can see, in a way, how certain notes are similar (similar in FFT terms, I guess).

Analyzing the Spectrum
I wrote a special program to first capture all the notes. This is extremely boring. I decided for each string I would want the note of the open string, and then the first nine frets from the head down. So that means that there are ten notes per string for six strings, meaning I had to capture 60 notes. Each note is comprised of an array of peaks from the FFT. When the FFT spectrum is analyzed, the program looks for bins that have a significantly larger value than the rest, and these are then classified as peaks. I played every note and had processing save out a text file that had the bin peaks save to it, and yes, I did that 60 times and again yes, that did get very boring. Here’s an example of what the spectrum looks like for open High E string:

Once I had all the notes captured as an array, I could on to using them to make visuals. All I had to do was when input from the guitar comes in, rip though an if statement of 60 arrays and check each one for matching bins values. This actually runs really smooth with Processing, which is one of its strengths. I bet there are more efficient ways to do this, such as with math, but nothing I could figure out. The spacing between peaks are supposed to theoretically be even, but from what I saw that is extremely theoretical.

Creating Visuals
This was another difficult step, as I wanted to make something interesting and different. The danger I now faced from moving away from the more-solid concept of Folk Guitar Hero, or even just a Guitar Hero rip-off, was that I had no design direction. That led me to flail around for some time. I first started off with just radiating circles over notes and then on a grid and that didn’t really seem all too interesting. I started experimenting more, again with a grid, with a paint analogy. The grid layout seemed too restrictive, where I had the string value on the x-axis, and the fret value on the y-axis, and the strength (‘volume’) of the note-determining radius. I wanted a little more abstract, so I decided to look into 3D. I made a neat looking 3D visual using the mouse with JAVA3D or OPENGL renderer in Processing. However, the Sonia class does not like to play with others, and even on ‘super computers’ the program ran incredible slow. My guess is that Sonia only can run with the JAVA2D engine. So I built out a pseudo 3D visual thing in 2D space, and had all sort of properties to play with. The string could effect overall color, the fret from one would change rotation speed, one string made starbursts, etc. But then I had just created another visualization like anyone can make in 10 minutes in win amp. I needed structure, an idea, a concept. Luckily I had a critique a few weeks before the project was due.

New Direction
During a class critique, one of my classmates Nate Wolf suggested that I have it build out and not erase or fade away, so things pile up over time, or even make it expand. All of this time I had imagined that the final piece would be a performance piece that would be recorded. I did a print piece previously this quarter, and I liked it, it was nice to have a tangible piece for once and not just a website or animation that only exists on screen as a series of interpreted zeros and ones. So I decided that a print piece would be cool to do, and that I wanted it to scroll out, so this would be a big print, something like maybe one foot tall by maybe 50 feet long or something utterly ridiculous. Of course, paying for a print that size would be out of my price range. With printing being $4 per square foot, a print that size would cost over $200. So I guess I would make a “theoretical print” instead.
I figured that what I was doing, even if it was sloppy, was mapping the notes of the guitar. It would generate a representation of the make-up of the song. Basically like the genetic make up of it. That struck me – the genetic structure of a song. I looked up some images of DNA patterns and found my inspiration. I used blurry squares overlapping each other to simulate DNA patterns. Rather than making running a blur filter every time a new note is drawn, I had the program pre-draw all the notes and then when it comes time to make one it just slaps that image down. This little cheat is necessary since doing filters when Sonia is running and getting real-time results is a bit iffy. I also made the value by which the program determines a peak in the FFT by dynamic, so if I decrease that value it lets more bins be seen as peaks, and thus more notes. This makes a more muddy response, an extreme decline in accuracy, but greatly enhances the visual quality. I control this when capturing by using the up and down arrow keys. I also have done a variation using circles instead, which when asked 50% or people preferred circles over DNA, but at the same time the other half liked the DNA one better. So I figured the tiebreaker came up to me, and since this was ‘Generative Art’ and not ‘Generative Design’ that I can make it look like however I want. So I picked the DNA style.

example of a DNA pattern (source)

Creating the Render Engine
Of course, I was not finished yet. I had the program saving out screen shots every 100 frames or so, and the Photoshop could run the ‘Photomerge’ script and stitch them all together to make my panoramic-like image. (I also draw guide numbers below the image, sort of like visual meta data to help Photomerge align everything correctly). But, to make a print from this would be either really tiny or really shitty because I was only saving out a 1000 x 400 pixel image. That could make a really crummy 72dpi 13.8” x 5.5” print, or a high quality tiny 3.3” x 1.3” microprint. In order to achieve something large and with high resolution, the image needed to be atleast1800 pixels tall. Since the images are stitched together later the width of them is not as important, as long as all the data is drawn correctly. And when trying to run that real-time, drawing out an image that big presents two problems. One, you can’t actually see the entire image, especially not on a 640 x 480 projector and two, it’s not going to run smooth, you’d be lucky with getting over 1fps.
So my solution was to separate the performance part from the print part. The performance part will show on screen whatever is being played in low resolution, so it runs faster. As it does this, it saves out all the note data to an object array. Once the person is done, the collected data is written to an XML file and created in a new folder. To make it easier, I made up my own auto increment system that checks and writes to a text file the names of the folders created. The XML data is structured so that each frame has its own node, even if it is empty. Inside the frame node are note nodes, and each one of those contains the string, the fret, and the value, which is more or less volume. There’s also a meta section I included with the date, time, and information the user can input so if I were to show this during class I could later build someone’s visualization for them.
Now I can render the session out whenever I please. I open up my render program, declare a few variables such as what session folder to use, what color scheme, and height in inches and the resolution. It then parses though the XML file and builds it out just like it would from the guitar, except now I can scale it larger. Once it is done, I throw it into Photoshop and trim it up, and then it will be off to the printers.

Technicalities
There is a lot of backend stuff here. There is an issue right now with Sonia v0.91 where it won’t work in any browsers, so right now I cannot efficiently deploy this. I made a few external classes to help my program run more efficiently, or at least keep it a little more organized (which if you saw it, it is still all over the place). One useful tool was making a color manager that has arrays of color schemes. I just need to set it to a color scheme, the ColorManager will remember what it is set to, and then I can just call colors from it. So, for example, I can just call background = Colors.getColor(0) and that will get my background color. It kind of works like CSS in a way, where I just tell it to use a different style sheet.
One thing I wanted is to be able to set which session folder to draw from in the render engine without having to edit the .pde file. I couldn’t find anything about a input text field, so I had to make my own. It’s really simple, click on it to set focus to it, and then what you type just adds to it’s own string which is displayed in the box and updated every time it changes. The only issue here is that I only pass key values to it, not keyCode values such as SHIFT or ENTER, other wise they show up as question marks. I can also use the delete key to remove the last character from the sting. But for some reason, on my PowerBook at least, the keyCode for the delete key is neither one of Processing constants of DELETE or BACKSPACE. I found out later that it’s 8, but I also put the left arrow key in there just in case. I had the same issue for return, both keyCodes RETURN and ENTER did not work, but a key value of 10 did.

Resources
Sonia Processing Library: sonia.pitaru.com
Processing 1.0 Beta: processing.org
FFT information: http://en.wikipedia.org/wiki/Fast_Fourier_transform

[Download Guitar Genetics source code]
Be forewarned, I wrote this for myself so organization of the code… yeah…

After talking with colleagues, I’ve decided that I want a print to be the final result of my project involving using an electric guitar as an input device. I like the idea of something resembling sheet music, but also showing how the music is made up. So I looked at DNA pattern images, and based it off of that.

I just finished building a prototype, and the next step is hooking it up to the guitar part. Makes me kind of nervous, because I fear it might run too slow once integrated with the sonia library. This image is just based off of the mouse, but for the final one there will be six colors, one for each string, and the vertical placement will be determined by fret.

(Please open the article to see the flash file or player.)

(Please open the article to see the flash file or player.)

(Please open the article to see the flash file or player.)

(Please open the article to see the flash file or player.)

(Please open the article to see the flash file or player.)

Graeme putting up images:

Old Man Arena:

(more)

Brendon showing up late:

Close up of one of the walls:

Sarah hangs up prints:

Old Man Arena and the VE class:

Here are a handful of some images “I” have made with my ImageMapper program. I figured that I would show what the photo I used was to generate the image. Kind of takes all the magic away from it, seeing them side by side. Again, there are more on flickr.

Joshua Tree National Park

Clock from digital display on a stove

Charcoal grill fire

Portion of a painting by Erin

Muscle shells

Turtle eating a hot dog off a stick

Wrist watch

Merlot, my parents’ African Grey parrot.

Here is a demo of how it works. You can click and drag to control the sample point, and the up and down arrows control the range of the random while you’re clicking and dragging. Below are some of my favorites that have come out:

There are more examples in my flickr gallery.