June 12, 2024No Comments

Holograms and the Ideal Display

This short essay is also included in my 2024 update of my Survey of Alternative Displays, but I figured it was worth highlighting on my site as well.

The word hologram is probably the ultimate misleading term in the experience design and AV industry. True holograms, by definition, are recordings of interference patterns/wavefronts - typically done as a photographic film process.

In the language of digital display tech - everything from light field, to transparent, to autostereoscopic, to persistence of vision is branded as a "hologram" to suggest something more than 2D, even if that's (usually) all it is. However, it can be helpful to look at some of the characteristics of what tends to be the most desired elements of a futuristic display. Is there a way we can look at some commonalities around what exists as a "hologram" display today and what might be needed to get it to a next stage?

The characteristics of a display marketed as (heavy air quotes) "holographic" often include:

  • Their image seems to float against a real-world background utilizing parallax to trick us into a sense of depth (Pepper's ghost, persistence of vision, projection on scrim, transparent LCD/OLED, etc)
  • Some sense of stereoscopic depth/alternate views of an object to tickle the brain into thinking it is a real object (autostereoscopic, light field displays)
  • Some combination of the above (swept volume displays, volumetric projection)

However, I often get the sense that to pass as a "real hologram" or to become an ideal holographic display, the tech needs to take things much further into science fiction. While current displays are cool, many fall well short of meeting some of these common requests or needs below:

  • The ability to show a full black to white image in any lighting environment, no matter how dark or bright the ambient environment is.
  • Be able to scale and be viewable from the smallest detail to a very large size (changes in depth cues would be a big thing to address in that scale).
  • Be variably transparent to the environment, or not necessarily contained within a rectangular frame or require looking into the display (like seeing a floating object from a full 180 or even 360 off the surface of the display).
  • Render things from diffuse points of light to crisp details.
  • React to their environment (more on this below).

The Promise of a Hologram

Holograms as a pop culture concept seem to hold some sort of power or promise of a certain type of experience, but I've always been curious what the actual end goal of using holographic displays might be.

The drive to market a digital display as a hologram seems to stem from a couple of factors. First, there's the "cool factor" of a novel display that enhances or spices up otherwise less exciting content. People might also want something more immersive or mysterious than a glowing rectangle on a wall, and selling something as a hologram feels like selling a previously unattainable goal - like cracking cold fusion, new battery technology or anti-gravity. The future is now, etc etc.

Second, there is an element of holding a sort of promise of utility and function. Some research suggests that holograms can streamline information processing. Our brains evolved to see and process fully 3D or 4D content much faster, and this is part of the reason why medical and defense industries look towards these displays for an enhanced ability to see and process complex scenes. Looking at a flat 2D map takes longer to parse and understand, and may contain less critical information than a fully 3D map.

Moving one step further into the future - I think that even we were to achieve the famous Princess Leia and "Minority Report" holograms, some people wouldn't consider that "good enough". Someone would want something more crisp, more colorful, more opaque, larger - the arms race for 2D displays certainly went that route, and even head mounted/AR displays strive for a similar goal. To be fair, to make most of the above happen would essentially require many new levels of understanding about not only our physical reality, but also our ability to manipulate and steer photons in mid-air. Additionally, if with the floating images, we would probably need to add physical sensations as well - touch, texture, heat, pressure, etc.

What does the endgame of a "perfect holographic display" seem to be? Is it just a desire to essentially manipulate visual reality itself? Let's imagine we had truly holographic displays tomorrow that could manipulate realistic visual reality at room scale or larger (some AR headsets are already working towards this approach). As dystopian or sci-fi tin-foil-hat as it sounds, that sounds like borderline dangerous technology for anyone to have in their possession - there are already concerns about AR layers disrupting the physical, but real reality being subject to a shift sounds wild. There are obviously utopian takes as well, but it would need some serious reckoning about the appropriate way to use the tech, much like our current discussions around ethical AI usage. To achieve anything close to that level of realism still feels decades off, but it can be fun/terrifying to think about what this could all be headed towards. 

Moving from Output to Input

While a lot of what I've covered in the alternative displays survey focuses a lot on displays as an output mechanism, I think to truly get to the realism of a promised hologram there is still a bit of nascent research to be done around the input to displays and their content. When I say inputs, I don't mean just HDMI cables, and physical sensors and touchscreens and other interactives either, but rather the idea of the capture of the light field surrounding the display.

Most displays these days are essentially blind to the world around them. Some current displays and devices have built in tricks for sensing ambient light and adjusting brightness or color, but that's about it for the popularized adjustment capabilities. Some displays today (Like Sony Spatial Reality) even rely on head or eye tracking to create their 3D illusion. If we got to the point of realistic holograms but still had to light the 3D scenes in the same way we do today, I think everyone would quickly realize that the next hurdle to creating true realism is capturing the world around the display. 

The hologram should, in an ideal scenario, react to changes in the environmental lighting conditions just as a regular physical object would. I should be able to shine a flashlight on it, cast a shadow onto it, or bring it inside or outside, see myself in the reflection of shiny objects, see through transparencies and block opacity. All of this would all need to happen with extremely low latency. 

Right now we have displays we send pixels to, and some may have sensors, but a full 360º lightfield sensor that is integrated into a lightfield display would make things that much more impressive. Showing a nice shiny rendered sphere floating in space that reacted to all real-world lighting cues would feel next level. XR Studios understand a bit of this technique in terms of using environmental LED to cast 3D scene light onto the IRL subjects being filmed, but turning it all inward feels a bit far out with our current understanding. There are also examples out there of things like using a fully spatially tracked flashlight, and using that flashlight to cast light or shadow onto a 3D scene - its a clever trick and looks really cool, but obviously still has its limitations. AR headsets are getting there in terms of reality capture as well and I'm sure AI models will also help in some of this. While there has been research on light field cameras like the Lytro from years past, I haven't come across a ton of projects that attempt to incorporate light field capture AND light field output.

Thanks for reading!

May 20, 20242 Comments

Creative Technology Ecosystems

This is a sort of continuation on my essays on creative tech and organizational structures, but could easily be considered its own thing.

My previous essays on this topic focused more on projects and clients and their impact on the working process. This essay discusses the broader ecosystems connected to the tools used in creative technology and experience design, tools that the industry relies on to make their work. Understanding the origins of these tools can provide valuable insights for teams working on experience design and creative tech projects.

Consider the chart below which is a sort of high level map of how I think about the “sources” of the tools used by creative technologists.

When we compare creative technology and experience design to other industries, it becomes clear that it is relatively small. How many experience design companies focus on creative technology as a core competency? How many projects to they handle annually and what are their revenues and number of employees? Despite some projects gaining significant media attention, the financial impact of creative technology remains modest compared to other sectors. Consequently, many tools used in creative tech are adapted from technologies developed for much larger industries.

Why is this important? Consider a technology like the well known depth camera, the Microsoft Kinect. The Kinect is old and essentially deprecated, but stick with me. This tool was launched in late November 2010 for the Xbox 360, a gaming console for consumers. The tech has origins from Israel's PrimeSense, and then Microsoft acquired them and moved the tech towards gaming applications. This research and development took years, and likely hundreds of millions of dollars. The original Kinect sold very well to consumers (over 8 million units. Additionally, despite the fact that the tool wasn't explicitly made for the creative technology communities, the Kinect also eventually began to get a foothold as a new sensing technology for use in interactive digital experiences. It was everywhere for a while, and we can still feel echoes of the "wave your arms at a wall of particles" projects popularized by the Kinect even today.

Now let’s consider a hypothetical case of direct impact of the creative technology community actually buying that tech.

Let’s (generously) say there are 500 companies in the US doing creative technology projects in a given year. Of those companies, maybe Kinect projects are really hot right now, and they have 3 projects each that year that all use 5 Kinect's per project (so 7500 Kinects across all companies). At $200 a pop ($400 for the Azure Kinect), that brings the grand total of Microsoft's take to something like $1.5 Million for 7500 Kinects - an optimistic estimate and a slim percentage of the total 8 million units sold. While these projects might enhance the Kinect's visibility and perceived value, $1.5 Million is insignificant compared to the consumer market and other industries. On top of that, many of these large tech companies occasionally take on these projects not as a profit-making venture, but almost more as an intensive abstract marketing effort - make a "cool" technology that can be used in a really "cool" project and attach Microsoft, Intel, Google, or Apple onto it. Consumers may not go out and buy a Realsense camera, but they might remember that Intel does "cool" stuff like that. See also, the concept of a loss leader (the Kinect 2 was reportedly sold at a loss).

So if all of these technologies aren't being made for "us", the creative technologists - who are they for? Well, there is the consumer market, there is the defense and weapons industry (remember how Hololens got a $22B contract with the military?), there is the medical industry (see Google Glass Enterprise), and a lot of other high profile and well funded industries. And I'm mostly talking about the high profile sensor technologies that a lot of us use - there is a whole other side to this in large scale AV technology like LED walls, computers, microcontrollers, etc etc. Supply and demand still mostly rules here. Below is just a partial list of tools or frameworks and either their original “funder” or what I would assume is a primary industry for that piece of tech:

  • Defense/Military
    • Depth Cameras
    • Infrared and Heat Sensing
    • Headset displays for VR and AR
    • GPS
  • Retail:
    • RFID/NFC
    • Touchscreens
    • "Digital Signage"
  • Automotive and Manufacturing
    • Computer vision
    • Various sensors (pressure, distance, etc)
    • Robotics
    • LIDAR
  • Entertainment (professional and consumer)
    • Video and display standards
    • Cameras, connectors and lenses
    • Audio equipment and standards
    • Displays, Projectors and LED
  • Medical
    • Heart rate sensing

*Caveat that the above points are based on observations and not detailed research

Since these technologies aren't usually designed specifically for creative technologists, the discipline must get really good at adaptive reuse. A key skill for a generalist creative tech is to be able to see through the noise of the tech landscape and make recommendations about what will and won’t work - a lighthouse in the fog. We’re sifting through other industries to find tools that enable new creative capabilities and methods of expression. However, to my point above, creative tech can be a small voice to direct change in those technologies. 

The idea of adaptive reuse gets at the heart as to why working on creative technology projects can feel either exciting or truly challenging. We're often using and bolting on tools that weren't designed for the purpose we're using them for. They are often tools that were profitable enough to be produced at a high volume for a broad set of purposes (or one specific case). This is certainly nothing new to the world of creativity or working in general, but it does answer some important questions that come up over and over.

Creative Tech people may know what we want or need in a sensor, or a depth camera (higher frame rate! high resolution!), a computer, a software tool, but there may never be something that gets released that is just the right fit - so we make it work. For people reading this that might be more non-technical, I think a key takeaway is allowing for some grace when things don’t work as intended.

So with all of the above and the challenges of a broader ecosystem, how can organizations be more responsive and less reactive to tech trends? What sorts of technologies can the creative tech space have greater influence over or is the only option to try and make your own product and hope you can recoup the costs? This topic is a meaty one meant for its own separate writeup, but at a high level I think that things like creating room and curiosity for prototypes and research both inside and outside of projects can be a huge help. Having an awareness and sensitivity to the forces at play outside of the creative technology and experience design space can be a crucial step to identifying paths forward.

May 8, 20242 Comments

Creative Technology and Organizational Structures – Part 3

Link to previous parts: Part 1 and Part 2

Project Structure within the Company

Let's zoom out further and look at the entire company and other elements involved in actual projects. 

Here we're not only considering the departments and how they work together in a single company, but how the company works with the client and other vendors involved in bringing a project to life. Large projects inevitably involve large teams, and decisions made by one team can significantly impact others. The purpose of this section is to look at even larger systems at play, providing insight into the level of influence the Creative Technology team can wield in shaping the outcome of a project. Creative Technology is not merely an isolated group being passed tasks to execute; it is an integral part of a larger team of team, each driven by its own motivations.

Below is a partial list of some of those factors that might drive different teams but often can be diametrically opposed (see also: the good/fast/cheap triangle) :

  • User experience
    • Ensuring users have a positive and meaningful interaction or experience.
  • User engagement/earned media/and “buzz”
    • Maximizing reach, shares, and visibility of the project.
  • Representing “The Brand”
    • Ensuring the final project aligns with the tone and quality expected by and of the client.
  • Creative output
    • Striving for a high degree of polish and artistry, aiming for innovation.
  • Team makeup/staffing/morale
    • Assessing capabilities and pushing skill boundaries of a team.
  • Use of technology
    • Prioritizing seamless, engaging, and reliable technology.
  • Timeline
    • “It’s Valentine’s day. This project needs to be live for SXSW in March”
  • Budget
    • Managing financing constraints - often an immovable wall that influences most of the factors above

The push and pull between teams to optimize their goals above are what make projects challenging and fantastic at the same time. While all teams recognize the importance of these factors to some extent, their prioritization of those factors may vary. The tensions are also not often diametrically opposed foes - an increase in one doesn’t often equal a decrease in the other. A superior creative output can come for “free” and feel effortless with the right talent and vision. These can be contrasted with expensive, big budget projects that fail based on poorly conceptualized goals. A longer timeline and bigger budget doesn’t always mean the use of technology will be better or more magical for the user. It’s a complex, multi-dimensional problem, and I would be wary of a company that says they have it all figured out for every project. Internal cohesion and balance within a team are what help stabilize against the external pressures from clients, vendors, and market forces.

That said - I think it can be helpful to look at these forces not only within the company as in previous parts of this essay, but also in relation to external influences. In the next couple sections, I will show some charts that illustrate how a company and department might find itself within a project. These charts are drastically oversimplified models - we must acknowledge that each project is drastically different. However, they do offer some insight into the diverse decision-making processes within departments and companies, particularly in the context of short and long-term projects.

Agency and Production Company

To begin, we’ll look at a typical structure I’ve often encountered in fast-paced projects, typically associated with large or medium sized events such as booths at CES or pop-ups at SXSW (or something along those lines). It’s important to note that while this structure offers a glimpse into a potential approach, it’s not prescriptive of how things always happen - but just that they could.

In the first chart below, we focus on a scenario where a client or agency collaborates directly with the company or vendor organizing the event. That client may then hire or already work with an agency, and that agency would then hire a production company to execute the idea. For instance, you could frame this as: an automotive brand wants to showcase their new technology at CES using an interactive installation - they use their agency of record to strategize the installation, and the agency hires a production company to actually create it. The automotive brand is just a vendor at CES and is beholden to things like venue and event rules, but still has some sway - they could be considered a peer. Any vendor that the automative brand hires must also abide by these venue and event rules and restrictions ( for example: load in times, access times, power, internet, etc.)

It is worth noting that sometimes the event organizer or the venue itself will also have an established technology integrator that is responsible for installing all of the AV technology for an events space (or there may be a union involved). In those cases, the creative technology agency or production company will have to work directly with that specific AV integrator to make sure that everything is in place to make the event happen. Notably, the power dynamics are a little different if working as collaborative vendors versus a production company that hires and pays their own vendor directly.

Another aspect to consider regarding the chart above is the indicate layers of approval and acceptance within the “Client” category. While communication often occurs directly with client teams that may be involved primarily with marketing/strategy/creative, larger projects may require approvals from higher levels of executives that aren’t as close to the project. That decision making structure can have a lot of trickle down effects for all the parties below if big change or a wrench is thrown towards the end of the project that requires a lot of adjusting.

This communication dynamic is relevant for both event and “permanent” based work. It is crucial to consider how project status is communicated across various levels, the frequency of communication (and reviews), and who oversees the communication flow. Identifying who needs to be informed and involved at each stage is key to maintaining project balance.

Let’s consider a common path of: client makes a request -> sends it to account management -> account management asks production or tech if its possible -> production and tech teams report back on budget and timeline implications to address the change -> change is made (or not made). Sometimes a change is minor and considered within the realm of feasibility/teamwork/overall project vision, and sometimes it is major enough to trigger a considerable adjustment to the project (and budget). Sometimes a request is so obviously outlandish (“Can we have 10 of these instead of 1?”) that it can just stop before flying around to all parties, and sometimes it is less obvious and needs to be vetted (“What visual tweaks can we change on site?”).

Sometimes having more links in the chain (in the case above with the agency between production company and client) can sometimes complicate communication channels. However, experienced folks can recognize this and strive to communicate as clearly as possible so messages get passed along to the right people at the right time. 

Next we’ll look at both a permanent installation/long timeline project model and one that has a client working directly with a company making the final product.

Direct to Client

Let’s explore another version of this for permanent installations and/or direct to client work. In this model, there is no intermediary agency between the production and the paying customer. To be clear, this removal of a middleman doesn’t mean that communication problems disappear; rather, certain communication challenges just morph into different considerations. This communication might be conceptually the same as the above, but that role of internal guidance of the client is just folded into the project and creative management on the production company’s side.

In this scenario, the agency doing the creative technology work is hired as a sort of "full service" shop, handling all of the aspects of bringing this project to life for the client. This can include project oversight, creative conceptualization, design, fabrication, development, installation, and maintenance, etc. For permanent installations, additional parties may be involved such as the client’s architect of record, the site’s general contractor, or even a content production team responsible for creating visual content.

This direct-to-client configuration is often preferred or seen more often in longer term projects which are typically more extensive (and expensive). Such project may span years and undergo team changes over time. Additionally, there is usually more time allotted for things like discovery phases, design, and production compared to event based projects. These extended phases allow the smaller team more time to deliberate on specific decisions and ensure they are the right ones before moving onto the next large phase.

Questions to consider at this level:

  • How do things change as projects scale up or down? How many cooks do you need to make a meal?
  • How many reviews or check-ins between teams are too many or too few?
  • What does the client's team look like? Do they have a project manager on their side helping to steer the ship or are there multiple stakeholders?
  • What is the value of an agency when working on large projects? There may always be a knowledge gap between the client's knowledge and interests and the expertise of the company doing the creative technology work. The layer of translation has to occur somewhere.
    • Additionally, for very large branding campaigns that utilize things like print+websites+video content+social media AND experiences, it may not always be feasible or economical for the creative technology company to plan AND build all of that if they also want to focus on "just" creative technology.
  • How can you better facilitate communication between a client's marketing team and their technology team?
    • Marketing teams can occasionally propose using their own product in a way that it is not designed to be used. The idea they have may not be vetted internally before beginning to work with an external creative technology team. Can you provide a common roadmap for the client on how to quickly arrive at a productive solution?
  • Is the value your organization brings to a project more about your creative strengths and ability to problem solve, or is it about some kind of secret sauce cutting edge technology that “only you” know how to wield? What are the costs of always staying at the cutting edge of technology versus working with reliable, tried and true methods?

That wraps up part 3 and generally all I will be covering on company and project structures. In the next part “Creative Technology Ecosystems”, we’ll kind of move on to something that could be it’s own series but in a way feels like part of the continuum of discussing the structures at play with this work. The intent in that section will be to look at where the technologies we use come from and how that can influence our working process.

August 25, 2020No Comments

18 years of Bright Eyes Play Count Data

I've been a huge fan of the band Bright Eyes since my junior year of high school in about 2002 when they released "Lifted...".

Bright Eyes just released their latest album "Down In The Weeds Where The World Once Was" last week, their first studio release since 2011.

Somehow, I've managed to retain my iTunes play count data since 2002 up until this year. I just sat down for an hour to visualize some of it, and I thought I'd share.

Note that I'm missing several important EP's, Splits and other compilation albums, or I just have insufficient data for them - sorry about that...

Anyway - enjoy the peek into some of my favorite tracks of the last 18 years!

PDF of everything available below the last image

Bright Eyes Playcount PDF (2373 downloads )