Completely done

Stand-alone to social: Applications are no longer solitary experiences (think Word or Excel). Software no longer serves just a single purpose, it is also a collaboration space around that function.

Pretty Much done

Desktop centered to mobile centered, with tablets as something of a grey area there. I think it’s a given now that most software targets mobile platforms first and foremost. In fact, I think I’d jump through hoops to figure out a mobile interface now before resorting to one that requires a fat client. Even a lot of heavy HTML5 mobile sites are too slow, but the devices are catching up.

Premise-based to cloud-based: Most software has at least a cloud component now. Most mobile apps I install on my phone have their guts hosted in the cloud, even if the UI runs natively on the phone itself. Enterprises still lag here because of security concerns, but I think they will eventually cave. I actually predict a sort of hybrid enterprise, simply because there are so many legacy apps that can’t be easily converted.

Still In Progress

Tailored information: Software experiences are being tailored to the user’s tastes and profiles. Pandora does this extremely well, for example. Information feeds are routinely customized using user-provided information such as social graphs, user profiles, pick lists, etc.

Real-time updates: As large-scale, distributed information systems become accessible to developers (see Twitter), real-time information is becoming a core feature of more and more software. The popularity of native mobile apps makes interface robustness less of an issue, but in any case Websockets are almost ready for prime-time.

Massive amounts of data: As the number of devices connected to the Net balloons along with the number of users, the data produced every day grows exponentially faster. That data is slowly be harnessed, and the platforms needed to crunch it–such as cloud clusters and Big Data engines–are finally maturing.

What’s Next

All-purpose to specific: The all purpose development tools of yesterday are slowly being replaced by more complex solutions that provide finer-grained control. For example, relational databases are a kind of all-purpose data storage and lookup tool. NoSQL requires much more attention to detail and planning ahead, as you must finely tune the schema to your application. Relational databases do not require this, the underlying engine can be adapted to whatever it is you need to do. Data visualization libraries are another area where very specialized tools provide greatly improved functionality from the previous all-purpose charting and graphing tools.

Abstract to tactile: user interaction is shifting away from abstracted devices such as keyboards and mice and toward analogies based on the real world. Gestures, body movement and interactive environments are all artifacts of this. Touch-based hardware is the first wave of this and has been incredibly popular, but I can’t wait until I have my mobile Kinect camera allowing me to NOT touch my tiny little phone screen.

Mining the wisdom of crowds: The big data part will finally allow the wisdom of crowds to be harnessed. Or, should I say, the sentiment of crowds–I have my doubts about the theory of crowds being correct, technically. But the sentiment of crowds is definitely useful–supposedly people are making bank mining Twitter for stock predictions etc. And… people do tend to think alike, so it’s relatively easy to predict how people will react to things using all of this data (no matter how much of a special unique individual I like to think I am, there is a cluster of lots of people who are extremely similar).

Manual user input transitioning to automated information discovery: Today’s software requires a lot of user input to divine things like preferences and tastes, this will shift to automated discovery using data provided by public and private information feeds (like Twitter feeds or email histories). Someday, Pandora will be able to create stations for you simply by looking at your feeds.

And here’s a bonus I’ve been thinking about lately:

Legal implications of data and data pedigree: As governments and corporations increasingly use data to automate information discovery which is used to fuel policy and law enforcement decisions, data accuracy and sourcing will be extremely critical. Wrong information could and probably will result in mistaken identities, beauracratic nightmares, erroneous arrests, and could very well be a matter of life and death (thinking of drone attacks on suspected terrorist hideouts, for example). Data pedigree and history will become very important. I suspect there will regulations around this requiring some sort of data verification mechanism or some other nonsense.

Am I wrong about anything? Have I missed anything?

I try to keep an eye on new BI companies and products, and there are some really great new ones out there: QlikView and Palantir come to mind. Technology platforms and the like aside, I feel like there’s one underlying assumption that’s reflected in these projects that I would consider legacy at this point. Or at least old-school, one that’s going to have to change to keep up.

That underlying assumption that’s going to change is that people look for data, instead of data finding people.

Google’s Eric Schmidt once talked about this, saying:

“I actually think most people don’t want Google to answer their questions. They want Google to tell them what they should be doing next.”

This is an easy thing to gloss over, but if you think about how this applies to BI & analytics, nobody is doing this right now. You still have to use a tool to find things that are interesting to you, instead of the tool becoming invisible and delivering interesting things to you.

Pandora is the closest thing I’ve seen to this–I’ve read their patent and it’s very specific to music. Somebody is going to get this right for broader analytics. Soon.

In addition, anyone can make it, it doesn’t get used up, and the industry of using data to create new and valuable things is still so young and ripe for profit-making. In fact I think it’s one of the areas that America needs to focus on if its economy is to recover because for the most part it’s still virgin territory and it’s going to create a lot of economic value. What I really don’t want to see is foreign companies being the first to capitalize on the data as that would suck most of the value out of our economy, just what we don’t need right now.

Data as a commodity is also interesting to data producers, which is potentially just about every business with an IT department. It is a new revenue stream that most businesses don’t even realize they should be considering. Any business that keeps a data warehouse needs to start thinking about if and how it wants to monetize that data. Topics like data anonymization and regulatory restraints need to thought out well in advance (meaning, right now) so that the wheels can get moving. I can tell you that from my experience next to nobody is taking advantage of their data, seeing it as a the backend behind their reports at best and a storage expense worst, instead of a potential revenue center.

In order to monetize data, however, or to mine it for value, you need some type of exchange where buyers and sellers can exchange it. One of the trends I’ve been keeping an eye on is the development of these marketplaces, and there are now several marketplaces offering data as a paid commodity, most coming online within the past year:

Microsoft Azure DataMarket prices its data by transactions per month. The main advantage it has right now is a normalized OData schema which provides a baked-in integration layer which is a huge value-add for developers. However, it doesn’t seem to have a very mature API for content publishers to put data into the marketplace, which is unfortunate.

InfoChimps seems to have the most interesting data sets at this point (depending on your use, obviously) like data mined from the Twitter firehose such as trust and authority ranking, and some really oddball stuff like UFO sighting reports. However it seems to require a different API for each data set which doesn’t allow you to easily integrate multiple data sets.

Factual has a nice API which allows developers to correct data (although I’m not sure if the corrections are shared across developers). However, I wasn’t able to find any API at all for data providers to put data into the marketplace, and again there is no common schema for data meaning that all integration is pushed out to the consumer.

None of these API’s appear to support streaming for real-time applications, which is unfortunate, but I’m hoping that changes as the space matures. The data publisher side really needs some love. Aside from the Microsoft offering they don’t seem to provide any type of help with integration either, which is definitely going to make it harder on consumers of the data as it puts the onus of correctly integrating the data sets on them.

I haven’t been able to find any really good examples of game-changing data in these marketplaces yet–it mostly seems to be cleansed/normalized/mined versions of large public data sets, which is unfortunate (some of the Twitter data sets on InfoChimps being the exception as far as I can tell). It’s going to be interesting to watch this space to see who ends up with the most data sources, the most data publishers, the easiest data integration, and thus the biggest competitive advantage. It would behoove each of these guys to focus on that, I believe. The hardest problem there is going to be lowering the barrier to participation for all types of businesses and create a streamlined process for adding and vetting data sources.

I believe this is going to be a huge, huge, opportunity and that data marketplaces–especially if they offer out-of-the-box integration–are going to make money hand over fist. It’s also going to be an area ripe for mergers and acquisitions. It’s still so early here that it’s like the Wild West, we’ll see if we get the gold rush again.

Anyway, the reason I’m thinking about Yahoo Pipes is that it’s kind of a unique and easy way to think about data compared to traditional data. Instead of thinking about data as something you go out and get, it treats data as something that trickles in and is then categorized, put into the right place, and so on when it arrives. This always struck me as a very elegant way to treat data, rather than as the chunks or flat files that are batched, filtered, treated with rules, and so on, and then sent to the next batch process.

But most of all I like the pipe concept–that data can flow between two points. It’s a very user-friendly idea and it works well in practice (aside from the tragic bugginess of the Pipes product itself). It’s an idea I’ve been exploring a lot lately, and I think two-way pipes are interesting as a concept for “Relationships 2.0″ (I’m sorry for the 2.0, please forgive me).

But most of all I like the idea of a pipe as a wormhole, a way for data to be in two places at the same time. This is just a location in data-space that, when you walk through it, you’re in a completely different place but looking at the same stuff. An odd concept, but think of all of the old-school relationships that covers in one idea. One thing–this pipe, wormhole, whatever–and you get from the stuff your’e looking at here to all of this related stuff over there. In fact, there’s no difference, you’re seeing it all at once.

All of this boils down to thinking about relationships between locations in data-space rather than between entities. If an entity is in one place then it also in another place. If an entity moves into one point in data-space (and by data-space I mean considering each piece of data–a field in a traditional database–as its own dimension, very much like OLAP does) then it pops into existence in the other point as well. Whatever you’re looking at becomes dynamic because wormholes can pop open at any time and a bunch of new information can tumble through.

So say I’m looking at something of interest–a property the bank mispriced, to give you a real-world example. If I want to see more “like that”, I’m going to end up looking at other properties that are similar in some way. Say, for example, it is owned by a particular bank, Countrywide Financial for grins. If that bank is taken over by the FDIC and is acquired by Bank of America, I can do one of two things to make my data right: I can create relationships from all of the properties that were owned by Countrywide to Bank of America, or I can create a pipe–a wormhole–from where Countrywide used to be to where Bank of America is. And then if the merger is squashed for some reason I can undo the whole thing by removing the relationship.

If I’m looking at a mispriced asset that I’d like to find more similar assets, this is huge. By creating this one thing, a two-way pipe, wormhole, whatever, a whole new world of data is exposed to me. The amount of data that is quite instantly visible from the same place (whether I’m looking at a report, a dashboard, an app, whatever) has grown instantly. (What’s cool, too, is that you can expand or contract the wormhole requiring more or less similarity to flow thru it. A geeky way to say that you can easily adjust how similar the stuff is that you’re looking at.)

This sounds like a pretty abstract thing, but it has me kind of excited because this type of merging and un-merging is what data integration is all about. It’s a huge, hairy problem, as you know if you’ve ever been involved in an ETL project, and this seems to solve a lot of that. Now, trying to implement this has not been a walk in the park either, but it’s definitely a nice challenging programming challenge.

Some earlier thoughts about data wormholes located here.

After thinking about this problem for a long time, I feel that the concepts themselves that will be used to think about data on the back end will change significantly. Instead of thinking about things like keys and indexes, we’ll be talking about things like similarity and focus.

Entities, as defined by “some data”, are effectively clusters of data that behave and are treated as one. They can be formed and ripped apart in real-time as data becomes available or is invalidated by better data. Because of this entities are effectively transient. However, instead of disappearing completely their data is simply sucked through a wormhole“>wormhole to the data location of the entity they’re merging into. This allows the data to flow back to the original location in the future if needed.¹

Data Locations are spots in n-dimensional data space where entities reside. Entities can be in the same location in some dimensions and in other locations in others, just the same as you might be standing in the same exact place Napoleon stood at one point, just at a much different point in the time dimension. Or, take two points on a cube which are at the exact same place in the 2D x,y dimensions but at very different places when viewed in three dimensions.

Similarity, then, determines how close two pieces of data are to each other, based on the “important” data dimensions (as defined by cardinality). Above a certain similarity threshold they are considered to belong to the same entity, below it they are considered two distinct entities. This is effectively measuring how close the two entities are in “enough dimensions to matter”. This is going to be different in each instance and depending on the data available.

Focus, then, is the act of looking at more or less data. Loosening the filter parameters slightly. If you’re looking at a point in data space, decreasing the similarity you’re looking at expands the result set you’re looking at. Increasing the similarity tightens your focus on a spot, reducing the amount of data returned.

This changes the inputs to all sorts of things. Data integration points, data mining tasks, alerting, all kinds of things are touched by this idea.

I know I’ve been veering into the deeply geeky here, but I find this stuff fascinating, sorry! I’ll leave you with one more Schmidt quote from the same talk:

“I spend most of my time assuming the world is not ready for the technology revolution that will be happening to them soon”

That is, given a bunch of data about something, the boundaries between one thing and another are pretty fuzzy. They might be different things, or they might actually be the same thing that I know different sets of things about. This is essentially the crux of data integration, but approaching it from a slightly different angle.

This might sound kind of abstract and pointless, but I’m starting to see that it’s not. And the reason for that is relationships. Creating relationships between two different entitie–saying that they are the “same as” each other for instance–is not the same as having two different places in space where data can collect and accumulate and saying that those two places are actually the same (what I’m calling a “data wormhole”). In one model the entities are connected, in the other two areas in n-dimensional data is connected via a wormhole, and data is transparently in both places at once. It’s like a graph database without entities. It’s useful because you can just store bits of data and they are returned as one or more entities depending on the precision you’re looking for or how similar you want them to be, and the context is built up around it automatically.

The problem is that this totally destroys the relational database paradigms because all of a sudden there are no keys at all, only n-dimensional coordinates. This in turn destroys the traditional concept of a query so you’re required to completely reinvent that as well–instead of saying “I want all entities related to X” you’re saying “I want everything within N similarity of Y”, which almost hurts my brain. You also have to index everything, which is not feasible in a traditional relational environment.

I’m pretty convinced the power this approach brings to the table outweighs the time it takes to rebuild the concept of a query though, and I can’t think of any other way to accomplish it.

The gist of it is that personalization algorithms tend to give us more of what we click on the fastest, which can quickly create a myopic view of the world. The presenter’s solution is for the systems using personalization (Google, Yahoo News, etc) to integrate “socially responsible” filtering mechanisms to make sure that everyone gets their daily dose of “information vegetables”.

This is backwards.

Personalization is only one half of the spectrum of information consumption. The other half is sharing. If everyone you socialize with online shares what’s interesting to them, assuming you don’t surround yourself with clones, you will see content that you otherwise wouldn’t. Conversely, if you share content that you find interesting, that content will make its way into your social circle regardless of personalization.

Systems like Twitter are the antidote for this problem, not some sort of official “information food pyramid”. If you’re not getting important information it’s because you’ve put yourself in an echo chamber.

This means that it takes very little data or storage to index things like sex, state, height, age, etc, because there are relatively few values for those. When you start getting into things that vary more (income, city, phone number, etc), indexing that data becomes prohibitively expensive.

This law, the curse of cardinality, came from computer science. It really has very little to do with the underlying statistics, which don’t really care about the cardinality much at all. It’s an equipment-imposed restriction that is holding back all kinds of progress.

In fact, after looking at this problem for a while, I think cardinality is actually a very important piece of information that can be used to change the way data is worked with.

For example, if I want to know what a user finds interesting about an entity, let’s say a person, there are a lot of options. We know his name, city, state, income, height, hair color and eye color. If the user indicates that she likes this person, which features of the person does actually like? This is a classic classification or recommendation problem.

Cardinality, in this case, is very helpful. If you know that income varies much more than state, you can assign more weight to it as you do your classification.  Knowing that there is more variation in something makes it much more interesting, because that’s how you find the outliers. It’s virtually impossible to find outliers in say, the state that someone is living in, because there are only 50 choices. You can look at the unique factors around the thing that you’re looking at, and find more “like that”, leaving out the general, uninteresting stuff. This is the field of “similarity indexing”, which consists of a few research papers and whatever code is hidden away in skunkworks projects.¹

All that to say that the data that varies the most is usually the most interesting, and the data that you probably actually want to be paying attention to. However, cardinality is typically dealt with as an obstacle to overcome rather than a potential asset. The problem is that the tools available today don’t actually allow you to use cardinality for anything. Good news is, I think that is going to change. The maturation of some of the Big Data platforms is going to allow the types of solutions that are needed to get past this limiting mindset.

The nascent field of stream mining is doing some very interesting things with real-time stream sampling which allow this type of analysis to be done accurately and in real-time.² The problems are difficult and some of the solutions are still in the research stage, but those are the types of problems that are also the most interesting.

This kind of analysis, I’m convinced, is the future of data. It is all about recognizing what makes data interesting to you, and then giving you more of that.

This is truly a glimpse into the future, some of this stuff is mind-blowing.

I can't wait to see what the next 5 months will bring. I am so excited to have access to the SDK now, I'm really looking forward to building some interactive data visualizations with it.

In said project there is a bit of code that watches the database for changes to an entity, and when it sees a change to that entity it automatically re-evaluates it against nearby, similar entities.  I want to keep a record of the entities that are being compared against one another, so I create a relationship and store it for future reference.

This is pretty standard stuff. It's basically exactly what a relational database does. However, I ran up against one concept which I haven't seen much about.

That is, I realized that there is a whole gradient of similarities between entities, from "exactly the same" to "completely different". If you turn this into a binary value, either related or not, you lose that entire piece of information.

This seems like quite a waste because if you're looking at entities that are related to a specific one that you're looking at, it makes sense to prioritize them based on their similarity. And if you're doing something like, say, using those similar entities to come up with a market price for the entity you're interested in, you probably want to use something like a weighted average based on how similar the other entities are.

As far as I know there is no way right now to store that relationship strength in existing data systems. Relational databases in particular do nothing for you here, you can either choose related, or not related. RDF-based systems could, I suppose, accomodate this, but what you really want in this case is a "strength" attribute on the "sameAs" property, and I have never encountered such a thing in the wild.

Anyway, I thought it was an interesting and useful idea. Since I'm da boss on my project and it sounded fun I went ahead and spent a day adding it in while I was in that bit of the code, so I'll let you know how it works out.