Flash Product Mgr. Weighs In RE: Flash 10 And CDNs

The Adobe Sr. Product Manager for Flash responds to me regarding my comments about Flash, p2p and CDNs.

For those of you too lazy to read the whole thing, my 100 word translation:

We are not saying you can't do what you have said in your article. However, this was not our intent, and we don't like the idea that you are saying Flash is going to kill CDNs. We *love* our CDN partners and don't want any of them pissed off or scared. That said, if you do want to do any of this p2p stuff, you will have to authorize through the Flash Media Server. By the way, Flash 10 is *really* cool!

Full Text:

Hank,

The new RTMFP protocol in Flash Player 10 provides an alternative to RTMP allowing customers to develop real-time communication applications with an improved user experience. Applications like chat and games are great examples of what you are likely to see make use of this technology.

Adobe has made no announcements regarding Content Delivery Networks and file sharing. Adobe highly values its CDN partners they are an integral component in delivering rich video experiences to Flash Player. This new protocol has been designed for communication solutions, not massive content delivery.

We chose UDP because it’s an efficient protocol for low-latency streaming video conversations, voice conversations or other similar solutions. UDP combined with the high-fidelity of SPEEX audio, will allow our customers to build great voice and audio solutions targeting Flash Player 10. To enable RTMFP and the UDP transport, authorization from a future server-based technology such as Adobe Flash Media Server will be required.

Flash Media Server will continue to be used to stream massive amounts of content to Flash Player because it can serve and protect video and reach the largest possible audience. New information within Flash player 10 allows a developer to monitor how a stream is arriving and make decisions if a different bit rate will improve the video experience for the end user.

We are very excited about this new communication technology, the protection around it and being able to improve the delivery experience for both communication and media delivery to Flash Player.

Justin Everett-Church
Sr. Product Manager, Adobe Flash Player

Flash 10 p2p and CDNs – Deeper Analysis

Yesterdays piece on Flash 10 received lots of interest and generally a great response. However there were a few complaints about the fact that the Adobe press release didn’t actually say p2p, or that the Amicima technology was only about VOIP, etc.

One esteemed member of the Flash community and old friend, Brian Lesser, a Flash server guru and author of Programming Flash Communication Server suggested in the comments to yesterday's article that there might be restrictions on p2p usage that might limit some of the CDN possibilites I described yesterday.

On the other hand the piece did get some great coverage over at GigaOM from the likes of one of my blogging heroes, Om Malik, who essentially concurred with my assessment of the import of all of this. And so given the high level of interest, I thought it would be useful to do a deeper dive into the subject and respond to some of the commentary.

1. Adobe didn’t make any announcements about a CDN.

This is true, but it is not the point. My point is that developers will be able to build p2p networks on top of Flash 10. Flash 10 provides low-level tools that are the core of what p2p is about. NAT traversal and the general ability to connect one Flash client to another is all that is required. In this case, the p2p coordination of this will be based around Adobe’s Flash Media Server, or perhaps open source Red5 once they figure it out. In any case a server of some sort will be required for any of this to work to serve as the central hub for establishing connections.

We don’t know the details of how discovery will work, but presumably Flash clients will register with a server, and then it will be possible for registered clients to connect with each other. This core functionality will allow p2p CDNs to be built.

2. This technology is only UDP or only for VOIP and so can’t be used for files.

Bits are bits. I am quite sure that RTMFP (the protocol that is responsible for these p2p services) does not specify that it can only be used to deliver voice, and not data files or video. This would be stupid and based on the current Flash architecture, likely impossible. Moreover, as stated on their Wikipedia page, the Amicima technology before Adobe acquired it, was demonstrated using voice, video, files, and presence. Regarding the point that you cannot do files with UDP because it is lossy (i.e. designed to be able to drop packets), this purported limitation of UDP is misinformed.

Yes, UDP is not a guaranteed protocol like TCP/IP, but many p2p systems use UDP. The way such systems work is that the sender and recipient have to communicate about what is being sent and what is actually received. So if you are expecting 100 packets labeled 0-99, and packet 49 does not arrive, the sender needs to send it again. This is a *very basic* aspect of designing services on top of UDP. UDP is indeed a very effective low-level building block for anything, and is totally capable of supporting guaranteed delivery at a higher level of the design.

3. p2p will not be able to offer security for content on user's hard drives.

This is, just not so. Kontiki, introduced a highly secure p2p CDN in 2000. As far as I can tell there will be no impediments to implementing security since all that is required is file encryption on the hard disk (which is trivial) and transmission encryption, which is built into the protocol.

4. Couldn't Adobe's security model get in the way here?

It is true that we don’t know how Adobe is going to approach security. How restrictive are they going to be? In order for Flash 10's new RTMFP protocol to work for building any kind of CDN, it must be accessible programmatically (i.e. from Flash's programming language ActionScript) without user intervention.

For example, if Adobe requires that every time you want to establish a p2p connection between two Flash clients, you need to click on one of those Flash security dialogs then the whole thing comes crashing down. I could see if Adobe required that you click on such a dialog once for each website that required p2p access, but if they required that user interaction for every p2p access it would really be problematic. Aside from such draconian security measures, I cannot see any other impediments to implementing full-blown p2p CDNs. Of course any of you serious Flashers who have other thoughts please leave them in the comments.

5. Can this technology *really* kill CDNs?

Well, not in a literal overnight sense. There will still be a role for CDNs, but as I see it the heyday of growth, at least in their current form, is coming to an end. This is because companies like Amazon are commoditizing the central storage benefit through S3 (and will continue to do so) and because p2p technology will reduce, though certainly not eliminate the need for premium, centralized storage.

Of course the best such CDNs, using Kontiki as a design example, will of course also have servers that provide central storage and delivery for when the requested content is not available from another peer – a blended strategy. So the idea of centralize services will not go away. But killing growth is, in my estimation, the same thing as killing a category. If you do not grow, you die.

And so I believe, p2p is now poised to put a serious hurt on centralized premium server downloads. Of course it is entirely possible that CDNs will find some other significant value add, but as I see it this technology will take a huge bite out of the CDN market and for the CDNs that just ain’t good.

Adobe Introduces P2P Flash Player, Kills CDNs

Update: I have written a follow up to this article which is here.

Today Adobe announced the public beta of a new Flash Player that is going to change the way we all use the Internet. More importantly, the new player changes the economics of the Internet.

Interestingly the two really key features are not getting much play in the news yet as people have focused on the new graphics capabilities. But graphics is not what is really interesting here.

The first key Flash Player feature is a peer-to-peer (p2p) technology from a company they acquired called Amicima in 2006.

The second key feature is that Flash can now save files directly to your local hard disk.

These two new capabilities together make any type of p2p application trivial. Whether you are trying to roll your own LimeWire, or you have more honorable intentions such as implementing a server-less voice over IP (VOIP) application such as Skype, the new Flash Player will make these kinds of applications trivial.

What is even more interesting than cloning existing applications, is the innovation that will be unleashed by making p2p technology an assumed part of the web protocol stack. For example, it will be a few hundred lines of code to write an AIR application that will allow you to drop a file onto an icon and have that file appear on your buddy's computer.

But the most significant impact of all of this will be economic. As I see it, this is death to the content distribution networks (CDNs). The CDNs make their money providing companies the ability to centrally store and serve heavily accessed and/or large files. For years, companies like Kontiki and Red Swoosh tried to popularize p2p delivery networks that would make it possible to move large files without using such central servers. So if you wanted to watch a video or download a large game, these p2p systems would pull that file from other users who already had it instead of from the central store.

The problem with these p2p delivery networks was that not enough people wanted to download an application that only had the purpose of saving the service provider money. In other words, companies like Red Swoosh and Kontiki didn't have enough end-user benefit to get people to download them.

But because people download Flash anyway, the chicken/egg problem associated with getting the p2p software onto people's computers is eliminated.

What all of this means is every major video site will now be able to deliver video streams, while using only a trickle of bandwidth. When the Skpye founders introduced their video service Joost, this was what they hoped to be their secret weapon. They wanted to operate a video network in the same way they operated Skype, i.e. at very low cost. But people preferred to watch videos in their browser without downloading anything special.

But baking p2p into Flash means *all* web video sites will be able to deliver video using minimal central server bandwidth. As a result, the largest and best CDN customers will disappear, or at least will radically reduce their needs because, in effect, every video website will have the low cost of operation of Joost.

Of course this is not just about video. If you have any large file movement needs that require a CDN, Flash-based p2p will be on your short list of engineering imperatives.

This new version of Flash, combined with Amazon's super low cost S3 storage system will be fatal to many CDNs. Personally, I'd be shorting every public CDN stock around the time that Adobe leaves beta with this stuff, because it spells real trouble for that market.

And if you ignore me now, don't say I didn't warn ya. For the CDN market, this is Armageddon.

Is Marc Andreesen A Conehead?

Coneheads: movie about alien visitors.

Suspected replacement of the real Marc Andreesen.

The above photo, taken by Dan Farber, and included in this article by Dan Farber, requires we at least consider the possibility that this is not really Marc, but a replacement Conehead. Marc's purported comments in Dan's article lend credence to the theory. The gist of the article is that Marc is annoyed about Google's new platform "Friend Connect" But he won't come out and say it.

I am a Marc Andreesen fan, and so I am worried that this guy parading around as Marc is messing with his reputation.

Apparently the faux Marc says in the Wired article Dan references is:

...I think a lot of companies have what I call 'strategitis.' Instead of launching a product, which would apparently make too much sense, they come up with a 'strategy.' There's a strong temptation for companies that don't have strong social networking franchises to roll out social networking 'features' instead of products, and in reality, consumers like to have products.

How ridiculous.

It sounds like Marc is trying to describe a trend. How many "companies that don't have strong social networking franchises" are there that are "rolling out 'features' instead of products." How many such companies does it required to create a trend that would justify classifying this as a general non company specific point?

If I believed this was the real Marc, I would tell him that if you are going to pick a fight with Google, go ahead. Don't be a wimp about it. There is no trend. There is only one company that fits your description, and they only have two such APIs. You were all over Google's first API, Open Social, because it allowed you to compete with Facebook. Now the second one, Friend Connect, the one that makes this "trend" bothers you. The reason it rankles is it allows any company to make their web applications "social" (ugh I hate that word but what the heck), making everyone a potential competitor to your company Ning. But instead of just admitting it, you attempt to get all philosophical and abstract.

I personally don't think the real Marc Andreesen would be at all defensive about Google's moves vs Ning. And I certainly don't think he would be so illogical as to make a "trend" out of one company's single action.

And so, I want the abductors to bring back the real Marc Andreesen -- the smart, intellectually honest one. I'll personally chip in to send the Conehead back to Remulak.

Microsoft Shepherding Intellectual Property Discussion

As someone who has been around the computer industry long enough to say that I was there before the birthing of the microcomputer (the term we used before PC), my view of Microsoft has always been dim. The reputation of Microsoft as a company acting on the edges of propriety is legendary. There are many examples, from accusations from GO Corporation's Jerry Kaplan that Microsoft stole the core concepts they pioneered in pen computing in the early 90’s, to Microsoft’s anti-competitive arm twisting of PC manufacturers.

When Google was formed, they took as their motto, “Don’t be evil.” This was clearly code for “we won’t be like Microsoft.” And certainly, at the time, history was on their side.

But indeed times have changed, and some might even suggest the roles have reversed.

Windows is clearly not gaining ground in devices or phones. Web applications using HTML and Javascript, and Adobe’s Flash and AIR have almost totally supplanted Windows as an application development target. Microsoft’s web strategy has, for the most part, despite spending billions of dollars, been a failure.

To quote Bill Gates, “Success is a lousy teacher. It seduces smart people into thinking they can't lose.” Indeed, if Gates is right, in recent years Microsoft has finally had the opportunity to do some solid learning at the hands of Google.

The tipping point for the change might have been the loss of the anti-trust case. Or perhaps it was tied to the retirement of long time Microsoft general counsel Bill Neukom, and the ascendance of kindler gentler general counsel Brad Smith in late 2001. But whatever the cause, the result is clear. Microsoft, right before our eyes has become a very different company from the one we knew ten years ago.

The first indications became clear when, on the arrival of Smith, Microsoft began settling cases that they never before would have settled. The old Microsoft fought everything tooth and nail to the bitter end. Microsoft was famous for not giving an inch – on anything. Ever.

More recently, Microsoft has indicated a desire to turn another corner. Tom Rubin, Chief Counsel for Intellectual Property Strategy at Microsoft, gave a speech last week in New York about the future of intellectual property and creativity online. The core of his argument, is the basic idea that technology companies and content companies need to figure out a way to get along. He frames the issue as primarily a social and structural one more than a technological one – a point with which I wholeheartedly agree. The term he uses for this needed awakening is “stewardship.”

Tom’s basic point is that recent data is showing us that the Internet is currently not a platform that can economically sustain content creation. As such we need to figure *something* out to allow creative efforts to remain economically viable. Tom defines stewardship as follows:

“So stewardship means appreciating the impact of one’s actions on others and on the media ecosystem as a whole. It means making sure that new species of individual creators can thrive and survive, and that new technology is given room to flourish, even if they both challenge your traditional ways of doing business.”

After the talk I had an opportunity to sit down with Tom, and I discovered that the speech reflects more that just a philosophy about intellectual property. The speech is really a marker for a new concept for how to approach intellectual property by Microsoft. It reflects a desire to truly move the discussion forward in some substantive way. Microsoft wants, in some respects, to be a shepherd. As Tom explains it, Microsoft’s desire is not to impose what is right, but to engage us all in a discussion that leads to a future we can all live with.

Tom pointed out in our conversation that Microsoft is both a technology company and a content owner, and so they do not take the perhaps extreme positions that pure tech, and pure content companies tend to get into. Of course, the fact that they are Microsoft, will indeed, in the minds of some, taint their credibility.

But what is really fascinating here is the idea of Microsoft *seeing themselves* as a kind of reasonable, thoughtful, verging on academic voice on this issue – almost an honest broker. (Yes I do see the irony in that last sentence.) Of course it is not at all clear that this will be possible. It is entirely possible that such talk will not actually fly internally at some point because of competing interests. It is also possible that the outside forces are so entrenched and Microsoft’s history hopelessly undermines some kind of new role in the industry. But the fact that this kind of talk is even happening is hugely significant since it would not have happened in another era.

At the end of the day only time will tell if Microsoft really can bring everyone together to come to our collective senses. But it seems to me some kind of intellectual property rapprochement is critical to our economy, as well as to our culture. And so, in my book, Herculean though the task may seem, it is certainly an admirable and worthy effort.

And so, perhaps for the first time in my professional life, I can actually say, at least on this issue, I truly applaud Microsoft. Godspeed.

Blaine Cook Writes First Blog Post Since Twitter... About Scaling

Today, Blaine Cook, formerly Twitter’s chief architect, writes his first blog post since leaving Twitter. It certainly seems like he has some stuff to get off his chest.

The gist of the piece is that languages don’t scale, architectures do. It seems clear why Blaine might want to say something about scaling, since it is clearly the number one issue at Twitter. And he did take some heat for the problems there. And in some corners of the Web, Ruby, the language that Twitter runs on, has taken even more heat.

Blaine's arugment: Scaling is fundamentally about the ability of a system to easily support many servers. So something is scalable if you can easily start with one server and go easily to 100, 1000, or 10,000 servers and get performance improvement commensurate with the increase in resources.

When people talk about languages scaling, this is silly, because it is really the architecture that determines the scalability. One language may be slower than another, but this will not affect the ability of the system to add more servers.

Typically one language could be two or three, or even ten times slower. But all this would mean in a highly scalable system is that you would need two or three or ten times the number of servers to handle a given load. Servers aren't free (just ask Facebook), but a well-capitalized company can certainly afford them.

The problem comes when your architecture is such that you can’t just add more servers. While Blaine does not discuss this, the primary reason things don’t scale has to do more with the cost of data access. Databases are almost always your bottleneck, because all your data typically needs to be stored in some central repository.

So how you architect your data storage and access will determine your scalability. For example, do you use RAM based caching like memcached to improve performance and limit the need to read the database? If so, is your caching architecture good enough to limit most reads from the database, or just a few? These are the kinds of architectural decisions that will determine system performance.

In Twitter's case, there is zero chance that the problems there are in any way related to their language. It is likely that there are architectural challenges which come from the fact that it is very hard to cache a Twitter data request since no two people ever get the same data. And even for a given user, the data requests change quickly since users are always receiving tweets. This is a hard, though not unsolvable problem that requires a very specialized caching architecture. Eran Hammer-Lahav, has done some interesting work in this area and talks about it in an extensive blog post.

The bottom line is languages don’t kill scaling, programmers do. As such, Blaine's piece, while sounding a bit defiant, might really be read more like a mea culpa. Though, to be fair, despite all the chatter and criticism, scaling Twitter is indeed a non-trivial problem.

Media Darling Powerset vs. Non-Media Darling Hakia

Over the weekend, The web was abuzz with discussion about Microsoft considering the acquisition of natural language search company Powerset. Today there is yet more coverage. Some time ago I had heard a rumor that someone was looking at Powerset, but was relatively uninterested. Hearing that the potential acquirer is Microsoft certainly makes it more interesting, but I have to say the concept leaves me more than a bit incredulous.

From skeptic to user

I became familiar with Powerset's only competitor, Hakia initially because they are a New York company. I became intrigued with Hakia because several months ago I tried their search engine, and it worked – really well. This was a surprising result for me since I have always been a skeptic regarding all things relating to artificial intelligence, speech recognition, natural language processing, and other such fuzzy technologies.

At least in the area of natural language processing Hakia that has changed my mind. In fact, it has become common for me to use the Hakia search engine when Google does not deliver sufficient results.

Hakia and Powerset are part of the same general area of natural language search. The idea with both services is that you can actually ask specific questions and get answers. But there are critical differences between Hakia and Powerset. And those differences bring me back to my incredulity at the idea that Microsoft is taking a serious look at Powerset.

Powerset indexes 750 times slower than Hakia!

I have no expertise in natural language processing or semantic search, or any type of full text search for that matter. But as far as I can tell, Hakia’s technology is *far* superior to that of Powerset’s. Why would I say that?

Well first, as I have already said, it works. It is a real live search engine. I use it. I can’t say the same for Powerset. Powerset has yet to show anything but a search engine for Wikipedia. A big part of the reason Powerset doesn’t seem able to offer a real search engine is the fact that according to their own reports, it takes them about 25 seconds to index a page, based on an average of 25 sentences per page. According to Hakia it takes them 1/30th of a second to index a page. Essentially this means that Powerset cannot scale. It is seven hundred fifty times slower than Hakia!

Now you might assume that Powerset is slower because it’s applying some serious, and superior indexing mojo, and therefore what it is doing is much more valuable than what Hakia is doing. But alas that is also not true.

Hakia really knows how to read

Hakia is doing something called “ontological semantics”. What this means is that over the last four years, Hakia has developed an “ontology” for human expression. In layman terms, what this means is that what Hakia does when it indexes a page is to look at each sentence and figure out what the *questions* are that each sentence answers. Any given sentence usually answers 3 or 4 questions. These questions are coded and go into what Hakia calls their Qdex, or question index.

In order to be able to figure out what the relevant questions are for a given sentence, Hakia’s indexer has to literally read the sentence. By “read” I mean it has to understand the actual meaning of the sentence semantically. This is a big deal.

Powerset uses statistics + syntax but can’t actually read

So, while Hakia is actually reading, Powerset, does not actually attempt to understand what sentences mean. It uses a system that parses the syntax of the sentence and guesses matches based on statistics. But this approach means that for questions that do not match previously encountered syntactical patterns, the system will not be able to find answers, even if there are in fact answers in the database.

Powerset benefits from the Silicon Valley echo chamber

Now, if, for a moment, you presume that it is true, or even *possibly* true that Hakia is the superior service and technology, or if you even assume that Hakia is just equivalent to Powerset, why would Powerset be so continuously celebrated while Hakia is overshadowed?

The only answer I can come up with is that the west coast is such an echo chamber that very little sound gets in or out. And so it must be shocking when a New York company develops a technology that seems to beat the pants off something that should be pure Silicon Valley. Just a thought.

In any case, it seems, for the record, worth noting that we have the clear leader in natural language processing and search technology right here. And, as an admitted New York partisan, after a while it does get a little annoying to hear such continued fawning over a west coast company that is very likely, at the end of the day, just another Silicon Valley also-ran.