Last Weeks Trip Report

Sorry for the delay, but I've been woefully behind.  It seems I never catch up anymore.  Anyhow, I made it worse for myself by deciding to get on a plane and head out to Silicon Valley last week, which I have been able to successfully avoid doing for over a year.

First, the trip started poorly.  Sunday evening I was forced to watch Peyton Manning finally beat the Pats in a game that mattered.  I like Peyton, so it isn't about that, but to watch him come back on us in such a way, and to see that Mr. Brady is human every now and then was a mongo bummer.  Even worse, I had a house lined up in Miami for the Super Bowl - and if you have ever attended said event, housing is the hardest thing to find.  So I limped off to bed at 10:30pm with a buzz and a belly full of really bad (for you) food.

My alarm went of 8 minutes later (4AM) so I could make my 6:30 AM United to SFO flight.  There are no longer any flights between Boston and San Jose, two of the major business capitals of the country.  I hate airlines, and truly wish the bad would die.  Who would want to go to San Jose?  Only anyone who sells too or buys from one of the 18 zillion tech companies along the 101.  Was there ever an empty flight to San Jose?  Maybe American felt the planes were getting too heavy. Fortunately, I was in first class - which is a nice way of saying I was in a slightly bigger, really cruddy uncomfortable seat with only one person to have to climb over to go to the bathroom vs. two.  The good news is I was in row 1 so even my 14" legs had to be bent in half to sit.  There had obviously been well over one billion other rear ends in my seat previously, as it was about as padded and comfy as my driveway.  I don't have a rear end, per se.  Big gut, no butt.  I was in pain after 25 minutes.  5 hours later I was almost a cripple.

We landed on time, and a car was nicely waiting to pick me up to bring me to my first meeting, with Data Domain.  DD is an impressive group - they invented the whole Data De-Duplication gig and have ridden that wave brilliantly.  Now everyone and their brother is trying to catch up, but these guys have some great stuff and a big, impressive list of customers.  Best of all, their CEO Frank Slootman, is Dutch.  Therefore, upon meeting him, I immediately had to steal the line from Austin Power's Dad and say "there are two things I cannot stand - intolerance, and the Dutch."  Frank looked at me as if I were insane, as apparently he hadn't heard that one before.  We worked it out.  I chatted with a bunch of their smart folks theorizing about where other implementations of this technology could really affect change in the world, and found quite a few.  What if you could get the performance attributes required by a high percentage of today's applications on a primary store that happened to get 40 to 1 compression rates?  Imagine the economic advantages and the consolidation potential.  What if everything were stuffed into one place?  Seems it may be easier to find things if it were all in one place. 

Which got me to thinking about the fact that this is really the first time in this industry in a long, long time where so many "emerging" players have become legitimate, going business concerns.  Historically there have always been start ups trying to become the next big thing, but most focus around a new technology or building a better mousetrap, and less about a getting to market and solving a heretofore unsolved problem with that technology.  It is very rare indeed that someone comes around with a new new thing and be able to reach financial critical mass.  Normally those companies are acquired along the way, or die trying.  Rarely do they affect the incumbents, either positively or negatively.

I look at the landscape now and see folks either at or rapidly approaching ciritcal mass ($100,000,000 for hardware, $25M for software) in revenues - in a market long dominated by just a few big guys.  DD, Pillar, 3Par, Equallogic, Lefthand, Compellent, SANRAD, Copan, CommVault, Isilon, Riverbed, BlueArc, and a host of others, and I wonder why.  It has to be because of two factors - the first is the market requirements have changed - i.e. the nature of the business itself has changed because this industry was developed on the fact that transactional data was where all the value was.  Today, most of the data created is not transactional, it's fixed or persistent.  It doesn't change.  Even transactional oriented data can start as fixed - it may be an event for example, but eventually even if it is changing initially, it becomes fixed at some point.  Treating all data the same at creation until we eventually nuke it is illogical.  Therefore, the second element required for newbies to have real businesses is that the incumbents simply are not providing solutions with the attributes to address the new world order.  No IT person wants to buy from a newbie, no matter how cool the stuff.  There is too much risk, and lets face it, too much work to do to justify doing it.  So if a risk averse, already overloaded IT dude is going to go to bat for a newbie company, it only makes sense that the newbie must be solving a problem the others don't.  I find that interesting.

I went to dinner with Dave Hitz and Kris Newton of Netapp that evening, at one of the two meat joints in the Valley.  Why are there no restaurants, nor hotels in Silicon Valley?  It's the tech capital of the planet, there is TONS of money there, and yet there seem to be 3 hotels, a motel from 1948, 3 restaurants, and 12,000 Starbucks.  Dave was kind enough to bring a bottle of his brother's wine - Chateau D'hitz (pronounced "ditz" of course).  It was a fresh 2005 Meritage named Screaming Priest (I didn't ask), and it was darned tasty.  Dave and I spent hours talking about the world of IT years from now (I can almost feel the boredom on my childrens faces).  He is one of the best people on the planet to talk about ethereal IT concepts with.  I was in town to speak to Dave and some others for a book I'm finally writing (it is one of six that I've been threatening for years).  He's a fascinating guy and a superb human, inside of a complete weirdo genius (which I mean entirely complimentary, I could not have more respect, admiration, nor flagrant envy for the man).  He is worthy of a book all on him, so maybe I'll make it seven.

I spent the next day with various mucky mucks I'm not telling you about.  I did have a nice dinner that evening with Mike Klayko, Tom Buiocci, and Dan Crain of Brocade fame.  They had just received their FTC approval a few hours previously for the McData acquisition, so all was well.  I was a tad nervous that our government was going to make the wrong call on that one, which would have not only really irritated me as a tax payer, but would have been the cause of McData's death - the very thing they supposedly were trying to prevent.  Without disclosing confidences, Brocade's integration efforts behind the scene are about the most thought out and complete as I've ever seen.  The proof will be in the pudding, as they say, but from a planning and contingency perspective, they have their act together.  Perhaps the most interesting part of the conversation, unfortunately, came about during some tangent on Cancer - of which I'm a survivor.  Mike shared that his daughter (Christina) in-law is going through some tough stuff and dealing with a harsher dose of treatment than I went though - and mine sucked.  Her link tells her story, which everyone should read - and donate twenty bucks to the cause.  Puts the problems of IT into a different perspective.

Speaking of Pillar, I spoke at their sales meeting.  I get asked to do this stuff a lot, which is flattering, but i refuse most, as I really do hate to travel.  Since I was there, and I've been following the company since it wasn't a company, I said yes. (Really - my first trip to Israel in 2000 - I end up in my hotel after a brutal day of meetings, tired and semi-loaded, when on the way to the elevator a small man stood up and said "excuse me, Steve, may I speak with you a moment?"  Too stunned to be terrified, I'm thinking oh oh, the Mossad.  They aren't kidding when they tell you everybody knows everybody in that country.  I never did find out how he knew I was coming, or when, or where I was staying, or how long he waited.  He introduced himself as someone working with Digital Appliance, and asks if I'm familiar with them, which I am not.  He explains how Mr. Larry Ellison had DA started to create the ultimate scaleable database machine - to support all the thin clients that were going to happen.  That didn't work, but they stumbled upon a storage architecture that did.  He wanted to know if there might be a business in what they had.  I thought there was. Now I figure if this hits, Larry owes me a billion or two.) 

Being a veteran of a million years and a million sales meetings, I thought I'd seen it all.  Two things occurred that were very new to me.  First, the evening before as I was heading into the Santa Clara Marriott (which is the interviewing capital of North America in case you were wondering), I heard a loud ruckus.  It was Pillar.  I popped my head in only to witness the most organized U.S. vs. Great Britain beer swilling contest I've ever encountered.  There were two lines facing each other and one person on each team guzzled their beer to completion prior to turning it upside down and putting it on their head, and the next guy would go.  It was quite exciting, and the Americans won, but I'm fairly confident cheating was involved.  Tough to out drink a Brit, I've found. 

The next day I went down to do my thing, and walked to the back of the room to first listen to CEO Mike Workman's speech, which is where I witnessed the second unique thing of the sales meeting.  Mike was on stage wearing a WWII jacket and helmet, in front of the Pillar version of a giant flag.  Mike was doing his Patton thing, which is fine, except Mike makes me look like Bill Walton - the man is seriously short.  His speech was hilarious, albeit loaded with profanity.  (Yes, I said that.  I was no longer concerned with my content.)  It says something when your leader can have some fun and do semi-insane things in an almost public setting.  His act was funny - but his messages were deadly.

Pillar isn't ashamed of the fact that they have been put on this earth via Larry Ellison to single handedly alter the entire economic and operational landscape of storage within IT.  Their mission is to build storage for the new world order - simple, scalable, Q of S, unified storage that is smarter than you are - and cheaper than lunch.  Why you ask?  My theory is that Larry doesn't like all the value that is placed on something as mundane as storage infrastructure.  Larry wants the value to remain entirely at a higher place - namely the Database.  If he can help move storage the way that IP networking evolved down an inevitable standardized commodity curve, he can grab more of the higher value dough.   Larry has spent well over a hundred million bucks on this so far, and that's just what he found between the cushions of his couch.  He can be patient.  He isn't a VC who needs this exit to save his portfolio.  He's got all the time in the world.  That's a pretty good advantage, since I've always loved saying that all problems we face can be solved with enough time and money.

I don't mean to imply that the Pillar folks aren't trying to solve real problems with real engineering - they are - as are a lot of others.  They have what looks like a brilliant story around their unification story, the data center efficiency improvements and their ability to optimize the cost per performance per measure of space. The difference is that they are pursuing this goal knowing that they have a seemingly endless pile of money to get it done.  These guys will be around for a long time I think. This could very well end up being book number 8 - maybe with enough intrigue, suspense, and violence to be a movie!

I took the red eye back, which might be the worst overall traveling experience one can do.  The airport is closed, and there is no one around except the living dead of the IT industry trying to get home.  It looks like a B-grade hung over horror movie. The only planes that ever arrive early are the red eyes back from the west coast - so if you were going to be able to sleep all cramped up and wildly uncomfortable, forget it, because you just landed.  The good news is I didn't drive, I took a car service, so being early was of absolutely no value.  The better news is it was approximately 2 degrees (Fahrenheit, which is roughly 1100 below zero Celsius) out, and I had nothing but a sport coat, which I clung too while tromping by the endless line of limo clones, none of which had my name misspelled. 

I'm looking forward to not getting on a plane next week.

My Data Was Stolen - Again

TJX had a little theft issue last week - and they aren't even sure how much was stolen.  All they know is that potentially hundreds of thousands of customers credit card transactions have been stolen - over a very, very, long time potentially.

TJX owns Marshall's, TJ Maxx, Bob's, and some others.  I love a bargain, so I shop at all of them.  My card hath been defiled.  Again.  This marks the third time within 12 months that I've had to deal with someone enabling others to latch on to my credit card information.  Marriott, The Boston Globe, and now TJX. 

The Globe reports that our friends at TJX were a tad lax in adopting the security standard put forth by the PCI (Payment Card Industry Security Standards Council), which is made up of Visa, Mastercard, Amex, etc.  According to the article only 31% of folks comply.

I don't know about you, but I find it ridiculous that this problem has not been fixed.  I hope it costs TJX an outrageous fine along with huge loss of goodwill.  It would have cost them diddly to put the right things in place to keep this from being an issue.

My pal Jonny Oltsik can shed the light on what's happening with the PCI and the entire world of security much better than I, but every time one of these boneheaded moron maneuvers happens I at least have the satisfaction of being able to say I told you so.  For 3 years I've been telling anyone who would listen that it is inevitable that the only way to protect data is to encrypt it.  You will never keep bad things from happening (though I'm not advocating that you shouldn't try), you can only mitigate the damage done when it occurs.

TJX doesn't even know how it happened.  It could have been a nice inside job, as you can fit an awful lot of small transaction data on a USB key, let alone on your 60GB iPOD.  I'd like to think it was outsiders, but Mr. O tells me that the overwhelming percentage of data thefts still occur from insiders.

As a CEO, or really any legitimate seasoned business executive, you already KNOW that no matter how great a person you are, how fairly you treat everyone, what a superb environment you create for your people, some of those people will never be happy.  Humans are odd, they have wildly differing views of the same thing.  A person who got paid $48,000 two years ago and was seemingly happy made $150,000 last year and then quit when you showed them a pay plan that only got them to $175,000 if they perform this year.  We know in our own minds that we would be tossing rose petals at our feet for such treatment, but that's not how others think.  So if you know that people are flawed and you know that your only reason for being is your customer - and yet you turn a blind eye in protecting the sacred information your customer entrusts you with and not only piss them off, but put their world into jeopardy.  Getting out of identity theft dilemma's has not proven easy.  It is a nightmare.  You caused that nightmare.  Worse for you is there is really nothing you sell that we can't find elsewhere.

I'm still waiting for the first intelligent retailer to market the fact that they truly protect their customer data.  Wouldn't you love to see L.L. Bean or Land's End shouting at the top of their lungs that they "Guarantee To Keep All Consumer Data  Encrypted And Secure" ?  Wouldn't the marketing win alone make the cost of implementing such a solution minuscule?  I'd buy something from them just because of that, I don't even care what it is.

So I am a believer in keeping the governments of the world out of my life as much as possible, but since the vendors I use simply refuse to practice safe business, it seems until our fine elected officials mandate the rules nothing will change.  A simple law that says the CEO is personally responsible for any non-encrypted data being lost, potentially lost, or stolen and has a minimum jail term of 5 years ought to solve the issue in short order, I suspect.  And don't cry about the cost, that's bull.  Vista gives it away, and there are plenty of other alternatives if you don't want that one.

Come on people.  This is stupid.

On an completely unrelated note, as a real fan I'm hoping the Bears beat the Saints, because after beating the Colts (yes, again) the Pats would face the Bears in the Superbowl.  We've already kicked their butts this year AND it would be a tremendous redemption for the 1986 whooping they put on us.  That loss, to the likes of the Fridge and Jim McMahon's was like getting a giant wedgie by the curly haired jock stud moron in 8th grade in front of all the cheerleaders.  Thankfully, the jock is bald and works as a security guard at TJ Maxx now. 

As a lover of all things underdog however, it is awfully hard not to be pulling for the Saints.  I hope Chicago beats them so we don't have too.  However in the highly unlikely case that the Pats lose to Indy (which statistically is an actual possibility, believe it or not), I'm all about the Saints.  Good move Marty, that Brees kid is all done.  Clown.

Real Data Center Costs

My latest in CW on thinking about power, cooling, air flow, standards, and all sorts of other fun stuff.

Nothing says Friday like an iSCSI debate....

I loved Chuck Hollis' (EMC) blog, and equally enjoyed both Dave Hitz and Tony Asaro's responses.  With EMC, Netapp, and another ESG'er pontificating on an issue, it must be good.

All of the aforementioned folks are smarter than I, but let me give you some different things to consider.

First, I'm an iSCSI proponent.  I remember meeting an intern who worked for Andy Bechtolsheim (who's name I can't recall, but I do remember he looked 12 and had more brain capacity in his pocket protector than I've ever possessed) who was charged with figuring out how to push block data over IP.  I think it took me 13 seconds to recognize the implications.  It took me a bottle of wine to come back to reality - which is: 1. even if you could use IP networks for block traffic, it will be a long time before people use it to replace core data center technologies, such as fibre channel and 2. eventually it will not only replace fibre channel, but whatever other new specialized connectivity protocol/transport the world comes up with.  I shall explain.

I remember within a very short time of the iSCSI initiative announcements people immediately got on one side or the other - there was not middle ground.  I was in Silicon Valley when Nishan announced their IP storage router (in an awesome display of bravado, arrogance, and downright stupidity) by telling all that the days of fibre channel were over.  They said this in the height of go go valley boom, when there were more than a dozen fibre channel players all getting piles of dough and driving Lamborghini's.  My timing was brilliant, albeit pure luck.  My cell phone rang with abandon, and everyone from the Wall St. Journal to the Financial Times was asking what I thought about Brocade going out of business.  I remember Ashok Kumar from Piper Jaffray getting on CNBC or some equivalent and discussing the end of the fibre channel market.  I believe it was early 2000.

Billions of dollars of value were traded in short order.  I got a call from Brocade and was asked to come in.  I had never met Greg Reyes, but knew most of his team, and certainly knew of him.  They wanted my take - and I gave it to them - death is not near, but it is inevitable.  I haven't changed my tune.  Greg told me point blank "you can't put block data over IP".  I suggested he not say that.  I suggested he instead say "Brocade builds block storage networking equipment designed to enable sharing of high-performance, mission critical storage assets across multiple servers.  In order to guarantee both performance and more importantly - reliability - we do that today using fibre channel.  We don't care about fibre channel per se, nor any other specific protocol or transport - we care about providing the most reliable block networking services available to the thousands of customers we have around the world.  If IP, or any other technology advancement comes along - as things always do - Brocade will adopt them and offer them to our customer base."  In short, I suggested he tell the world this was no big deal, and if it mattered eventually, they would offer it.

He didn't.  He said "you can't do block data over IP, you simply can't".  He was wrong.  The message was wrong, and this single statement did more to polarize the factions than anything else in this debate.  (Brocade, of course, has subsequently adopted this position, rightfully so, as have most all fibre channel players.  Once you remove the emotion and realize no one is throwing out the baby with the bathwater, cooler heads prevail.)

So Chuck's observations are hard to argue with - and Tony's right in stating that it's really a matter of perspective.  Since I prefer my perspective, and feel I'm the most qualified to share it simply because I was there on day one (with no regard for competence, mind you), here's what is going to happen and why.

History is littered with proprietary solutions that are developed to solve very specific tactical problems that turn into markets which last long enough for some to get rich, others to die trying, and then Darwinism does its thing and some sense of reality is applied to that market for that period of time - and to the victors go the spoils.  Then, eventually, the reason for that market no longer will exist - it will be replaced with the next thing, be commoditized and standardized, and the original proprietary solution will cling to life as long as possible with arguments like "if it ain't broke, don't fix it".  Water always finds the lowest point, and so do technologies.  Remember MassBus (I know, I'm old.), SDI (then ESDI), IPI and a zillion other block interconnects and protocols which existed in the day - only to ultimately give in to SCSI for a protocol (and a transport) which won because commoditization and Darwinism do really work.  How about Token Ring, DECNET, Banyan, x400, blah, blah, blah.  IP won.  Ethernet won. 

Things don't eventually win because they are better in the short term - they win because they are more ubiquitous.  They become more ubiquitous because they take advantage of commoditization economically and VALUE can be generated because of that shift.  IP killed the other networking technologies over time.  Once we gave in, life didn't end - because only those who fought IP died.  Those who recognized the inevitable were able to plan accordingly and either jump on the bandwagon and fight to own the commodity market or moved their value up the food chain above the commodity fray.  Those who hoped it would go back to the old days or defended the fact that they were better, slowly and painfully died.  Banyan might have been better, but now they give me phone numbers on-line. (Switchboard.com).

The reason iSCSI wins long term is because the ubiquitous market elements are in place (IP is everywhere and Microsoft has added it to the O.S.), and the cost is zip.  Betting against that is like being the guy at the party talking to the prettiest girl when Tom Brady and Derek Jeter walk in and still thinking you are the guy she's going home with.  Theoretically possible, but I don't like the odds.

So, since you all now totally agree that I'm right (eventually), from a more pragmatic perspective, here's why you shouldn't care.  99% of the time today, iSCSI should not be used to replace fibre channel.  It should be used to augment and enhance all the brilliant things fibre channel does.  90% of the servers in fibre channel SAN shops are not connected to the fibre channel SAN.  If the block storage network is good enough for 10% why isn't it good enough for the rest?  iSCSI enables companies - from small to enormous - to extend the value they received by having a FC SAN to their internal mass server/connectivity market.

What surprises me is that the EMC's/IBM's/HDS guys who are in the core haven't been the ones leading the push toward iSCSI.  The name of the game is "filling up the box" and what better way to help a customer spend more money with you than to show them how they can take control over the data on 90% of the servers vs. 10%?  To NetApp's credit, they did figure that out a while ago.  They ship a huge percentage of their Filers with iSCSI connectivity as well - though I don't know how many use it nor what percentage of the systems are used for block vs. file.  Regardless, the other guys own the core, and if they don't use iSCSI to add connectivity and capacity to the core (which steals share from all their downstream competitors, fyi), it's naive to think that someone else isn't going to create a parallel iSCSI network.  Once that happens, users will realize that this stuff is pretty good, and as a matter of fact, some of the "cheap" arrays I bought for my cheap iSCSI SAN happen to work way better than my more expensive core stuff.  Oh oh.  Do you really believe none of Equallogics' thousands of customers are big enterprises - or better yet, big EMC enterprises? 

The argument that the SAN guy doesn't want to deal with it is also inevitably destined to fail.  The backup guys still don't like to deal with desktops and laptops - and hate the thought of having to deal with remote office data - because their world is held together with spit and chewing gum.  Adding more complexity to their lives is not something they are interested in - but it doesn't matter.  Whether its mandated by law or becomes the good corporate practice it should, the market will force those attitudes to change, as it will with iSCSI.

The one point I think Chuck nailed was the services piece, but not around requiring services to move from FC to iSCSI.  Rather, the services - which will ultimately lead to automation technology opportunities, lie in moving from DAS to SAN - or DAS to NAS - regardless of protocol or transport.  No one seems to be arguing that networked storage and disaggregated components is not better than tightly coupled "systems" at least.

So, the good news is iSCSI is righteous, and it is happening.  The bad news is that the society we live in (IT) would benefit greatly by a more rapid adoption of this, and many other commoditized/standardized technology implementations but tread slowly because they aren't being shown the strategic potential they could gain by doing so by their big vendors - who have a natural vested interest in selling more of the same stuff they know how to make, sell, and support. 

Darwinism always works, but it takes a while sometimes.

Great Article

Jonny Oltsik's latest is superb - and right on. 

The best part is we've read this book before - I especially like his comparison of IPv6 to Y2K - which bodes well for the likes of Cisco above all - there will be a huge amount of money spent on mandated upgrades.  Did anyone make more money than IBM in the Y2K doom and gloom?

Second, his point about how absurdly over-funded startups have become marginalized as features inside bigger fish is something the storage business has dealt with for years.  CDP anyone?

Best of all, I love the prediction where public enemy number one Microsoft ends up winning - again.  All is fair in love and computing, but we spend way too much time focusing on wrong issues predicated on emotion than actually trying to win by fixing problems.  Crapping on MS security is easy - of course they are the primary target - they sit everywhere.  If you wanna be a real criminal you go where the money is, not the donuts.  If you aren't part of the solution, you are just whining.  Most of all, I dig the fact that they are going to prove me right sooner rather than later for a change - encryption is the only way to really protect information itself, and they will make it commonplace, erasing the voodoo mystique and unfounded fear of the unknown.  Will it screw up?  Of course, but give them the credit they deserve, once they start, it's only a matter of time.

Latest CW - Best of 2006

No grief on this, it's my birthday, and i'm not young anymore.  Here it is.