Trip Report

Life is weird, thankfully.

Months ago I shot my mouth off to IBM storage mucky  muck Andy Monshaw about something or another, and the outcome was that we would go break bread, drink drink, and solve all problems.  If that didn't work, we'd do the former and I'd listen to him bash EMC.  Andy has an almost worrisome, deep-rooted hatred of the Hopkinton folks, and it's good fun to get him spun up on something they did or didn't do. Or might do.  Or might think about doing. 

So after endless schedule massaging by poor people on both our ends, a date was set for last Thursday in Manhattan.  Wouldn't you know that day was also going to end up to be Game 2 of the Franklin Little League National League playoffs - and the Dodgers, of which my 12-year old Jason is a proud member of, were up 1 Game to nill.  I was bumming that I was going to miss the game, but a date is a date.  Plus, we were up one already.

I arrived via train (which is a far more civilized way to get to NY until one can rake in enough dough to fly private) right on time, and headed off to 590 Madison ave, where I was shuffled off to the 17th floor of some ridiculously expensive real estate.  Upon exiting the elevator, I was greeted by a lovely receptionist who sat seemingly alone amongst pieces of art that probably were individually worth more than my entire collective being.  There were no people anywhere.  Quite a lovely spot, however.

Andy appeared out of some hidden wall somewhere and we got down to business.  I won't tell you what was discussed, but suffice it to say that the conversation turned lively and interesting - not adjectives typically used when describing anything IBM.  We were joined by Mary Coucher, VP of Biz Dev for Andy's group - a downright smart, passionate, and non-standard cookie cutter IBM'er.  I thoroughly enjoyed myself.

Prior to dinner we stopped at some roof top hangout where one of Andy's "comms" people was having a going away party.  I don't know what a "comms" person is, but there were lots of them.  While Andy was kibitzing over a Jameson or two, Mary and I worked on solving world peace.  We got onto the subject of next generation storage applications like video surveillance and the home media market, which got us to what each of us watches on TV (recorded, of course).  She has Desperate Housewives, Curb Your Enthusiasm, The Soprano's, and 24. As she is saying this my phone rings, but I don't answer it.  A few minutes later I take it out, and on the screen is John Slattery's number.  Slats is my college roommates bro, who I happened to call the day before as he lives in NY to see if he could hook up for a beer.  He is also "Dennis" on Desperate Housewives and has been making out with Eva Longoria for the last few months.  I was curious to research just how hard work that was.  Plus, he's married to George Clooney's X, and that alone is worthy of discussion. I've known him forever, and even though he's been in a million things, no one really knew him that much until this gig.  Now he's a regular in People and my 15 year old daughter is trying to hook me up with her.   

My phone rang again, and this time it was my wife.  Mary was chatting with someone and Andy continued to kibitz with his whiskey, so I answered.  She proceeds to tell me how Jason just hit his first home run - a monster 3-run shot to spark a slow rally to tie the game and send it to extra innings.  I was psyched, and completely horrified that I missed it.  They ended up losing in the 7th or 8th, but Jason didn't care.  He spent 10 minutes describing the whole thing - clearly out of his mind happy.  Made me feel like crap.

Anyway, I was knee deep in IBM sport drinking by then, and still had dinner to attend, so I left well enough alone.  We all went off to some Mexican place (that was out of this world, but of course I can't remember the name of it nor where it was) and argued everything from CMOS to Salsa.  It was entirely refreshing to engage in debate with very smart IBM folks who weren't just pushing their tired old "we're IBM so we are right" rhetoric.  Andy, Mary, and Carl (Andy's "ops" guy, whatever that means) were definitely willing to listen - and not automatically assume they were blessed with divine correctness.  I've had IBM executive discussions that have been wildly irrational - the kind that end up sounding like a business version of the Black Knight scene in The Holy Grail.  This, thankfully, was not one of them.  Perhaps there is hope for the mighty Blue Knight after all.  The way they were talking, I can't wait to see what they come up with, as they were downright giddy with how "un-IBM" some of their impending moves were going to seem.

So I leave the joint and am about to jump into a cab when Carl, the Haitian French speaking Andy Ops guy tells me that I just need to walk up a block or two and over a block or two and I'll be back at my hotel, The NY Palace.  43 miles of poor shoe walking shin-splint inducing hard labor later I found the hotel.  It was still early, and I wanted a nightcap.  The Palace is a beautiful hotel, but one that tends to frequented by 80 year olds with zillions of dollars who must head off to bed by 10 because the 6 seat bar from the Shining tucked into the corner was all closed up.  Figuring that this is NY I just popped out to find a watering hole with a bit more zip to it.  I rounded the corner and ran into the W on Lexington - which is a modern joint that caters to 20 year olds with zillions of dollars.  Upon entering the Whiskey Bar, I realized that I was too old, too ugly, and too lazy to fight my way to a glass of wine.  I gave up and went back to the Palace.  When I got there, I asked the doorman why the Shining shut down at 9pm?  I half expected him to call me Mr. Torrance.  He told me that the real bar - Gilt - was on the second floor, and that was open and ready for me.  I found it, and it was perfect.  Neuvo, Ian Schragerish joint without a big crowd, and with the best per glass wine list I've ever seen.  With the exception of the art deco purple chunk of art that made one end of the bar look like you were inside a golf ball, the place was great.  I drank my $35 glass of a nice Barolo and headed off to bed.

I headed out early to catch the 8am express to Boston.  As I'm walking through Penn Station I notice this guy leaning against the stand where the cops and soldiers are always hanging out - right outside the Acela express club.  It was Jeff Garlin - who is Larry David's manager in Curb Your Enthusiasm.  In an amazed state (this was number two on the list of discussed TV shows the previous evening), I say "Hi Jeff, I love the show", because A: I do - it's one of the funniest shows ever put on TV, and B: because you say that even if it weren't true, because that's just what you do.  An hour later I'm in the cafe car on the train (and I don't care what anyone says, that train food, nuked in a plastic bag, is fantastic) talking to my wife on my cell about seeing Jeff, and doesn't he walk right in and stand in line 12 inches in front of me.  Not being shy, I say "hey Jeff, say hi to my wife" and make him take my phone.  My wife has no idea what I've done, happily continuing her tale of how the dog ate the curtains or something. He patiently listens to this, until he gets a chance to say hi and introduce himself.  My wife loves the show as much as I do, so it all worked out.  He couldn't have been a nicer guy.  He also lost about 50 pounds and had wiffle instead of his big curly hair.  I almost gave him a show idea based on my real life battles with the recycling people, but figured I had worn out his hospitality.

Friday night the Dodgers (our squad) won the first series.  Every person there felt compelled to tell me what a great hit my son had.  Too bad you weren't there..... Sunday we played game 1 of the NLCS.  Trailing 5-4 in the bottom of the 6th (we only play 6 innings, so it's really the bottom of the 9th), our superstar Kyle draws a walk.  Our #3 hitter, Frenchie (a 2 handicap - which is supremely annoying), smacks a double and brings Kyle home to tie the game.  Our cleanup hitter strikes out.  Now it's Jason's turn.  My son, god bless him, inherited all of my genes, the poor bastard.  He is smart and funny, but short and stumpy.  He makes me look tall.  He is slightly faster than a large office building.  He is good looking though.  Anyhow, Jason rips at the first pitch for a strike.  He looks at strike 2.  He fouls off the next pitch.  Even in Little League, you don't want to throw a strike when its 0-2, and this guy just got away with one.  He grooves the next pitch, and Jason belts it over the fence for a walk-off home run and team victory.  As he goes by me at first, he says "happy fathers day. I didn't get you anything else".

How cool is that?

There Are Only Two Kinds Of Data - Dynamic And Persistent

Last month I explained my theory of why we’re so screwed up infrastructure-wise or at least how we got to this point. This month I’ll try to show you the way out of the situation.

For a few minutes, forget everything you know or at least everything you think you know. Accept my argument that almost everything we’ve done in commercial IT has been based on transactional requirements. Open your mind.

There are two distinct types of data: dynamic and persistent. Dynamic data is in flux; this is where transactional data begins. Persistent data is fixed. It’s what it is and will never be anything else.

Just because data is dynamic doesn’t mean it starts and dies within an RDBMS. Structured database data starts as dynamic, but at some point it becomes a nonchanging record. It’s persistent. You may have reasons to keep it inside a database forever (although I doubt they’re valid ones), but those records are still persistent; they are what they are.

Here are a few rules that will help you:

Rule 1: Don’t confuse how something begins its life with how it will end. Everything begins dynamic and ends persistent. Stop delineating between structured, unstructured and semistructured. All types live dynamically for some period, whether it’s a Word document, a movie, a credit card transaction or an email. It all ends up as fixed digital content.

Rule 2: The attributes and requirements for each type of data are different. Read/write performance, throughput, redundancy, DR, etc., count more in the dynamic phase of data life; however, we’ve extended all of those philosophies to data that has stopped changing. Building data redundancies and protection schemas to handle real money transactions is good business; backing up a nonchanging data element a thousand times isn’t. Having your bulletproof transaction system capable of handling all the dynamic money events thrown at it is good business. But adding processing power, capacity, network infrastructure, etc., to keep it churning away rather than removing the 90% of the data that isn’t dynamic and can interfere with the real transactional stuff isn’t.

Rule 3: The ratio of true dynamic data (and data being “treated” dynamically) to persistent data is approximately 1:10, and that ratio will rapidly evolve to 1:100 and beyond. Dynamic data just doesn’t stay dynamic for very long.

Transactionally oriented systems are all about doing things fast. Perform the transaction fast, store the data fast and load the data into other systems fast. If it sits in a database, it’s easy to find, which is the point of a database. The persistent data world is all about finding things. The whole categorizing/ classifying/indexing/search thing is designed to add structure so we can find things. However, it seems to me that if we created two distinct “virtual” places to look for each distinct type of data, it would be a heck of a lot easier to find what we want. If all our dynamic data sat in one place designed to handle things like that and was then moved (based on business rules) into the persistent digital content store, we’d be able to architect this store entirely differently than the dynamic store.

If the dynamic store is about speed and redundancy, the persistent store is about infinite dynamic scale, finding things easily and quickly, and an autonomous self-managing/self-healing infrastructure. It should also be cheap to buy. Stop trying to turn the dynamic store into the persistent one, and also stop trying to make the persistent store dynamic. If you act differently, you’ll realize you can get back to making IT a competitive advantage.

Here's the follow on to last months' article, as seen in Storage Magazine.

A Practical Example Of How Process Change Will Save Huge Dough....

Lets assume we are Joe IT guy in the ABC company – an upper middle market company with a few thousand employees, a dozen sites, and all the problems folks like us deal with.  We run our transactional production systems and our distributed windows stuff.  We have big SANs and file servers.  We have stuff everywhere.  We back things up, we do some DR.  We are tragically overworked and chronically undervalued.  We are Joe IT.

Let’s pick one little area where process improvement can yield big results – Test and Development.  Everyone has T/D operations.  Most operate in some version of the following:

1. T/D is used to make sure internal and external software applications, new infrastructure, and upgrades work the way the are supposed to prior to them being rolled out into production.
2. In order to perform step 1, T/D has to get real life data that is complete and current from production systems into their own systems.
3. In order to perform step 2, T/D has to beg, borrow and steal – and lives at the mercy of the production people.  It takes time, planning, and prayer.  The application normally has to come down, the database quiesced, the infrastructure specialists turning knobs and pushing buttons, and the data moved.
4. Once step 2 is complete, the actual testing can occur.  Usually T/D will make additional copies of the data sets (which are probably already out of date by the time they are used) to test different things.

Now, let’s talk about the practical inefficiencies of what just happened.  First, the production system runs some application(s), most likely on top of a database.  We tend to find our production data really important, so normally we would have at least two copies of that data.  We tend to keep our production data on very expensive, very big, very power hungry infrastructure.  We create the data on that infrastructure, and then we keep it there.  We then make more copies of that data, also tending to keep it there.  We might have 2, 3, 4 + copies of the exact same data, at various points in time or at the same point in time.  That means our cost of housing that data is 2, 3, or 4 plus times the cost of housing it as a single instance – and not just for capital – but for power, cooling, footprint, etc.

When we make our test copies, we also tend to create and keep those copies on our production system.  Sometimes we delete them when we are done, but sometimes we leave them around for a long time.  Sometimes we forget about them altogether.  Those copies take up more space, power, and brains.  Those tend to be backed up along with everything else on that mission critical production system.  That means we might be backing up 23 copies of the exact same data in one single backup.  If we do a full backup each week, we will create new backups of the exact same 23 copies of the exact same data each time.  You see where I’m going with this?

We have test servers that we run that sit around sucking juice whether they are used or not.

It gets worse.  Not only was the process of getting the database copies difficult, and expensive, but no one considers the security implications associated with having potentially hundreds of copies of real production data floating around – in the production system, in the test systems, on the backup systems, at the DR site, and on the tape at Iron Mountain.  Backup is good, right?

Having 2, 10, or 100 copies of the same, non-changing data sitting on “production” systems isn’t even the real problem.  The downstream effects are the issue.  Extra stuff on the production system slows down the production application, slows down the database, and slows down the user.  We tend to combat this by buying more hardware.  Bigger, faster hardware that sucks more juice, takes up more room, and causes more disruption.  More “data” means other processes suffer – networks get clogged so we need more bandwidth, backup servers get bogged down so we need bigger machines, backup targets get full faster, etc.  The rest of our processes don’t know or care that it’s the same data – only that there is more.  More causes problems.  Problems cost money. 

The conundrum is that if you are the vendor in the production system, you kind of like it when Joe IT calls for more stuff.  It’s hard to tell Joe not to buy more – but instead to just change some of the behavior that causes the problems.  If we were to try to really help Joe, however, we’d lay it out like this;

1. Define the objectives for Test and Development – in a perfect world
   a. Get a complete, accurate, and timely copy of the exact production database
        i. Zero impact – non-disruptive to production
        ii. Non-disruptive to production IT
        iii. Automated
   b. Put that copy somewhere else – NOT on the production system
        i. Don’t create work for production systems or people
        ii. Do it dynamically
        iii. Run virtual machines everywhere you can
   c. Create a protection policy for the Test/Dev data
        i. Do we back it up? 
        ii. If so, when, why, and how often?
   d. Create a security policy for the Test/Dev data
        i. Protect the assets as if it were still in production
          1. This assumes you are not TJX or TSA
        ii. Enforce disposition/destruction
   e. Define a Data Repurpose Policy
        i. Who else could use a copy of this data?
          1. Should we use this copy as a backup copy?
          2. Should we replicate this copy as a DR copy?
        ii. Are there other applications that could use this?
          1. Data Warehouse
          2. Business Intelligence
          3. Those guys in marketing
          4. Business partners

If Joe did this, nice things happen.  By simply grooming the production systems of Test/Dev copies, Joe unclogs a lot of space, performance, and all the other associated costs.  Joe just took a universal pain in the rump from the production IT staff and made it disappear.  The Test/Dev folks have a consistent schedule to get fresh whole data sets to play with.  The security people are happy that there are less copies of stuff everywhere.  The backup guy is happy that the extra stuff isn’t killing his systems.  The Finance guy is happy because the cost of the test/dev infrastructure is an order of magnitude lower than the production stuff.  The licensing costs alone for running the applications and database on bigger machines are huge – and now they will be pushed our or even go away.

If Joe did this, he might also figure out that not only should ABC keep Test/Dev data off of production – but that even the production data itself is “groomable”.  90% of the data that makes up production is fixed content, or data that isn’t going to change.  It is no longer “dynamic”, and as such, the same questions can be asked of it.  If it isn’t going to change, should we still back it up the same way?  Should we still have 4 primary copies of it?  Shouldn’t we put it on infrastructure that has attributes that are more aligned with the current state of that data? 

If the data isn’t changing, but the attributes are – then continuing to do things the same way is illogical.  If, after some period of time, access of a certain piece of “formerly transactional” data went from frequent to never, why don’t we put it through the same exercise that we did with the Test/Dev data?  If we moved that data out of “production” physically, but maintained its place logically (i.e. we could still access it in the same way if required – but it might take longer to show up) – we could then do some really interesting things.  By delineating our production data into Dynamic and Static (fixed) buckets, under a single consistent logical view, we would change things forever.  Our static data would have a different “lifecycle” – with different attribute requirements as time (or whatever metric) moved on.  We would move it out of our “tier 1” outrageously expensive gear and onto much lower cost gear, which would mean that our tier 1 gear would perform at optimal levels all of the time.  In theory, if our Dynamic to Static flow was predictable, we may never have to buy another piece of hardware or upgrade another license for our production systems again.  We would not only move data off of our most expensive, mission critical stuff, but we would change the attribute requirements of that data when we did.  For example, we might keep 4 primary mirrors of that data while it is dynamic, and back those up every hour to a disk target, and replicate those targets every 8 hours off site, etc. – but we’d stop doing that once the data became static and moved to the “static production” state.  Perhaps we would make sure we had one mirror locally, and one on-line backup locally, and another copy at the DR site, with 3 “oh no” copies on tape.  Then we’d never back it up again – because what would the point be?

If you did that you’d save a lot more than 25% of your power and cooling budget.  You would save so much money you would probably affect the earnings per share of your stock.  You might want to negotiate a bonus up front!

Vendors, who naturally are opposed to such intelligence unless they aren’t the ones getting all of your money, are starting to wake up.  It’s hard to upset the gravy train, but they are starting.  The fact is, the very nature of data is changing, so just jamming more of the same down your throat might help them short term, but eventually will cause them more problems because you won’t be able to ingest any more stuff from them.  They are already seeing it.  Those that learn to cannibalize themselves are the ones who survive long term.  The principal of adaptability in Darwinism works.

IBM bought Softek to do things like this – at least some of the things like this.  They are the ones who are at the very core – they are the Mainframe.  By providing customers a way to groom production systems, they take food out of their own mouths.  There have been heated arguments inside the blue machine for just this reason.  The bet is that while they may push off mainframe revenues, they pick it up in other areas.  EMC owns tons of the storage market attached to those mainframes.  Oracle gets a big chunk of that pie too.  By helping customers groom their production worlds and create new efficiencies, IBM takes money away from EMC and Oracle – and gives themselves a shot to compete for that storage business, and the data management functions that Oracle provided.  Network appliance has been working with Solix to pull things out of those same environments and have them land on Netapp boxes.  Not only is a Netapp box a heck of a lot less money than a big giant tier-1 mega array, but once the data is there people are astounded at how easy it becomes to do really useful things.  One big wig at a Fortune 1000 told me that they could now create zero-footprint copies of their production data instantly using Netapp’s writable snapshot, which lets them do things that haven’t even been remotely possible previously.  He said “it was like we hired 4 more people with the time we saved – and now things that took days take seconds.”  Most folks really can’t afford to stop production to do backups or run reports, but they have had no choice.  Now they do– and they now do it securely.  EMC won’t like losing the core disk business, but if they can take the server dollars, the application/database dollars and get the customer to see the light, they have a whole new opportunity to sell the other thousand things they have in their bag.  If it’s fixed content, why not let Documentum manage it?  Who needs Oracle? It’s sort of like being a stock trader – you don’t care whether it’s up or down, only that a transaction is occurring.  If you are an “emerging” player – Isilon, Pillar, or anybody else – this is an opportunity to shine.  If you are a little guy trying to displace NetBackup, this is your chance.  Real change opens doors.

Process changes of this magnitude are truly game changing – for IT and vendors alike.  It tends to take a massive event to get either side to move out of their comfort zone however, and not being able to buy any more power might just be that event.  Once something like that occurs, and people are forced to look at new ways of doing things, and then Pandora’s Box might fly open.  With the lights turned on, people can see.  Sometimes things don’t look as good with the lights on – just ask my wife.

Think Differently, Live Better. (And Drive Fast Cars)......

In last month's Storage Magazine article I explained my theory of why we are so screwed up infrastructure-wise, or at least how we got to this point.  Now I'll give you the way out.  Then, I'll talk about cars.

Forget everything you know for a few minutes, or at least everything you think you know.  Accept my argument that most everything we’ve done in commercial IT has been done based on transactional requirements.  Open your mind.  Breathe through your eyes, Danny.

There are two distinct types of data – dynamic and persistent.  Dynamic is data that is in flux – this is where transactional data begins.  Persistent data is just that – fixed.  It doesn’t change.  It is what it is, and will never be anything else.

Just because data is dynamic doesn’t mean it starts and dies within an RDBMS.  Structured database data certainly starts as dynamic – but at some point it becomes a non-changing record.  It’s persistent.  You may have reasons to keep it inside a database forever (although I doubt they’re valid ones), but those records are still persistent – they are what they are. 

Rule #1:  Don’t confuse how something begins its life with how it will end.  Everything begins dynamic and ends persistent.  Stop delineating between structured, unstructured and semi-structured.  All types live dynamically for some period, whether it’s a Word document, a movie, a credit card transaction or an email.  It all ends up as fixed digital content.

Rule #2: The attributes and requirements for each type of data are vastly different – in most every sense.  Read/write performance, throughput, redundancy, protection, disaster recovery, etc., count more in the dynamic phase of data life – but we’ve extended all of those philosophies to data that stopped changing, which causes 90% of our operational issues.  Building data redundancies and protection schema’s to handle real money transactions is good business – backing up a non-changing data element a thousand times isn’t.  Keeping your bullet proof transaction system capable of handling all the dynamic money events thrown at it is good business. But adding processing power, capacity, network infrastructure, etc., to keep it churning away rather than removing the 90% of the data that isn’t dynamic or possibly even relevant that can interfere with the real transactional stuff isn’t.  Thinking you can’t make this happen is wrong.

Rule #3: The ratio of true dynamic data (and data being “treated” dynamically) to persistent data is about 1:10 – and that ratio will rapidly evolve to 1:100 and beyond.  Dynamic data just doesn’t stay dynamic for very long. 

Transactionally oriented systems are all about doing things fast.  Perform the transaction fast, store the data fast, load the data into other systems fast.  If it sits in a database, it’s easy to find – which is the point of the database.  If we could, we’d have everything inside a database, but they weren’t designed for that.  The persistent data world is all about finding things.  That means you need to know where to look.  Having stuff everywhere, inside and outside of the building, makes this task all but impossible.  The whole categorizing/classifying/indexing/search thing going on today is designed to add structure so we can find things.  It just seems to me that if we created two distinct “virtual” places to look for each distinct type of data, it would be a heck of a lot easier to find what we want in either.  If all our dynamic data sat in one place designed to handle things like that – no matter if it’s Word or big dough – and then was moved (based on business rules) into the persistent digital content store, we’d be able to architect this store entirely differently than the dynamic store. 

If the dynamic store is about speed and redundancy, then the persistent store is about infinite dynamic scale, the ability to find things easily and quickly, autonomous self-managing/self-healing infrastructure--and it should be really, really cheap to buy.  If the dynamic store requires knob-turning specialists, the persistent store requires the occasional dusting.  Stop trying to turn the dynamic store into the persistent one – and stop trying to make the persistent store dynamic.  If you think differently, you’ll act differently.  If you act differently, you’ll realize you can get back to making IT a competitive advantage.

And Now For Something Completely Different....

If I had stupid money, I'd buy a Mercedes SL65.  The car wouldn't get noticed at the shopping mall, but it is a 700HP rocket ship.  Silly, really.  It is the very same car my girlfriend, Lindsay, just smashed while being smashed, out in L.A. 

I stumbled across this thing called "Supercar Life" (www.supercarlife.com) in one of the high end rich folk magazines I read and drool over.  It turned out (in a very legal, tax deductible way) that one of my biggest customers is a car freak, so as a purely business oriented gesture, I took him to this.  The owner, Joel, a guy with way too much dough on his hands, figured out that there are a lot of "men" who still act like they are 12 when it comes to cars.  So, for a piddly $5,000 you can fly to Pittsburgh where he rents an entire race track (Beaverun), and only lets 10 folks on it.  He has (2) of each - brand spanking new - Ferrari F430, Lambo Gallardo, Porsche 911 Turbo, Astin Martin DB9, and the SL65 (Joel had his $450,000 Porsche Carrera GT sitting out front, but didn't offer to let me drive it for some reason). There are about 6 or 8 real race car driver guys - not local car club pro's - real dudes that race really expensive cars really fast that teach you how to not destroy the very nice cars that you don't own.  Every few minutes, you swap cars.  By the end of the day, you have driven a million bucks worth of cars really, really fast.  It was awesome.  If you are thinking of buying one of these rocket ships, why not spend an entire day test driving them first, and learning how not to "Lindsay" your ride?  It was really first class - they put you up at the Marriott Resort and Coal Mine, feed you well, and were just terrific folks.  My customer may have had to have surgery to remove the smile from his face.  Now I just need to find some other car crazy customer who gives me piles of dough to justify my return trip.....