Knight Capital, a firm that specializes in executing trades for retail brokers, took $440m in cash losses Wednesday due to a faulty test of new trading software. This morning reports were calling it a trading “glitch”. The broad outline of the story is here and more colorful, bloody details are here.
Briefly, here’s what happened: Knight Capital’s worst day in IT started Wednesday morning with a test run of its new trading software. The company set up the software to work with only a few stocks. They also set the buy/sell points well outside where the markets were currently trading to ensure that nothing would actually execute.
But somehow – and this will probably be the subject of several lawsuits, books, and maybe even a Broadway musical – the software didn’t behave as expected. It went out and did what it was designed to do: execute lots and lots of trades very, very quickly.
Unfortunately, the trading algorithm the program was using was a bit eccentric as well. Knight Capital’s software went out and bought at the “market”, meaning it paid ask price and then sold at the bid price – instantly. Over and over and over again. One of the stocks the program was trading, electric utility Exelon, had a bid/ask spread of 15 cents. Knight Capital was trading blocks of Exelon common stock at a rate as high as 40 trades per second – and taking a 15 cent per share loss on each round-trip transaction. As one observer put it: “Do that 40 times a second, 2,400 times a minute, and you now have a system that’s very efficient at burning money”.
As the program continued its ill-fated test run, Knight’s fast buys and sells moved prices up and attracted more action from other trading programs. This only increased the amount of losses resulting from their trades to the point where, at the end of the debacle 45 minutes later, Knight Capital had lost $440m and was teetering on the brink of insolvency.
Now what is more interesting than this major screw-up are the reader comments & the theories of how this happened!! Some say Chinese Hackers did that, Some say Iran and some say bad code and bad testing….Hilarious but makes you stop & think!!
This is a test. Repeat. This is a test.
Once we get you use to this, you’ll understand how a mishap like this can wipe your bank account out.
Oh, sorry, you can’t get it back, we don’t know where it went.
You can bet the Farm that once the software problem arose, every affected IT department (probably dozens) went into scramble mode to find the problem and point fingers saying “It’s not us”. Lots of shouting and angry phone calls demanding answers and politcal fallout behind the scene. Remember this was headline news on CNBC. A bunch of folks just got fired at Knight Capital is my guess. Customer as a Beta Tester? Not in this environment.
To your point. If the algo was going nuts last night, why not kill the program? Why wait so long? Your right, something else is happening under the covers. Could be a virus, malicious code purposely written to inflict damage. Revenge? Could also be a coverup for other trading losses.
Even the most junior of coder couldn’t fuck up code this badly.
This is a hit, plain and simple.
A smart group of algo traders / coders shorted “KITE” and then dropped this little trojan into their system.
Game Over, who’s next. Be afraid. 3 G’s is right.
Daily Bail – How about – maybe you are very close (the Chinese did it). How about, the reason China Cord Blood went up several X was because (the Chinese did it) it was hacked from Guongdong and the greedy hackers were greedier than they should have been. Instead of hacking into Knight and placing trades on say, American and Israeli companies and planting the seed that the smartest Iranian did it, maybe they made a boatload of money, but knew they could double it because they were already long a native stock?
I’m probably crazy, but what I hear is that the Chinese have hacked so far through some firms that that can be done is to have the problem covered with tar paper by the NYSE and others.
Chinese hackers? No way. If that were the case, every time I checked my routers, I’d notice that ports up and down my router were being randomly pinged – all by Chinese IP addresses – or at least 95% from China, and that it happened several hundred times each day.
Oh, shit, wait – that is what I see when I review my router log.
Knight got the whack this time. More coming soon. Looking back sometime in the future, the flash and this may become known as the straws that broke the camel’s back and the timing of PIMCO’s statement might become legendary. How many investors would there be if we all knew we weren’t just getting hosed by our brokers, who had the list of all the limit orders in front of them and all other GTC trades, but that the Chinese were hacking these guys? It’s more rigged than a Macao casino!!!
We’re FUCKED! (Oh yeah – let’s give those Chinese kids scholarships to MIT – that should help us. Let’s sell Lenovo to them – they probably couldn’t learn anything from owning the original IBM computer company. Let’s give them all our plans because they can build everything at 1/10th the cost – they won’t fuck us in the end.)
Why do deals with the Chinese all seem like the first deal street kids do with drug dealers?
Can you say “Flash Crash” – No , you say – “Frash Crash”. BITCHZ!
Your are way too innocent for this site. If they were hacked do you believe that they would tell the sheeple? NO WAY, it would be hidden like you can’t imagine with the complete complicity of the US government.
Otherwise, imagine the panic it’d create worldwide if the NYSE could be affected the way it was by rogue hackers that could be sitting in Iran or China.
This type of hacking could be lot more effective weapon than any neutron bomb that couldn’t possibly be used in today’s world, thus making them only “capital cost” that doesn’t produce anything tangible.
But the only way you’ll ever find out is by seeing the USA take NO real action against IRAN or CHINA when it’s obvious to all that it should. That will tell you that they have the USA by our neck.
Until next time,
Engineer
Note: never expect positive proof for sensitive news, you will need to infer them from the actions of the powerfull .0001%
I have worked designing and implimenting secure financial systems as a lead architecht and as a project or program manager for 20 years. I’ve worked at or with FDIC, OCC, the Fed and BAC. Done some smaller consulting with a couple of smaller banks and with some other financial groups. That and I have done work for Homeland Security and the Department of Defense.
One thing I have learned is that no matter how good you think that your security is….someone with time, resources and determination can find a way in. If hackers can get to the plans for the MX Missile they can get to an HFT algo.
AND, just as with any form of terrorism, they only need to get lucky once, you have to be perfect in defence every time.
How many of these HFT’s are out there?
What is the probability that at least a few will be vulnerable.
How many would you need to take over to execute a phased attack? Three? Four at most.
If you are the atacker, you do not even need to successfully mess up the trades or stock prices permanently. From the attackers perspective it makes no difference if the NYSE backs out or invalidates trades.
ALL that matters is that the very act of doing it completely undermines the trust in the market such that even the pros will not trade without a serious level of fear. Not to mention, that multiple attacks would see the regulators and congress scream to destroy and outlaw HFT and so take a huge chunk out of profits from some big players.
Been there, seen it happen. No one created it “intentionally”. Programming errors can have unpredicable consequences. And yes, it can take hours to figure out the issue and pull the plug because the complexity and lack of knowledge. I was in that industry for 30 years.
A system can have hundreds of software components each with multiple versions. It’s very easy when you package something together to pull the wrong version. There is actually configuration software to help you manage the process of assembling a system for production, but again if you label something incorrectly a month later it can come back and bite you because the version numbers changed, and you overwrote one tiny module of the current version with a prior one, etc. It can take days to track that down.
Programmers (aka Developer) are a lot “dumber” than they used to be. They do much less today. No more design, no more testing. Just get handed a spec with a delivery date. Write code, hand off, done. Somebody else tests it. And it has to interact with other code segments, which the “Developer” has no access to. Somebody else has to find all the problems before it’s placed into real-time service, and work with the Developer to solve them. And if they don’t…you can have a disaster like you did today.
Could even be a spec (design) error. Todays “Developers” are trained not to think, just code to spec and deliver. In other old days you actually had to understand the business. Not anymore. So, if somebody just verified this met the spec and signed off and then went live, well, they all probably just got canned for not catching a problem the Spec had.
… and the really funny bit is that the specs are usually crap, because the technical analysis they were based on was crap, because the business analysis it was based on was crap. One is more likelly to win the lottery than to find a Business Analysis document that doesn’t contain contradictory requirements.
The Business Analysts are ex-sales types that don’t know shit about an organized, structured gathering, analysis and validating of business requirements …
The Technical Analysts don’t really exist and instead are some senior coder or other who has never worked in more than one or two systems and never saw enough to really know just how many ways there are to fuck up a system design …
The Programmers are some cheap guys in India who went into coding because you make a lot of money as a Programmer over there and have ZERO natural ability for it, while the few really good ones have been promoted to Project Managers (which they suck at) because over there the salary framework is so completelly screwed that an exceptional programmer makes less than a crap project manager (and, trust me, the management style in India is ridiculously bad).
It’s a damn near perfect shit-in-shit-out system.
In my experience, the way Investment Banks try and make up for this is to use at least 3x more people as a Tech company would for the same results (I’ve worked in both industries).
everybody jumps to the conclusion of a rogue algo and millions of losses, but what if this was a (second) test of a (new) algo and it performed as expected? Or the algo has been working successfully for some time and now got exposed for some reason?
Some stocks were bid up, others sold down – what if the net result of these 30 minutes of crazy trading was in fact ZERO, for instance two bots playing 148 pingpong games simultaneously in concerted action? e.g. Bot A flings stocks to Bot B in 100 share lots, increases the price at some subpenny step and then Bot B flings it back to Bot A at the same increment, then this loop is repeated non-stop until a new increment is added – intermittedly some real trades are executed with the “outside world”, most probably at a small gain or a zero balance. Some stocks traded millions of shares where on a normal day a few thousand are exchanged only. How can we conclude here that the number of trades was all REAL?
Meaning they can move any stock to wherever they want in just minutes, the price of a stock no longer matters – the real gains may have been taken in derivative plays that depend on the level reached by the manipulated stocks. Meaning this was not uncontrolled but planned. And this type of manipulation could have been going on already for months, if not years. If this is true, they may very well have shorted their own stock, offsetting the loss of company value by a multiple gain in a leveraged derivative play.
After Draghi spoke, the whole market moved up parabolically and subsequently sudden non-stop buying frenzies were unleashed on no additional news, pushing indices further up – by whom or by what? There were no drops, like everything was bought above a certain ask price along ever higher support lines. This was partly short squeeze driven, but perhaps the above scenario was running? Derivative interest and currency plays paid out big time last week, for those in on it at least…
I agree it doesn’t make sense that this firm let this happen without noticing it for 30 minutes – i mean, they all went for coffee?
smells like bullshit yes
You must be logged in to post a comment.