Posted May 10, 2005: MORE ON COMPETITION

From communications with competition winners Savani and Veal, I've come to realize that while the May 8 description of their strategy's behavior is correct, they also implemented a number of other interesting pieces of strategic functionality. I've invited them to give me a report on what's under the hood, which I will post here when they've given it to me.

Meanwhile, a friend and colleague of mine who works on Wall Street read the results with some interest, and offered the following comments, which I think are well worth reading:

Interesting :).

I think it does raise some "real-world" issues that are very important
when designing automated trading strategies.  While it's probably
unlikely for one strategy to so completely manipulate the market in the
"real world" (since there are for example any number of non-automated
market participants that will recognize the price anomaly), you are
right that it is not entirely unrealistic, particularly in stocks that
are not highly liquid.  

I think what it does illustrate very clearly is how much trouble an
automated strategy can get in if it does not have sufficient "reality"
checks built in.  It is always hard to think of everything and to make
your strategy have the street smarts necessary to handle unusual and
unexpected situations but we've certainly witnessed firsthand the
dangers that a strategy can get into when it doesn't handle the
unexpected well.

One real life anecdote that occurred just a few years ago...A market-making
strategy, in addition to looking at order books, etc, also looked at the
Nasdaq futures and had some sort of correlation matrix with regards to
which stocks tended to follow the movement of the Nasdaq futures.  So
the strategy would make markets but then adjust its markets, making one
side more aggressive and thereby getting long or short,  if it saw the
futures move without the stocks following.  However, the futures market
also has a concept of "limit down" which is a slowdown mechanism after a
large futures move.  If the futures move the specified amount, then they
cannot trade lower for 10 minutes.  If they have not moved higher at
that point, then they halt for 2 minutes and then re-open with a
lower/wider limit in effect.  On side effect of this, though is that the
futures bid price appears as 0.00 at this time, since there is no bid in
effect (i.e. the offer price is at the limit/threshold level).  However
the futures bid price becoming 0.00 tricked the strategy (which had not
planned for this scenario) into thinking that the futures were "really
cheap" and it therefore started shorting all of the correlated stocks
fairly aggressively, since they had not yet reached 0.00!  

Fortunately the real-life story didn't end too badly... The strategy was
trading relatively small size at the time and the runaway behavior of
the strategy was caught pretty quickly and it was shut down.  But it did
definitely serve as a wakeup call that as smart and efficient as
automated strategies can be, they do lack fundamental common sense.  I
suspect many of your students had a similar reaction to their experience
during the competition.  They probably felt as though their highly
sophisticated trading strategy lacked the basic common sense that even a
relatively naive human trader would probably have.  Maybe not such a bad
lesson to learn this early in their careers :)

Nonetheless, congrats to the competition winners.  They also point out
another valuable lesson: ... Often a strategy is successful because
it anticipates how the other market participants are likely
behave/react and then exploits them.

Posted May 8, 2005: COMPETITION RESULTS

For some time now I have been exhorting PLAT participants to think in a strategic or game-theoretic manner regarding their strategy design --- that is, to think carefully about the population of strategies you might be competing against, and (at least partially) incorporate the potential weaknesses of oppponents into your design. Indeed, this year's competitions --- in which the market behavior was heavily influenced by a set of highly stylized technical strategy opponents, as opposed to real market data --- particularly encouraged this kind of reasoning.

So it was perhaps inevitable that some team would exploit this line of thought. And indeed this happened in our recent competition in simple but dramatic fashion. The crushingly dominant performance of a strategy submitted by Rahul Savani and Ben Veal exploited a universal flaw in its opponents. In the interests of full disclosure, it should be noted that Savani and Veal are doctoral students at the London School of Economics. However, their exploit, while clever and well-executed, is not rocket science. My conclusion is that graduate education, along with other intangibles, sharpens ones skills at devious thinking.

The strategy of Savani and Veal is simple to describe and even elegant in its own twisted way. The basic idea is to "clear out" one side of the market --- for instance, to simply buy all shares in the sell book. This has the effect of leaving a buy book, and thus a bid, but no sell book, and thus no ask.

The next step is to immediately place a buy and sell order at a very large price --- larger than the highest price paid to clear out the sell book. Since there is no ask, and the bid is far below this large price, this pair of orders becomes the new bid and ask, effectively leaving the current buy book far below the bid/ask.

The third step is to then self-execute a small number of shares with the new bid or ask, thus causing the last execution price to also be near the new large bid/ask.

The effect of these three steps is to (a) leave the strategy with a large long position (from the initial purchase of the sell book), and (b) move the bid, ask, and last price to a price far above the prices paid to acquire the large long position.

You can see where this is heading. Any strategy that only places orders with limit prices relative to the current bid, ask, or last execution price will blindly follow the artificial inflation in the market created by these steps, and begin trading near the new price. As long as there is enough such liquidity at the new inflated prices --- and in the recent competition, there was plenty --- the Savani and Veal strategy can then quietly start dumping its long position for far more than it paid for it. Genius incarnate.

The spreadsheet detailing the competition simulations is available here. In absolutely every setting Savani and Veal are dominant. Note that in all the planned simulations, there is a "background" liquidity-providing agent that works entirely on relative prices, and thus will guarantee the aforementioned phenomenon. One might argue that the strategy thus did not exploit its PLAT opponents so much as a structural property of the competition. Au contraire. We also ran sims among only the submitted strategies (no background liquidity or technical strategies). Again the Boys from LSE were dominant. The upshot is that just about everyone was taken in by this strategy, submitting relative prices without checking whether the absolute price "made sense" --- e.g. whether a precipitious, suspicious or unjustified rise had occured in the market recently.

It is worth thinking about how "realistic" the LSE team's strategy is. It's not as unreasonable as one might think, but one observation is that in a liquid stock it would require being a large player, and to have high tolerance for risk.

We note that the Penn team of Viren Kumar and Mohit Mutreja --- the winner's of December's competition --- submitted two strategies to the recent competition. These two were the only ones that finished in the black in the presence of Savani and Veal. We also ran the entire set of simulations with all of the submitted agents except Savani and Veal's, and again Kumar and Mutreja excelled.

So our May 2005 champions are Savani and Veal; congratulations to them both, and to Kumar and Mutreja as well.

Best wishes for the summer to all!

Posted April 22, 2005: ANNOUNCEMENT OF COMPETITION

We will indeed be holding an end-of-term competition. The initial details are as follows:

The new server for the PLAT project, plat.cis.upenn.edu, is up and running. Detailed instructions for logging in are provided here. If you follow the directions you should be able to find your files, which were ported over from crux.

I am optimistic that we will be able to proceed to hold an end-of-term competition. But for now, I encourage all of you to log in, run your strategies to make sure they are working, etc. It is inevitable that there will be kinks in the new system, so the sooner you run it through its paces, the sooner we'll find them.

Posted March 15, 2005: CRUX STATUS AND THE COMING MONTH.

This note is intended to provide a bit more info and status on our state of affairs.

First of all, when Elliot said in his recent note that crux was down, that was putting it mildly:). Crux was hacked by unknown parties and apparently had its root compromised; for a while it was being used to launch denial-of-service attacks.

As far as we could tell, no data or code was damaged or deleted (though it is possible we could discover otherwise later). However, the fact that its root was compromised essentially means that crux is an unsalvagable machine; it cannot be brought back online safely and must be scrapped.

We currently are waiting the arrival of a new server (which incidentally will be considerably more powerful, as crux was a few years old). Once this machine has arrived and is configured, we'll have to take several steps:

I am hoping that all of the above can be accomplished in time for us to have a competition in late April, and that you'd have time to prepare for it properly. However, since some of the steps (e.g. delivery of the server) are out of our direct control, it's possible this won't happen.

So what should you do?

First of all, if you are an external research participant or other "voluntary" participant (i.e. not getting any kind of Penn course credit), you should just sit back and wait for the new server to arrive and get running.

If you are a Penn senior design or independent study participant, you need to prepare for the possibility that we won't have a competition, and that your final report will need to have significant material not dependent on simulations and competitions. While I am open to a variety of material in this regard, one promising avenue is to try to do more formal analyses of the strategies you fielded in the December competition, and to suggest improvements to those strategies. In particular, I recommend trying to answer the following questions as precisely and analytically as possible:

I will provide more concrete and detailed suggestions along these lines over the coming weeks, especially as it becomes clear whether we will be able to hold an April competition or not. However, serious thought on the questions above will be beneficial to you in any case, and should constitute a significant portion of your final report.

Posted December 20, 2004: COMPETITION DESCRIPTION AND RESULTS.

Late last week and over the weekend, my doctoral student Elliot Feng and I designed the formal structure of the competition, and ran all of the required simulations. Many thanks to Elliot for his hard work.

Each submitted strategy was run in a total of 13 different simulations. To describe these, we will introduce some simple notation.

First, in each of the simulations one of the following background agents was present:

The technical strategies you have been examining and experimenting with for some time will be denoted CB for channel breakouts, MA for moving average, MM for market maker, MO for momentum, and RSI for relative strength index.

With this notation, here are the 13 simulations your strategy was run under:

The following two spreadsheets summarize the results for all 8 submissions:

There is a column for each strategy, summarizing performance in all 13 simulations. Your strategy is denoted by X in the code for each simulation, which are given in the same order as above. The last few rows give summary statistics, including the Sharpe Ratio, which is our main performance criterion. Rankings among the pool of 8 submissions are also given; these rankings change significantly after the liquidation penalty is applied.

THE RESULTS

The undisputed champions and Winners of the December 2004 Penn-Lehman Automated Trading Project Competition are the team of Viren Kumar and Mohit Mutreja, both joint SEAS and Wharton students. Their winning strategy combined market-making adjusted by volatility, momentum and contrarian substrategies. They liquidated completely in every simulation, a source of great difficulty for most entrants. They were one of only two teams with positive Sharpe Ratio after the liquidation penalty. Congratulations Viren and Mohit!

The runner-up team, earning congratulations and Honorable Mention was a spirited submission from abroad, namely the team of Ben Veal, Rahul Savani and Raju Chinthalapati who are graduate students at the London School of Economics. This team's strategy also achieved perfect liquidation and positive Sharpe Ratio. Congratulations Ben, Rahul and Raju!

Here are some general comments on the competition:

Thanks to all for participating! Enjoy the holiday break, and see you next term.

Posted December 14, 2004: INSTRUCTIONS FOR SUBMISSIONS TO COMPETITION.

This entry contains detailed instructions for submitting a strategy for the coming PLAT competition. Please follow all directions carefully.

We have created the publicly writeable directory /home/pxs/PAT/Dec04 on crux. In order to submit an agent, you should follow all the following steps by 6 PM (Philadelphia time) tomorrow (Weds Dec 15):

Failure to perform any of the steps above could result in your strategy not being entered in the competition.

Once we have your strategy, we will run some preliminary tests to check for basic problems in execution, and inform you of them if we feel they might be fixed by you quickly. The competition itself will be described and results reported as they unfold, most likely starting over the weekend.

Posted December 8, 2004: FURTHER COMPETITION DETAILS.

The deadline for submitting both source and executable code of your strategy for the coming competition will be 6 PM Weds December 15. The day after receiving your entry, we will do some quick tests of it to make sure there are no problems running it. Therefore, if you plan to leave town before December 17 you should make arrangements with us (by sending mail to pxs@gradient.cis.upenn.edu) to submit your entry earlier, including allowing a day for us to test your agent while we can still contact you in case of problems. Parties that submit their agent and then leave town run the risk that they will not participate in the competition.

The evaluation criteria for your performance will be taken over multiple simulations, and the criteria are provided in great detail in the Dec 1 entry below. The competition will place your agent in simulations in which varying subsets of the following are also trading:

No further details on the nature of the simulations will be provided. Note that the information above is considerably more constraining that one would have in the real markets.

Posted December 3, 2004: SOME IMPORTANT ITEMS.

Some changes have been made to the way the technical agents behave and the simulations are conducted. Therefore, you should all re-copy the technical and background agents from the Agents directory to your local directory so you have the latest versions. The changes that have been made are summarized below:

Posted December 1, 2004 (2 of 2): RELEASE OF ASYMMETRIC AND REAL-DATA BACKGROUND AGENTS.

We have now released the previously mentioned asymmetric and real-data background agents; the documentation can be found on the updated documentation for PLAT technical strategies. Both of these background agents will be used in various ways in the competition, so you should test your strategies in configurations including these background agents and the various technical strategies.

Posted December 1, 2004 (1 of 2): EVALUATION CRITERIA FOR THE COMPETITION.

The evaluation criteria for the coming competition will be essentially the same as for the April 2004 competition, which you can read more about in the News and Notes for 2003-04. Note that although the criteria will be the same, the form of our competition is quite different, since rather than simply running your strategies against each other on MSFT data, your strategy will be run against mixtures of the technical and background agents. But for the sake of argument, let's suppose that your strategy is evaluated by its performance over 10 different simulation "days" (the actual number might be different). Then the evaluation criteria will be as follows.

(a) In terms of raw performance, there will be a single criterion for the competition, which is the Sharpe ratio of your client's 10-day profit and loss. More precisely, suppose that on the 10 days, the final profits or losses of your client (see below for how we will compute these) are p1, p2, p3,..., p10, where each value might be positive (profit) or negative (loss). Then your client will be judged by the value

(average of p1, p2,...,p10)/(standard deviation of p1,p2,...,p10)

which is (one form of) the Sharpe ratio.

(b) There are no limits on how many shares your client can buy or sell in a day, but your client MUST LIQUIDATE ITS POSITION by the 4 PM close of the market. More precisely, there will be a monetary penalty applied to your profit or loss each day according to how many shares you fail to liquidate by the close. There is a necessary asymmetry between buying and selling here:

(c) PXS will be run in transaction cost/rebate mode for the competition. More precisely, recall that every time PXS executes a trade, one side of the order must have already been sitting in one of the order books, and the other side of the order must have been the "incoming" order. For each share executed by PXS, the party whose order was already in the books shall receive a REBATE of $0.002, and the party that was the incoming order shall pay a transaction FEE of $0.003. This is exactly the policy used by Island.

Items (b) and (c) above will be applied to your clients raw profit and loss figures in order to compute the values p1,...,p10 in item (a) above.

Posted November 30, 2004: MORE INFO ON THE UPCOMING COMPETITION.

In this entry, I make a number of comments regarding the upcoming competition, so please read carefully.

What you can do now: As mentioned below, in the competition your client strategy will be run in a variety of simulations in which the other strategies include at least the technical strategies you have been studying, and variants of the background agent. The command-line arguments of the technical strategies may be varied (for instance, changing their trading frequency or volume), resulting in markets whose liquidity is dominated by one subset of the strategies or another.

So a good way of getting started is to design, implement and test a strategy that can fare well under these conditions. If you have not already been working on a strategy, you will need to get the documentation on building and running a new strategy.

What you can do shortly: In addition to the above, the competition will also feature some variants on the background agent. One of these variants will let the distributions of arriving buy and sell orders be asymmetric --- for instance, it will be possible to run this agent in a way that makes the mean of the arriving buy limit order prices be 1 cent below the bid, and makes the mean of the arriving sell limit order prices be 3 cents above the ask. (In the original background agent that you have experimented with, these distributions were always symmetric around the bid/ask.) Such a setting models a case in which the buyers are more anxious to transact than the sellers. You might want to think about what might have happened in the first assignment under such conditions.

The other variant on the background agent will have the distribution of arriving buy and sell orders (or more precisely, the distribution of differences between the current bid and ask) be determined by files derived from actual market data for several NASDAQ stocks.

Both of these background agent variants will be released to you shortly, and both will be used in the competition. You should start with designing a strategy to work under (variations of) the first assignment conditions, but be ready to revamp for these background variants.

Meetings with Prof Kearns: I realize that many of you have requested meetings with me recently, and I apologize that my schedule has been too busy. But I plan to make myself available for discussions sometime this Friday, December 3. If you are interested in meeting then please send me mail, along with any constraints you have that day.

Comment for Senior Design participants: For those of you using the project as your Senior Design, I'd like to emphasize that this project is not a formal class, and Senior Designs are a research project, not coursework. Thus you should view the exercises and competitions I hold as a lower bound on what you should be doing, not an upper bound. In general, you should be using the project and simulator as a testbed for designing and testing an interesting trading strategy, and should view the competitions as periodic benchmarks on your progress. I expect that successful Senior Designs will show a significant amount of creativity, effort and testing beyond what I request in the exercises and competitions.

Posted November 29, 2004: FIRST ASSIGNMENT REPORTS; UPCOMING COMPETITION.

Welcome back to all from the Thanksgiving break. Thanks to everyone who turned in reports on the first assignment below; I've just started to look at these and will provide brief individual feedback shortly, but overall I am pleased with the level of analysis they show. In case anyone didn't know already, Market Making is the dominant strategy.

There will indeed be a PLAT competition to end the term. I hope to design and post details regarding this competition in the next day or two, so please check back for updates. The basic format will be to design and implement a trading strategy that can perform well when the other participants consist of some unknown mixture of the technical strategies we have been studying, variations on the so-called background agent (including ones that have asymmetric buy and sell distributions, and that use real market data), and possibly the strategies of the others on the project.

My rough plan for logistics is to have you submit the executable and source files for your strategies around Dec 14 (end of the Penn reading period), and we will then run the competition in following days to provide you with some entertaining relief from final exams.

Posted November 15, 2004: FIRST ASSIGNMENT DUE TODAY.

Reminder: the first assignment discussed below is due today. Please send electronic mail submissions with your report as attachment to Prof. Kearns.

Posted November 8, 2004: MORE ON FIRST ASSIGNMENT; DUE DATE EXTENSION.

Here is the latest update to the documentation for PLAT technical strategies. This time there are minor corrections to the command-line arguments for the market-maker strategy, as well as to the sample plotting command.

There have been some questions on whether you should be analyzing the behavior of the various strategies with respect to the simulator or the Island value and last price quantities. The answer is that you should be focusing exclusively on the value and prices according to the simulator (PXS), not the Island quantities.

Due to the various corrections, updates and confusions I am extending the deadline for the submission of a brief report on the first project until Monday, November 15.

Posted November 3, 2004: UPDATES ON FIRST ASSIGNMENT.

There is yet another update to the documentation for PLAT technical strategies. The only change is to the suggested command-line arguments for the technical strategies; we recommend you use these updates settings in the current assigment for better results.

Also, when I suggested below that you run "multiple PXS simualations" for each strategy pair, I meant multiple simulations in which the background agent file distribution.normal is changing. You can find multiple different background files in the directory /home/pxs/PAT/Agents/normal_distribution/ on crux.

Posted October 26, 2004: FIRST ASSIGNMENT.

In this entry, I am going to describe a formal assignment that I would like all teams to undertake and submit a brief report on by Friday, November 12. There may be further additions or alterations to what I describe below, so please be sure to check here regularly for updates.

The basic point of the assignment is to carefully investigate the questions first posed to you in the entry of September 26 below. Thus, you will be investigating the behavior of technical trading strategies we have provided to you.

The assignment is as follows:

The amount of time required just to set up and run the simulations is non-trivial. Please do not procrastinate, as the load on crux is likely to be high towards the deadline. Also, please remember to send all technical questions regarding running simulations to pxs@gradient.cis.upenn.edu.

Posted September 30, 2004: PROJECT MEETING TODAY.

Reminder: Project meeting today at 6 PM in Levine 315. All teams should have at least one representative present.

There is once again updated documentation for PLAT technical strategies. In addition to yet further additions to the "Notes on running these agents", there is an important correction to the input filename expected by the background agent. In general, at least in the near term, there may be frequent updates of this document while things get stabilized, so be sure to check it regularly.

Posted September 29, 2004: UPDATED DOCUMENTATION; MEETING LOCATION.

Here is updated documentation for PLAT technical strategies. The main changes are important additions to the section "Notes on running these agents". Please read these carefully.

The next project meeting will be Thursday, Sep 30 at 6 PM in 315 Levine Hall. Everyone should make every effort to be present, or to at least have one representative of their team present.

Posted September 27, 2004: CORRECTION TO THIS WEEK'S MEETING DATE

I had posted a fictitious date for this week's meeting, which I have now corrected below; the correct date and time are Thursday, Sep 30 at 6 PM, location TBD.

Posted September 26, 2004: RELEASE OF TECHNICAL TRADING AGENTS; MORE ON SENIOR DESIGNS

IMPORTANT: See Sep 24 entry below for info about an upcoming project meeting.

In our first meeting and elsewhere, I have suggested that much of the project will be focused on developing agents that can profit in an environment composed of a number of common technical trading strategies. Towards this goal, my doctoral student Elliot Feng has implemented a number of such strategies, and has made available documentation and code for these PLAT technical strategies.

All project participants should begin understanding and experimenting with these trading strategies. Further background on some of these strategies can be found in the links elsewhere on the project pages, and on the web at large.

In particular, participants should begin considering the following questions:

We will shortly be making these questions more formal and precise, in particular at Thursday's meeting. But the overall thrust of the project this year will be to examine questions such as the above for varying collections of common (and perhaps uncommon) strategies.

SENIOR DESIGNS: You may incorporate the new material above into your first reports. Prof. Taylor will accept electronic signatures, so if you need my sign-off please send me your report via email and I will send email to Prof. Taylor giving my approval if your report is acceptable. Please do not stop by my office Monday for signatures; I have a full schedule. For time-line purposes, you may assume that this year's schedule of progress and events on PLAT will be similar to those of past years.

Posted September 24, 2004: SENIOR DESIGNS AND NEXT MEETING

Anyone who is planning to participate in the project this year and has not completed the steps outlined in the Sep 17 entry below should do so as soon as possible, including the reading. Please do not request an account before you have completed the reading.

A number of you have recently requested one-on-one meetings to discuss the project and your ideas. While I will generally try to meet with teams periodically, please understand that in the immediate term my schedule is too constrained to accomodate all such requests.

Senior Designs, I am aware that you have an initial report due on Monday. As I suggested at your first meeting, since you do not yet have a detailed description from me as to the exact nature of the project and competitions this year, you can't elaborate on these aspects in your first report. However, the suggested readings in the Sep 17 entry should provide ample background for you to discuss the project and its larger context in the world of financial markets and trading, and I suggest you emphasize these in your first reports.

Finally, I hope to shortly post more detail about the format of the project and competitions this year, and to hold a second meeting next week in order to discuss it. The tentative date for this meeting is THURSDAY SEPTEMBER 30 AT 6 PM, LOCATION TBD.

Posted September 17, 2004: GETTING STARTED

All PLAT participants should start by reading the following article, which provides an overall introduction to the project:

AFTER YOU HAVE READ ALL THE DOCUMENTS ABOVE, you should send mail to pxs@gradient.cis.upenn.edu and request an account on crux.cis.upenn.edu, which is the server on which the project is run. You will also be assigned a range of ports for your client.

Please remember that any questions regarding systems or programming issues should always be sent to pxs@gradient.cis.upenn.edu, not to Prof. Kearns.

For those seeking further background and ideas, here are some other papers on the PLAT project. This page has some useful links regarding financial markets, trading strategies, and related topics. I have also cobbled together some papers on market microstructure along with brief synopses. Finally, for the truly ambitious, here is a pointer to some papers on financial modeling that are being covered in my current graduate seminar, as well as a theoretical paper on volume-weighted average price trading by myself and colleagues that was partially inspired by the PLAT project.

Finally, a great deal of information, as well as the spirit of the project, can be gleaned by browsing the News and Notes pages of the first two years (see links at the top of this page).

Posted September 6 (Updated September 9), 2004: INTRO AND ORGANIZATIONAL MEETING

There will be an introductory and organizational meeting for the Penn-Lehman Automated Trading Project on September 17 at 10 AM in 315 Levine Hall. Those wishing to participate are strongly encouraged to attend. In addition, please read the note of August 12 below. If you plan to participate but cannot attend the meeting (I realize this is during Rosh Hashanah), please send me mail and we'll figure out an alternate way to get you up to speed.

Posted August 12, 2004: PLAT 2004-05

The Penn-Lehman Automated Trading Project (PLAT) will once again be active and open to all participants, both from Penn and external, during the 2004-05 academic year.

This year we are planning some major and interesting changes to the form of the project and competitions --- in particular, to examine strategic and game-theoretic aspects of automated trading. We continue to encourage the particpation of undergraduate and graduate students, faculty members, and professional scientists and researchers interested in automated trading, electronic commerce, machine learning and statistical modeling, economics and related fields.

I anticipate holding an organizational meeting early in the Fall term for potential participants. If you are interested in being part of the project this year, send me an email with your background, experience and interests.