Clearing the Four Most Common Trading Analytics Hurdles
By Peter Simpson, Datawatch
Originally published on the TabbFORUM
Data analytics and visualization solutions have been transforming the capital markets for years, but a number of challenges still hamper buy-side and sell-side firms as well as exchanges from optimizing trading effectiveness. By tying real-time streaming data analysis and visualization solutions into the trading process, firms can achieve faster time-to-insight and action.
Data analytics and visualization solutions have been transforming the world of capital markets for years, but a number of challenges still hamper the trading effectiveness of exchanges and buy-side and sell-side firms. Following are four of the most common trading analytics obstacles.
Sales teams and trading desks are increasingly using visualization tools. However, due to both regulatory change and customer demand, there is also a growing need for internal and external transparency in trading effectiveness; most of today’s visualization solutions do not fit the bill. They may produce nice-looking visualizations, but they do little to actually improve transparency and trading effectiveness, given that they are based on end-of-day data and the summaries are too high level. Exacerbating the problem, trading analytics GUI development is still largely taking place in broker and fund manager IT departments, resulting in increased delivery cost, risk and delivery timescales.
To understand, gain insight from and take informed action on data takes time. Although most flow is now high frequency and too fast for human interaction, the monitoring of this flow and the markets themselves is still, for the most part, done under human guidance. Most of this monitoring is actually very tabular, with traders reviewing blotters with thousands of orders. This manual approach introduces human error into the trading process and greatly delays decision making.
3. Life Cycle Management
Analytics problems expand beyond the monitoring phase and actually impact the whole trading life cycle, from pre-trade basket analytics and order execution, to post-trade transaction cost analysis (TCA) and best execution reporting. Throughout the trading flow, e-trading desks are being swamped with data, while needing to make fast informed decisions.
Additionally, with Markets in Financial Instruments Directive (MiFID) and Markets in Financial Investments Regulation (MiFIR) regulations, there is a distinct blurring of the lines between trading analytics and trade surveillance, with a much heavier focus on trader effectiveness and trader surveillance (rather than on trade surveillance). Compliance surveillance analytics are converging toward trading analytics, as trading desks increasingly become responsible for regulatory reporting, and compliance desks need access to the information that was historically only available to the trading desk.
4. Big Data
Firms are trying to manage large data sets that are continuously changing throughout the trading day, due to ever-evolving market conditions and executed baskets. However, in these trading analytics use cases, big data doesn’t generally mean Hadoop or standard SQL databases. It means complex event processing (CEP) and time-series or tick databases, with a few niche vendors having a large proportion of the usage in most of the largest brokers and fund managers. These niche systems are optimized for storing the historic time series of orders, executions and associated market data, and provide out-of-the-box analytics, which are complex or inefficient in standard relational databases.
As a result of this landscape, firms are having trouble connecting visual analytics tools to these niche data analytics systems. Additionally, time-series analysis is a challenge. Many tools see time as a category, rather than as a continuous series where you can overlay executions against market conditions and playback through past trading behavior. This means that normalizing prices before and after executions to investigate toxicity is also tricky.
It should come as no surprise that trading firms gain a competitive advantage by solving each of these problems. But the question is how to do it. The answer lies in real-time streaming data analysis and visualization solutions. When visual analytics is tied closely to the trading process, firms can achieve faster time-to-insight and action.
Visual analytics tools can be purchased as a standalone software package, or embedded into an execution management system (EMS) or order management system (OMS). Regardless of the method, effective solutions produce faster analytics, including:
- Access to trading data – Access to live streaming of both trading flow and market liquidity.
- Understanding of data – Quick identification of trends, clusters and anomalous behavior.
- Investigation of trading anomalies – Rapid investigation into how anomalies have occurred and how they impact overall performance.
- More informed decision-making – Direct link to trading systems, so that problems can be mitigated and opportunities realized.
Without real-time visual analytics, firms are stuck looking at a tabular blotter with thousands of orders, and reviewing graphs from yesterday’s trading – neither of which is an effective strategy to quickly identify anomalies, trends, clusters and relationships.
[For more on the technology architectures firms are deploying to gain the ability to visualize both real-time and historical data, please contact TABB Group for information on our latest recent report, “Big Data Is Dead, Long Live (Real-Time) Big Data: Real-Time Big Data Analytics in Financial Services.”]
Some EMS providers have integrated visual analytics into their trading GUIs, providing real-time visual blotters that are enriched with high-density visuals. This allows traders to quickly assimilate their position, market impact and market opportunities. And it enables fast analysis of market conditions and trading activity, resulting in smart timing and execution decisions. It is still a balance between visuals and tabular data, as people become more accustomed and confident in the visual worlds. But traders are reviewing more orders and identifying problems that are typically missed with traditional displays.
The expansion of visual analytics from blotter monitoring to both pre- and post-trade has historically been the domain of quants working with tick databases. With the increased need for trading effectiveness and transparency, largely driven by MIFID, there are a number of projects across firms that can provide appropriate trading analytics to all the sales and trading desks. Typically, these projects elevate the tick database as the central trading data mart, both for T+1 regulatory reporting as well as for intra-day and real-time analysis. It should be noted that scalability can become an issue, as visual analytics is leveraged not only by a small number of quants, but also the entire trading floor, where each individual is concurrently performing his/her own analysis.
Regulatory mandates to store all appropriate market data, positions, risks, orders and executions for regulatory purposes, combined with delivery best execution and trader surveillance reporting, demand a specific infrastructure cost. Some firms view this continuously updating data repository as the institutional memory of trading effectiveness that shouldn’t be used just for regulatory reporting. Instead, it should be integrated into a fast analytics solution and existing trading effectiveness models to reduce the time that it takes to make trading decisions, and ultimately provide firms with a competitive advantage.
As with all analysis, gaining insight from and taking informed action on data takes time – and time has a direct financial impact on trading firms. This is valid across the trading life cycle, from pre- to post-trade execution analytics. The combination of real-time, intra-day and historic analytical displays allows traders to quickly assimilate their position, market impact and market opportunities. Integration into tick databases enables quick investigation of execution history and historic market liquidity, whether for a specific order or for all trading flows. And past trading can be played through, honing in on specific periods of high interest, from long periods down to tick-by-tick.
Firms now have many choices regarding how they implement fast analytics solutions. They can ignore the problems unique to trading analytics and use tools built for end-of-day T+1 analysis, focus significant resources and risk on development, or play with big data technology because it will be good for the resume.
In reality, trading firms need to leverage stored market and execution data to make smarter trading decisions and identify best execution. By using analytical displays to optimize strategy configuration, they can extract intelligence to improve customer flow, execution strategy, venue performance and back-testing of trading models. And these endless benefits are only possible with fast analytics stacks that combine real-time data processing, historic time-series storage and querying, and analysis across visual and predictive analysis.
Peter Simpson is the vice president of Visualization Strategy at Datawatch and is responsible for the Panopticon platform.
Originally published on the TabbFORUM
This column does not necessarily reflect the views or opinions of FinReg Alert or Tradeweb Markets LLC.