William Dealtry has been working in both Python and C++ for many years, and has been a member of the C++ standardization committee for more than a decade. Currently he is the Architect of a new open-source Dataframe database, ArcticDB, which is backed by long-time Python enthusiasts Man Group and Bloomberg.
At Man Group we ingest data of all shapes and sizes, from market prices to weather, ESG reporting to news media. Storing that data in a format that is useful both to researchers and to strategies trading more than a billion dollars is a unique technical challenge - on that has resulted in two iterations of our high-performance data storage product ArcticDB. The aim is nothing less than to bring the performance and analytical capabilities of a server infrastructure right into the Python client. With the client now available on Conda and PyPI. and the source published on GitHub, I want to take you on a tour of our database; the design rationale, the ups and downs of developing a new database while supporting a billion-dollar trading estate, and the lessons we've learned along the way.