Pseudo Markets Blog

As of 12/25/2021 12:45 PM CST, all services have been restored and are online. Thank you for your patience.

Over the past week, there have been several blips in outbound connectivity from our primary hosting facility in North Texas. Primary router has been taken out of load to investigate, and as such, all services will be down until further notice.

When the Pseudo Markets project was first conceived in mid 2019, I had very little experience working with distributed systems. All of the previous backends I had worked with were huge monoliths that connected to a single relational data storage system. Since then, I have had the opportunity to work with systems that handle daily transactional volumes I wouldn’t even be able to fathom just a few years ago. One of the design decisions I made when coming up with the primary backend system for Pseudo Markets was driven by my lack of knowledge of making certain services independent, decoupled, and small. Despite not running into any immediate issues with the current design, I took it upon myself to apply what I have learned in the past year or so to re-architect the entire solution, starting with decoupling the large and tightly packed core functionality of the Unified API.

Several large pieces, such as the core equities trading platform, market data service, and portfolio performance reporting service have already been decoupled into their own services. This allows horizontal scaling as each service is distributed, whereas the previous architecture would only benefit from vertical scaling. This also allows the Unified API to just focus on one job, bringing together or “unifying” data from different services. Decoupling login and authentication from the Unified API brings the entire platform one step closer to becoming fully distributed. While most of the work is porting out and cleaning up the existing authentication code, I also have an excellent opportunity to start utilizing the Real Time Data Store (Aerospike) as a fully functional system of record. Currently, the RTDS is used as a high speed caching layer for the Market Data Service, but I have plans to move all data from the Relational Data Store (SQL Server) into RTDS.

The new Unified Auth Service will feature the capability to sync account, user, and token data across RTDS and RDS, provide data from either source, and function as both a standalone API as well as a DLL. There will be 3 modes of operation:

• RDS Mode: All logins will be processed using data from RDS
• RTDS Mode: All logins will be processed using data from RTDS
• Hybrid Mode: Logins will go through round robin to use RTDS or RDS

You can track progress here: https://github.com/pseudomarkets/PseudoMarkets.Unified.Auth. I’m hoping to get this working in our dev environment first, then promoting to our new DR server in Austin, before finally pushing it out to our two primary prod app servers in DFW.

Starting next Tuesday, the Unified API on app.pseudomarkets.live will begin processing all orders through the new, OS service based Consolidated Trading Platform. This allows all trading logic to be fully decoupled from the Unified API, and can support higher volumes and availability thanks to its distributed nature. Currently, all trades will be routed through the a single queue into the “Pool A” service. In the near future, there the load will be split up with another queue into “Pool B”, wherein orders for even account numbers will go to “A” and odds into “B”. We look forward to the improvements the new trading platform will bring to the overall Pseudo Markets, and we can’t wait to share what we have in store next.

All services have been restored and are operating normally as of 5/27/2021 5:15 PM CST. We appreciate your patience.

Due to a power outage at our hosting location in DFW, our primary VM host had lost power and has been unable to start back up. Scheduling constraints prevent me from visiting the hosting location to restart the host server until the end of the month. We apologize for any inconvenience. Please check our status page for more updates.

All services are back online as of 2/19/2021 7:40 PM CST. We appreciate your patience.

Due to inclement weather conditions at our hosting location, all services are unavailable. In an effort to keep our hardware safe from excessive power cycling due to unstable electricity availability, we will remain to keep servers powered down until conditions improve. You can monitor status at https://status.pseudomarkets.live. We appreciate your patience.

I am very happy to announce that the Market Data Service has been deployed to production, and that the quotation endpoint in the Unified API, as of version 1.3.0, will consume market data from the new service. Aside from the vastly improved performance and cleanliness of the implementation, there are 3 key changes in how price data is formed.

Latest Price

The Latest Price endpoint will now use the midpoint of IEX TOPS bid and ask pricing during intraday requests, and will fall back to Twelve Data Real Time Price data during non-market hours. Since IEX TOPS does not have a rate limit, this is the best choice for the most frequent look-ups during regular market hours. This is also the price reference that will be used when making trades, replacing the previous Smart Quote system.

Smart Quote

The Smart Quote endpoint will now be composed of an aggregated average price across Twelve Data Real Time Price, IEX TOPS bid/ask midpoint, and Alpha Vantage Global Quote.

Cached Prices

In an effort to stay within data provider rate limits, we will now be enforcing a 2 minute cache on hits to Latest Price, Smart Quote, and Indices. Caching is done on a per symbol basis, and the data source field will reflect whether the pricing data was served from the cache layer or not. As data provider limits change, the cache TTL will be adjusted to accommodate our plan limits.

I’ve been busy performing infrastructure upgrades to allow the Pseudo Markets Unified API and supporting services to continue growing in functionality and scale, and am happy to announce that the latest round of hardware upgrades is now complete. During the course of this change, there is now a single app VM running the ASP.NET Core Web API backend system for trading, quotes, account management, and more along with 3 VMs running various databases. Two of the database VMs are running Aerospike Community Edition 5 in a networked cluster for full replication, and the third is running Microsoft SQL Server Express 2019 and Mongo DB 4.4. You’ll see increasing utilization of both Aerospike and Mongo DB as I continue to roll out more services. Continue to check GitHub for the latest updates. Along with moving the databases to run on separate VMs, all of them run on a Proxmox based physical host with a total of 24 physical cores, 128 GB of RAM, and approximately 2 TB of flash storage. All Pseudo Markets applications and data are stored in high speed flash storage, with plans to move backup and archival data to networked attached storage backed by spinning drives. Plans for creating a secondary backup Unified API server in a cloud environment are still under consideration, but I strongly believe with the latest network and compute infrastructure upgrades that a single instance will be sufficient to provide reliable and consistent service.