In the beginning of 2019, we noticed significant increase in load during early hours of European time. This resulted in intermittent degraded performance with increased latency, webhooks delays and elevated error rates. The immediate customer affect was the stalling of webhooks and delays for our customers using workflow automation in some occasions during the week.
In February, we implemented an improvement to our webhooks queue system, which was done to reduce the stalling of webhooks delays and Globiflow Advanced Workflows. After implementation of the new queue system, we have seen a general better performance of webhooks and workflows.
In March we identified the main cause behind the load issues due to specific updates in very large apps, serving more than 500k items in the same app. These apps are heavy for the system, as a single update in an app field, can result in millions of updates across both calculations, search indexing, and workflow automation. We also noticed an issue where calling the API to get data views and filtered views from these large apps, could sometimes cause a bottleneck, if the app was being updated in the same time by users manually or workflow automation. We have worked with our API consumers and customers serving the large apps to better spread usage of the heavy calls to periods where the apps are less active and not being updated at the same time.
As a result of these learnings, we will work on a larger project to refactor some of these heavy API calls to better protect calls where apps are being actively used while served from the API. In the meantime, we ask our customers and partners to review API usage not to perform concurrent updates of data via the API, while the data is being updated in the UI or from automated workflows at the same time.
We will also consider to implement a maximum limit of 500k app items per app in Podio in the future. Podio is a project management and work management tool but was never built to serve a database with 500k+ items in the same app. For this purpose, we recommend customers to use a real database functionality, and only service active items to Podio that requires collaboration and team coordination.
We will share further communication before introducing a hard limit of items per apps, however in the meantime, please review our known limitations here:
In the same period, the growth of very large apps on the platform, also caused an issue for the general Search performance, which resulted in 2 weeks of elevated search issues. We resolved this issue with a larger maintenance of the Search infrastructure on April 1-2.
Finally, in order to cope with the general increase in load, we had a database maintenance running in production that culminated on April 13th. This unavoidable activity resulted in few issues related to item availability & increased response time for customers. We now expect a general stability increase in the coming months.
We are also taking longterm actions to secure a scalable path to ensure we can better cope with unexpected load issues in the future.
Again, we are deeply sorry for the issues this caused to your business, and thank you for your continued support.