We are marking this incident as resolved although we will continue to validate possible solutions in our architecture to decrease the frequency of delays for WebHooks and GlobiFlow automation. We have updated settings and added additional tracking internally to help troubleshoot the delays. We are testing solutions and improvements in both the Podio and GlobiFlow architecture to help ensure better reliability of these flows in the future. Thanks for your patience!
Posted Jan 29, 2019 - 13:03 EST
Update
The delays have been resolved and the queues are back to normal. We have implemented new additional metrics and tracking to better understand the spikes in the delays for web-hooks. We are still validating hypothesis to further understand longterm solution to avoid these spikes on a regular basis.
Posted Jan 28, 2019 - 12:33 EST
Update
We are currently seeing slight delays for GlobiFlow and webhooks, team is using the spike to investigate root cause behind the intermittent delays and monitoring closely.
Posted Jan 28, 2019 - 12:16 EST
Update
We have seen good results today from the tweaking we have performed yesterday in the infrastructure to pinpoint root cause, however we still have additional testing to perform to fully understand cause and solution. We will keep this incident open until root cause has been fully confirmed.
Podio is currently fully operational however as the API/webhooks execution has been more unstable this week, we are keeping the incident open until a full action and solution plan is in place.
Posted Jan 24, 2019 - 14:59 EST
Update
GlobiFlow DB was seeing high load in the past hour but it has been resolved now. We continue to monitor the case and investigate on the root cause for the delays.
Posted Jan 24, 2019 - 04:57 EST
Update
We believe to have found the possible cause for the higher error levels and delays this week in our infrastructure. We are currently testing possible solutions to confirm if we found the right cause and therefore have a good future solution in place. There is currently no delays on webhooks and GlobiFlow but we will continue to keep this incident open until we confirm cause and resolution.
Posted Jan 23, 2019 - 11:08 EST
Investigating
We are investigating the issues causing degraded performance this week and delays for webhooks and GlobiFlow. Stay tuned.