Skedler is built to scale-up and handle a large number of reports. The following Skedler features help with the scalability
- CPU core based scaling - You can allocate specific number of worker threads for report generation. Each worker thread needs a CPU core for report generation and will generate 1 report at a time. If Skedler is run on a 4 CPU core machine, assuming 2 cores are used for standard operations in machine, you can allocate 4-2=2 CPU cores for the worker threads. Therefore 2 reports can be generated simultaneously at any time in that machine.
- Queuing for report generation - Skedler uses a queuing system for report generation. This ensures that reports are generated in a sequential manner even if the CPU resources are limited. The queuing system also persists the scheduling requests and helps Skedler recover from any failures without loss of report generation.
- Large report time-out - You can configure a maximum time-out for report generation. This is helpful when some dashboards have poor performance due to Elasticsearch or Kibana issues. The default time-out is 180 seconds. So if a dashboard takes longer than 180 seconds, Skedler will send a notification to the admin that it is unable to generate the report for that dashboard and move on to the next item in the queue.
The above features enable Skedler to provide best-in-class performance for report generation. Nevertheless, report generation performance is dependent on a number of factors such as Elasticsearch cluster performance, Kibana dashboard performance, number of charts/data elements in a Kibana dashboard that are beyond Skedler's control. Therefore, users should consider the overall performance of a dashboard since it has a direct impact on Skedler performance. Poor performing dashboard implies longer report generation time.
If you need to generate hundreds of reports per day, you can follow the below best practices:
- Verify that the dashboards are performing well (less than 10 seconds to load all the charts/data)
- Allocate adequate number of CPU cores for Skedler. Consider running Skedler in a separate VM or docker container with 4 or 8 CPU cores if your requirement is to generate a number of reports simultaneously.
- Stagger the scheduling of reports over a period of time so that all the reports do not land in the Skedler queue at the same time. For e.g., schedule 4 reports for 10 am, 4 reports for 10.05 am and so on.
- When it becomes too expensive to scale-up the server, then consider scaling-out of Skedler. You can scale-out by running multiple instances of Skedler, each in a separate VM/container, which are used to generate a specific number of reports. For e.g., if you need 20 reports to be generated exactly at 10 am and if you have only 4 CPU core machines, then you might need 5 Skedler instances to meet your report generation requirement. (Please note that each instance of Skedler instance would require a separate license. )