Information about the current status of running tasks and their rejections can be seen in the Work Items view under Administration > System Status > Work Items.
Mango runs every task at one of three levels, High, Medium and Low. Some tasks have priorities that can be changed, others do not. Mango provides a way to monitor the various rejections of tasks in the System Status --> Work Items view, this document describes what this information means and how to tune Mango for the best performance.
Task Queues for the High and Medium pools are used to ensure that a given task runs in the order it was scheduled. The ordering is based on a unique identifier and the queue can be limited or infinite. Limited queues can have a hard limit defined in the code or rely on the env property runtime.realTimeTimer.defaultTaskQueueSize. It should be noted that only the High and Medium pools can run Ordered tasks, however they can also run tasks that do not require an order. If an un-ordered task is submitted to the pool it will run at some point in the future but not necessarily before other tasks of the same type.
Understanding Runtime Performance
Running Stats
This area provides insight into any tasks that are currently executing. A task must be currently executing to be seen in this view, most tasks execute so quickly that this view will have little information in it in a healthy system. When tasks start backing up and taking longer to run this page can provide insight into what tasks are long running and potentially degrading performance.
Rejected Stats
Information about task rejections can be found in this section. A task is rejected for one of 2 reasons. First the queue may be full for an ordered task. This means that the system cannot process the tasks fast enough and they have filled that task's specific queue and thus are being rejected. This usually means that your timeouts are set improperly or the task is scheduled to execute too often. Pool full rejections occur when the thread pool for that priority is full and no more task can be scheduled to run until a thread becomes free.
High Priority Tasks
High priority tasks run in the high priority pool at the highest thread priority of the Operating System and are ordered based on task. For example tasks of different types may run at the same time while tasks such as the polling of a specific data source will run in its own queue such that the next poll can only happen after the current poll finishes. Each high priority task has a maximum queue size (outlined below) that when full and a new task is attempted to be run it will be rejected with a Queue Full task rejection.
The High priority thread pool must be large enough to run all High priority tasks that are scheduled at any given time. When the pool is full it will reject tasks with a Pool Full task rejection. The system settings page allows adjusting the core pool size, which is the number of threads that will always exist and be ready to process new tasks, and the maximum pool size which allows limiting the pool so the system does not run out of resources when the pool is full. When full, new tasks will be rejected until a thread becomes free to process new tasks.
| Task | Queue Size | Ordered On | Description |
|---|---|---|---|
| Point Value Batch Write | 0 | Only 1 | Writes point values to database asychronously |
| Process Event Handler | default | Instance | execute system process of event handler |
| Set Point Value | Unlimited | Point id | set a point value asynchronously |
| Http Image Data Source | 10 | Data point XID | Retrieves images asynchronously |
| BACnet iAm Request | N/A | No ordering | Sends iAm messages to devices |
| BACnet Object List Request | N/A | No ordering | Requests object lists from devices |
| BACnet Point Creator | N/A | No ordering | Create data points from discovery |
| BACnet COV Maintenance | N/A | No ordering | Manage COV subscriptions |
| Brewer's Dashboard Delay Set | 5 | Point id | Set point values at some time in the future |
| Input stream epoll | N/A | No ordering | Read input streams via epoll |
| Process epoll | N/A | No ordering | Monitor and provide feedback on executed system processes |
| Persistent Point Hierarchy Sender | N/A | No ordering | Synchronize point hierarchy |
| Persistent History Sync | N/A | No ordering | Synchronized data point histories |
| Persistent Range Counter | N/A | No ordering | Count a range of point values for a history sync |
| Persistent Sync Timer | N/A | No ordering | Schedule the synchronization tasks |
| Persistent Timeout Output Stream | N/A | No ordering | Check publisher's writing stream for disconnections |
| Interval Logging | default | Data Point XID | Interval log point values |
| Polling Data Sources | default | Data Source XID | Poll of each data source |
| User Event Cache Cleaner | default | Only 1 | Purge memory of user events when user has not logged in for some time |
| Timeout Event Detectors | default | Event detector XID | Any event detector that uses a timeout |
| Email Event Handler Escalation Task | default | Handler XID | Send emails triggered by events |
| Nightly Purge | 0 | Only 1 | Purge old data nightly |
| Upgrade Check | 0 | Only 1 | Check for upgrades every 24hrs |
| Work Item Monitor | 0 | Only 1 | Count executing work items every 10s |
| Schedule Configuration Backup | 0 | Only 1 | Schedule a configuration backup task |
| Schedule Database Backup | N/A | No ordering | Schedule a database backup task |
| Publisher send snapshot task | default | Publisher XID | Send a snapshot of the publisher's information |
| Publisher send realtime data task | default | Publisher XID | Send realtime data from publisher queue |
| License check task | N/A | No ordering | Check for a valid license |
| Schedule Excel Report | N/A | No ordering | Schedule a report to run now or later |
| JSON File Importer | N/A | No ordering | Import a JSON Configuration via the JSON Import module |
| Maintenance Events | 0 | Maintenance Event XID | Make event active or inactive |
| Schedule NoSQL Backup | 0 | Only 1 | Schedule a NoSQL Backup to run now or later |
| Meta point delay | N/A | No ordering | Delay the execution of a meta data point |
| Meta point cron | N/A | No ordering | Execute meta point on CRON pattern |
| PakBus Network Poller | N/A | No ordering | Periodically poll PakBus network for changes |
| Schedule Reports | 5 | Report XID | Schedule reports to run now or later |
| Scheduled Event | 0 | Scheduled Event XID | Make event active/inactive |
| Web Socket Ping Pong Tracker | default | session ID | Manage websocket connectivity |
| NoSQL Data Map Cleaner | 0 | Only 1 | Manage memory for writing point values |
| NoSQL Data Mover | 0 | Only 1 | Collate point values for high performance writing |
| NoSQL Status Provider | 0 | Only 1 | Provide status on write performance |
| PakBus Requests | N/A | No ordering | Send requests on PakBus network |
| Persistent Data Source Status | N/A | No ordering | Provide status on connections |
| Scripting Data Source Execution Delay | N/A | No ordering | Delay execution of scripting data source |
| Serial Data Source timeout manager | N/A | No ordering | Manage serial response timeouts |
| Advanced Schedule Events | 1 | Schedule XID | Manage event state |
| Data Source Initializer | N/A | No ordering | Initialize data sources when Mango starts |
| Data Source Terminator | N/A | No ordering | Terminate data sources when Mango stops |
| Point Hierarchy Event Dispatcher | N/A | No ordering | Dispatch events about the point hierarchy |
| Upgrade Downloader | 0 | Only 1 | Download and install desired upgrades |
Medium Priority Tasks
Medium priority tasks run in the medium priority pool just below the highest thread priority of the Operating System and are ordered based on task. For example tasks of different types may run at the same time while tasks such as a system backup will run in its own queue such that the next backup can only happen after the current backup finishes. Each medium priority task has a maximum queue size (outlined below) that when full and a new task is attempted to be run it will be rejected with a Queue Full task rejection.
The Medium priority thread pool does not have the same restriction as the High priority pool in terms of available threads to perform the work. This pool is capable of queueing tasks if there are no free threads. This means that in an overburdened system the Medium priority pool can grow unbounded. When the pool is full it will queue tasks and run them when a thread becomes available. The system settings page allows adjusting the core pool size, which is the number of threads that will always exist and be ready to process new tasks.
| Task | Queue Size | Ordered On | Description |
|---|---|---|---|
| REST Temporary Resource Timeout | default | Resource ID | Clean up expired resources |
| Data Point Event | Unlimited | Data Point XID and Listener ID | Send messages about state and value changes on a point |
| Configuration Backup | 0 | Only 1 | Backup JSON configuration |
| Database Backup | default | Only 1 | Backup database |
| System setting changed notification | 100 | System Setting | Update listeners as to what system setting was changed |
| Data File Import Status | default | resource ID | REST temporary resource for data file imports |
| Haystack Import Status | default | resource ID | REST temporary resource for history imports of haystack data |
| Snmp Mib Walk Task | 0 | Host and Port of Device | Task to explore a device using an SNMP Mib file |
| Audit Event | 0 | Unordered | Record changes to Mango configuration |
Low Priority Tasks
Low priority tasks run in the low priority pool at the normal thread priority of the Operating System and are un-ordered. This means that any low priority task can run independent of any other low priority task. The tasks are supplied to the pool and will be processed in no guaranteed order. This pool behaves like the Medium priority pool such that tasks that have no threads to process them are queued until a thread is free.
The Low priority thread pool does not have the same restriction as the High priority pool in terms of available threads to perform the work. This pool is capable of queueing tasks if there are no free threads. This means that in an overburdened system the Low priority pool can grow unbounded. When the pool is full it will queue tasks and run them when a thread becomes available. The system settings page allows adjusting the core pool size, which is the number of threads that will always exist and be ready to process new tasks.
| Task | Description |
|---|---|
| Alarm notifications | Notify about alarm state changes |
| Send Email | Send email notifications |
| NoSQL Backup | Backup NoSQL database(s) |
| NoSQL Restore | Restore NoSQL database(s) |
Configurable Priority
These tasks have a configurable priority via System Settings. If run in the Medium or High pools they are ordered, if run in the Low pool the tasks have no order.
| Task | Queue Size | Ordered On | Description |
|---|---|---|---|
| Excel Report | 10 | Report XID | Run an Excel Report |
| Report | 5 | Report XID | Run a report from the Reports module |