Administration & Troubleshooting
The host with can be run on either physical hardware or a virtual machine, with the following minimum specifications:
The following Windows features must also be installed:
The following applications must be installed prior to the Workbench:
3.2.xseries—MongoDB 2.x is no longer supported). See the MongoDB installation instructions for MongoDB download, installation, and configuration steps.
After installing, to access the Portal, use one of:
Note that additional configuration is required to allow Portal connections from other hosts.
If you are installing the Workbench on an Azure virtual machine, you should:
D:drive for storage; it is a temporary drive that will be erased each time the instance reboots. When you attach a new drive via Azure, you must also mount it via Disk Management in the OS and give it a new drive letter (e.g.
The minimum configuration above is sufficient for most test and moderate-volume instances but may not be sufficient for high data volumes, many distinct applications (or application versions), many additional custom indexers, multiple additional custom patterns, or large amounts of custom data. In such cases, please review the guidelines below, or contact support so we can review your needs and offer tailored guidance.
To begin estimating your resource needs, first review the System Requirements section of the Data Hub User Guide and consider how they affect the Workbench host.
For the normal "Hub + Workbench" host, the Hub just provides a temporary queue during Workbench upgrades or configuration changes - so data storage isn't usually much of an issue. It also provides an error queue for any messages that the Workbench can't parse, but that error queue is time-limited so it generally will not get large. Message rates into a single Workbench are also generally much lower than the Hub upper limits, so CPU and memory requirements are generally very modest.
In most normal scenarios, the Hub just needs ~10GB of free disk space, and ~1GB of free memory.
There are two primary consumers of CPU in the Workbench: the Computation Service and the Query Web Service (plus MongoDB). Please refer to the architecture diagram if needed. We assume that the Portal will be configured for remote access; if not, the browser will also consume CPU when viewing the Portal locally.
With the minimum requirements described above, the Computation Service can consume approximately 10 envelopes per second. This rate is not typically CPU-bound, so additional CPU probably won't improve the rate significantly.
Query load depends on the amount of data in the database, and on the number of concurrent Workbench users. The minimum requirements above can provide acceptable performance (below-5-second report load-time) for up to about 1 million app runs, for a single Portal user. This rate is usually CPU-bound, so additional CPU will provide additional scalability. Individual queries are mostly single-threaded, but some reports request multiple queries concurrently, so higher clock speeds and more cores will help. More cores will also help improve responsiveness with concurrent users.
Memory use is generally near-static, even under load. There are 4 primary consumers of memory:
The Computation Service will typically only use about 200MB of memory, with the default indexers. Very large messages can increase this amount.
The Query Web Service will typically use about 200MB per concurrent user, although it can use more (up to about 600MB, observed) with e.g. direct API access and a very large query.
RabbitMQ will limit itself to 25% of available system memory, but will generally use much less than that (a few hundred MB).
MongoDB will typically only consume a few hundred MB as "private bytes" but will often consume all available memory (over time) as "shared memory". For scalability purposes, the "private bytes" value is the one that must be supported. More available memory is always better for MongoDB, but 512MB is sufficient.
In total, ~1GB of available memory is sufficient for most installations, plus ~200MB per concurrent user of the Portal.
First, we recommend putting the MongoDB data and log folders on a secondary, dedicated disk. This limits disk contention, eliminates any risk of the OS disk filling up, and makes it easier to expand the available disk space if needed. Additionally,
as the MongoDB Production Notes recommend setting the Windows page file to the largest feasible multiple of
keeping the database on a separate disk permits a larger page file on the OS disk.
Second, remember that the Workbench has two types of storage: temporary and permanent (both in the same MongoDB instance). These two storage areas have growth rates that are independent of each other. Note that for the first few years of Workbench use, temporary-storage requirements typically exceed permanent storage requirements.
Third, for most use-cases for a single instance of the Workbench, 50GB of free disk is sufficient for multiple years of history.
Finally, calculating precise storage requirements is very complex and depends on exactly how your Workbench is configured. In general, these are the primary drivers of storage use (in order of significance):
30) has a roughly-linear effect on temporary storage.
1) affects permanent storage exponentially. With one or two patterns it might be a small portion of the permanent storage requirements, but with e.g. five patterns, it can dominate the permanent storage requirements.
7) has a roughly-linear effect on temporary and permanent storage use.
Here are some examples to help you gauge real-world scenarios:
|Scenario||Single app||Mid-volume, customized||High-volume, heavily customized|
|Average incoming messages per second||0.1||1||10|
|# of pattern indexers||1||2||3|
|Total # of indexers (incl. pattern)||9||10||15|
|Storage after 1 year||7.5 GiB||23.8 GiB||234 GiB|
|Storage after 2 years||7.8 GiB||29.0 GiB||273 GiB|
|Storage after 3 years||8.1 GiB||34.2 GiB||312 GiB|
Note: Please see the large / excessive data section for additional considerations.
The incoming message rate is a function of:
The net result of these two could be anything from "a few messages a day" to "30 messages a second" without stretching the limits of "normal".
To give you a sense of our experience: we rarely see a single production app exceeding 1 message/second. (Most apps are much slower.) But we have some that continually deliver 4-8 per second, and one that sends floods of thousands of messages in a few seconds, every few hours.
Note that if you observe the data transmitted over the network, individual messages are usually transmitted as part of larger message batches.
Bandwidth is used in two primary ways: incoming data, and outgoing queries (usually to the Portal).
For incoming data bandwidth, you'll need to estimate your incoming data rate and then please see the discussion below about incoming message size. In most cases this is extremely low - less than a few messages per second, at a few kilobytes each.
For outgoing (Portal) bandwidth, the rate can vary widely. The key idea is that some data processing is done in the browser - which means that the browser has to download raw data in order to do that processing. The amount of data depends on the amount of data in the Workbench, the time period being queried, the particular report in use, and the number of concurrent users.
To put it in concrete terms, the Overview page, with the default 30-day window, typically uses a few hundred kilobytes of bandwidth. The Version Adoption report, for the same time window, uses an amount of data that varies with the number of distinct application records in the Workbench, but typically 1-2MB. At the far extreme, the Custom Data report on a Workbench with a large amount of custom data, may transfer tens or hundreds of megabytes of data. (Please see large / excessive data for mitigation strategies.)
Most standard messages (e.g. session-start, feature-tick, exception, etc.) are rather small - 1 or 2 kilobytes. But it’s possible to make them much larger, as described below in large / excessive data, so you should consider your instrumentation approach before estimating bandwidth use.
It's also important to note that all the PreEmptive Analytics instrumentation clients/APIs automatically bundle individual messages into "batches" (also called "envelopes") which is what actually goes over the network. On average, under normal instrumentation scenarios, most envelopes have 3-4 messages in them.
To give you a sense of our experience: the average envelope size across all data coming to us from customers is just under 4KB. About 50% of envelopes are ~3KB.
The advice given above is based on some assumptions / averages about message size. Specifically, they assume "typical" patterns of data generation. Atypical patterns, especially around custom data or extremely-high volumes of a single message type, can have a negative effect on the performance and scalability of the Workbench.
To explain: an instrumented application normally sends a variety of pre-defined message types, like app-starts, feature ticks, exceptions, etc. These message-types are typically delivered in relatively-standard distributions - more features than sessions, fewer exceptions than sessions, etc. They also typically have a moderate amount of custom data (e.g. a few fields per feature). Instrumentation data that fits this profile will fit easily within the resource requirements outlined above. But it is possible for applications to send data that greatly exceeds these norms - in which case the information above may not be sufficient.
There are a few specific ways that this might happen:
An application can send an unusually-large amount of custom data. Most applications send some custom data - perhaps a few fields per feature, with values that are a few-dozen bytes. But it is possible for an application to send an arbitrary number of fields, each with a value that is kilobytes long - in which case the storage calculations above are insufficient, and the default Custom Data Server Query will take an especially long time. Most other aspects of the system will be fine.
If you plan to make extensive use of custom data, we recommend removing the default custom data indexers and creating purpose-specific custom data filters instead. This will use much less storage, will greatly improve query performance, and will probably provide much more value from using the Portal.
An application can send unusually-large exception messages. Some applications generate stack traces with multi-kilobyte messages, or thousand-line call stacks. This will increase the storage requirements and will slow the Exceptions report.
If an application sends an extremely-unusual amount (e.g. 1000/session) of any particular message type - e.g. features or exceptions - it can affect performance and storage requirements. Only rarely have we seen this happen, and typically it turns out to be an unintended consequence rather than something intentional.
If you believe that you do need to generate data with such extreme ratios, please contact us - we can often suggest better ways to instrument that will reduce load on the app and on the Workbench, and will make the data easier to analyze.