Workload Analysis for Server Farm Optimization (WASFO) toolset
By Mauro Gatti
For more on this see also: WASFO Data Collector, WASFO Analysis and Optimization,WASFO contributors and WASFO communities
For comments write to: Mauro Gatti
The WASFO toolset is a set of tools designed and developed to support the management and optimization processes of datacenters through data collection, data analysis and extensive use of optimization algorithms. It currently consists of two main tools:
- The WASFO Data Collector tool.
- The WASFO Analysis and Optimization tool.
In this web page we describe how these tools can be used to manage and optimize a datacenter; and we highlight the key benefits of their use.
We respond hereunder to the following questions:
- How the WASFO toolset can be used to improve server management and datacenter optimization?
- What are the most important benefits of using the WASFO toolset?
- What is WASFO Data Collector?
- What is WASFO Analysis and Optimization?
Server management and datacenter optimization
Next are typical uses of the WASFO toolset:
- Capacity planning for server virtualization projects
- Capacity planning for server consolidation projects
- Server inventory
- Virtual machines inventory, analysis and optimization
- SW inventory
- Server health check
For each of these typical uses we provide hereunder further explanations. We emphasize that there are many more possible uses than those listed in this page.
Virtualization capacity planning
Capacity planning for server virtualization projects is a complex effort that requires
- extensive inventory and workload data collection;
- inventory and workload data analysis; and
- the use of comprehensive and efficient optimization algorithms.
WASFO Data Collector tool can be used to collect inventory and workload data; while WASFO Analysis and Optimization tool can be used to analyze the collected data to optimize the server farm.
Server capacity planning
Capacity planning for a server consolidation project not targeting exclusively server virtualization has challenges that are quite close to those of a virtualization project. Data collection, data analysis and optimization problems are almost identical to those of a virtualization project. The main differences are:
- There is no need to check the virtualization feasibility.
- There is no need to estimate the virtualization overhead.
Server inventory
Virtually any firm has a list of its own IT assets but it is our experience that server farm inventories often have the following limitations:
- They are typically outdated. Some existing servers do not appear in the inventory; some other servers that appear in the inventory have actually been disposed; server configuration descriptions may be outdated because of upgrades that were not registered in the inventory.
- They typically provide only inventory information; namely they do not provide any data about workloads.
Moreover administrators often have a limited understanding of their access capabilities. They do not have access to all servers; they do not fully understand the limits of their security credentials, and so on.
Virtual machines inventory, analysis and optimization
The introduction of virtualization in x86 server farms has dramatically affected the server market. The size of the server farms is no longer expected to growth as quickly as in the past in terms of number of boxes. On the contrary the number of deployed Virtual Machines is growing and will likely keep on growing in the near future.
One of the unpleasant consequences is that the server farm administrators have now a new problem to face: the virtual machines sprawl.
An accurate inventory of virtual machines is important to keep under control the virtual server farm notably for:
- compliance (e.g., for licensing)
- maintenance and management efficiency.
SW inventory, analysis and optimization
The aforementioned VM sprawl issue can only exacerbate the problem of keeping the application portfolio under control. This is of course important in order to:
- Control licensing costs.
- Simplify the applications maintenance and management.
Server farm health check
Checking the health status of a server farm is a quite challenging activity that is usually postponed until big problems force to pay attention to problems. Windows servers provide a enormous amount of information through Windows events. Most of the time system administrators ignore warning events, often even error events, and to not see at all events anomalies (e.g., a spike in event production).
Following the old adage that prevention is better than cure we recommend events data collection and automatic analysis thereby getting an indication of the health status of the server farm and pinpointing those area that need an accurate analysis.
Key features of the WASFO toolset
Key features of the WASFO toolset are:
- Efficient multi-threaded data collection. Collecting from many servers is far more complex than collecting data from a single server as all those that have tried to do that using Microsoft Performance Monitor know very well.
- Automation of inventory and workload analysis. This is a very long and boring activity whose automation most tools gloss over altogether.
- Optimization algorithms that identify the 'cheapest' solution. This is a quite unique feature of IBM WASFO Analysis and Optimization tool that has required years of theoretical analysis and SW development.
Last but not least, the WASFO toolset has an extensible structure that allows to introduce quickly new features requested by the users. In the last few months we have introduced the following features on user requests:
- Serial number data collection (Windows and Linux)
- Services data collection (Windows)
- Events data collection (Windows)
- Custom WMI data collection (Windows)
Others data collection and analyses capabilities are being developed.
Data Collector
The WASFO Data Collector is a server farm data collector tool. It currently collects:
- basic inventory data (e.g., number of CPUs, CPU models, etc.);
- basic workload data (e.g., CPU utilization, memory utilization);
- basic virtual machines data (e.g., number of shares);
- applications inventory;
- basic connections data (output of netstat command).
Data can currently be collected from Windows, Linux, AIX or VMware vSphere servers.
The key distinguishing features of WASFO Data Collector are:
- Data collection is agent-less; in other words nothing has to be installed on the monitored system.
- Data collection extensively leverages multi-threading to make data collection as fast and efficient as possible.
- Tool developers are also tool users thereby fostering a design well tuned for the typical data collection work.
Analysis and Optimization
The WASFO Analysis and Optimization tool makes basic analyses of inventory and workload data so that they can be used to solve the optimization problem of finding the minimum cost way to virtualize the current server farm.
Inventory analysis consists in finding in an automatic way the performance capacity of the monitored servers based on the collected data; where by performance capacity we mean an estimate of the computational capability of the server. Such an estimate can be obtained manually from benchmarks (e.g., TPC-C) or much more quickly by identifying the server in IDEAS International tables. Although quicker than having to compute performance capacity estimate directly from benchmarks, the manual search of the matching server in IDEAS International tables is a slow and error prone process. WASFO makes such a process much faster by means of algorithms that support the identification process.
Workload analysis is a complex process that aims at:
- Handling "data holes", namely missing data due for instance to the fact that servers from which data were being collected were turned off during the data collection process.
- Normalizing the data so that it is possible to compare data collected from servers with different computational capabilities.
- Making basic statistical analysis so that we take into account that collected data are just one instance of a stochastic process.
A mathematical model of the virtualization process using Binary Programming has been developed and embedded in WASFO Analysis and Optimization so that the tool using an Optimization Engine could compute the optimal way to virtualize the server farm. Herein optimal means minimum cost, where costs include factors like servers purchase cost, power consumption, floor space utilization etc.
Contributors
The WASFO toolset has been developed over the years under the leadership of Mauro Gatti by many people.
Communities
Two communities have been created on Linked-In to create a social network among those that have a stake in WASFO:
The WASFO Development Team is the small community of those tha over the years have contributed to WASFO development.
The WASFO users is a larger community to which all those that use WASFO can participate to get in touch with the other users, get news about the development and interact with the development team.
