| Age | Commit message (Collapse) | Author |
|---|
|
|
|
This pull request converts the `opendc-simulator-resources` module into a flow
simulator and adapts the existing low-level models (e.g., CPU, network, disk)
to this new flow simulator.
The flow simulator works differently from the uniform resource consumption
model, in that it models flow through a system of connections, as opposed to
resource consumptions. Concretely, this means that while in the uniform
resource consumption model, consumptions with the same usage are propagated to
the resources, in the flow simulator, only changes to the flow in the system
are propagated.
Overall, this leads to less updates in the system and therefore higher
performance. The benchmarks shows that the new implementation obtains more than
double the performance of the old implementation. We have focused in the new
implementation on reducing the amount of work and memory
allocations/loads/stores per updates.
* Migrate from kotlinx-benchmark to jmh-gradle (for better profiling support)
* Use longer traces for benchmarks (to prevent measuring the benchmark
overhead)
* Use direct field access for perf-sensitive code
* Combine work and deadline to duration
* Add support for pushing flow from context (to eliminate the allocation for
every `SimResourceCommand`)
* Reduce memory allocations in SimResourceInterpreter, by revamping the way
timers are allocated.
* Simplify max-min aggregator implementation (by utilizing the new push
mechanism)
* Invoke consumer callback on every invalidation (in order to propagate changes
downstream)
* Lazily push changes to resource context (by not updating the flow rate
immediately after a push, but only after an update)
* Remove onUpdate callback
* Merge distributor and aggregator into switch
* Separate push and pull flags
* Remove failure callback from FlowSource
* Create separate callbacks for remaining events
* Make convergence callback optional
* Reduce field accesses in FlowConsumerContextImpl
* Optimize hot path in SimTraceWorkload
* Expose CPU time counters directly on hypervisor
* Optimize telemetry collection
**Breaking API Changes**
* The entire `opendc-simulator-resources` module has been replaced by the
`opendc-simulator-flow` module.
* `SimHypervisor.Listener` has been removed in favour of a new interface that
exposes the performance counters of the hypervisor directly. To listen for
convergence, use `FlowConvergenceListener`.
|
|
This change optimizes the telemetry collection in the SimHost class.
Previously, there was significant overhead in collecting the metrics of
this and associated classes due large `Attributes` object that did not
cache accesses to `hashCode()`. We now wrap this object and manually
cache the hash code.
|
|
This change adds a new interface to the SimHypervisor interface that
exposes the CPU time counters directly. These are derived from the flow
counters and will be used by SimHost to expose them via telemetry.
|
|
This change optimizes the hot path of the SimTraceWorkload class. This
class is used by multiple experiments to replay workload traces and
consumes a considerable amount of time of the total time of the
experiments.
By inlining the isExpired method, marking the properties on the Fragment
class as field, and caching several properties locally in the class, we
reduce the amount of virtual method calls in the hot path.
|
|
This change updates the implementation of FlowConsumerContextImpl to
reduce the number of field accesses by storing the flags of the
connection inside a single integer.
|
|
This change adds two new properties for controlling whether the
convergence callbacks of the source and consumer respectively should be
invoked. This saves a lot of unnecessary calls for stages that do not
have any implementation of the `onConvergence` method.
|
|
This change fixes the loss computation for both the UPS and PDU
implementation that was broken due to the new pushing mechanism. We
implement a new class FlowMapper that can be used to map the flow pushed
by a `FlowSource` using a user-specified method.
|
|
This change creates separate callbacks for the remaining events:
onStart, onStop and onConverge.
|
|
This change removes the Capacity entry from FlowEvent. Since the source
is always pulled on a capacity change, we do not need a separate event
for this.
|
|
This change removes the `onFailure` method from FlowSource. Instead, the
FlowConsumer will receive the reason for failure of the source.
|
|
This change reduces the number of operations necessary to manage the
timers of a flow connection.
|
|
This change removes the use of a HashSet for tracking the flow
connections that can converge. A HashSet requires an allocation for
every addition, which caused a significant overhead. The new approach
using an ArrayDeque should not allocate any memory.
|
|
This change separates the push and pull flags in
FlowConsumerContextImpl, meaning that sources can now push directly
without pulling and vice versa.
|
|
This change renames the `opendc-simulator-resources` module into the
`opendc-simulator-flow` module to indicate that the core simulation
model of OpenDC is based around modelling and simulating flows.
Previously, the distinction between resource consumer and provider, and
input and output caused some confusion. By switching to a flow-based
model, this distinction is now clear (as in, the water flows from source
to consumer/sink).
|
|
This change removes the ability to transform the duration of a pull from
the SimResourceForwarder class. This ability is not used anymore, since
pushes are now done using a method instead of a command.
|
|
This change removes the distributor and aggregator interfaces in favour
of a single switch interface. Since the switch interface is as powerful
as both the distributor and aggregator, we don't need the latter two.
|
|
This change removes the use of distributor and aggregator from the other
OpenDC components. For the future, we focus on maintaining a single
SimResourceSwitch implementation to achieve both use-cases.
|
|
|
|
This change hides the SimResourceState from public API since it is not
actively used outside of the `SimResourceContextImpl` class.
|
|
This change removes the `onUpdate` callback from the
`SimResourceProviderLogic` interface. Instead, users should now update
counters using either `onConsume` or `onConverge`.
|
|
This change updates the SimResourceContextImpl to lazily push changes to
the resource context instead of applying them directly. The change is
picked up after the resource is updated again.
|
|
This change updates the simulator implementation to always invoke the
`SimResourceConsumer.onNext` callback when the resource context is
invalidated. This allows users to update the resource counter or do some
other work if the context has changed.
|
|
This change simplifies the implementation of the
SimResourceAggregatorMaxMin class by utilizing the new push method.
This approach should offer better performance than the previous version,
since we can directly push changes to the source.
|
|
This change removes unnecessary allocations in the SimResourceInterpreter
caused by the way timers were allocated for the resource context.
|
|
This change adds a new method to `SimResourceContext` called `push`
which allows users to change the requested flow rate directly without
having to interrupt the consumer.
|
|
This change removes the work and deadline properties from the
SimResourceCommand.Consume class and introduces a new property duration.
This property is now used in conjunction with the limit to compute the amount
of work processed by a resource provider.
Previously, we used both work and deadline to compute the duration and
the amount of remaining work at the end of a consumption. However, with
this change, we ensure that a resource consumption always runs at the
same speed once establishing, drastically simplifying the computation
for the amount of work processed during the consumption.
|
|
This change updates the SimResourceDistributorMaxMin implementation to
use direct field accesses in the perf-sensitive code.
|
|
This change updates the JMH benchmarks to use longer traces in order to
measure the overhead of running the flow simulation as opposed to
setting up the benchmark.
|
|
This change updates the project to use jmh-gradle for benchmarking as
opposed to kotlinx-benchmark. Both plugins use JMH under the hood, but
jmh-gradle offers more options for profiling and seems to be beter
maintained.
|
|
This change updates the Gradle configuration to target Java 11 (instead
of Java 8) as the lowest denominator when building OpenDC. Since the
project has not yet been adopted by (many) other applications, we should
not restrict the project to such an old Java version.
|
|
This pull request addresses some issues with the current implementation of
the `ComputeMetricExporter` class.
In particular, the construction of `ComputeMetricExporter` does not require a `Clock` anymore.
- Ensure shutdown of exporter is called
- Do not require clock for ComputeMetricExporter
- Do not recover guests in non-error state
- Write null values explicitly in Parquet exporter
- Report cause of compute exporter failure
**Breaking API Changes**
- `ComputeMetricExporter` is now an abstract class that can be extended to collect metrics
- `ParquetComputeMonitor` has been renamed to `ParquetComputeMetricExporter` and extends `ComputeMetricExporter`
|
|
|
|
|
|
|
|
This change drops the requirement for a clock parameter when
constructing a ComputeMetricExporter, since it will now derive the
timestamp from the recorded metrics.
|
|
This change updates the CoroutineMetricReader to ensure that the
exporter is shutdown when the metric reader fails or is shutdown.
|
|
Bumps [semver-regex](https://github.com/sindresorhus/semver-regex) from 3.1.2 to 3.1.3.
- [Release notes](https://github.com/sindresorhus/semver-regex/releases)
- [Commits](https://github.com/sindresorhus/semver-regex/commits)
---
updated-dependencies:
- dependency-name: semver-regex
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
|
|
This pull request extends the trace API to support writing new traces.
- Unify columns of different tables
- Support column lookup via index
- Use index lookup in trace loader
- Add property for describing partition keys
- Simplify TraceFormat SPI interface
- Add support for writing traces
**Breaking API Changes**
- `TraceFormat` SPI interface has been redesigned.
|
|
This change adds a new API for writing traces in a trace format.
Currently, writing is only supported by the OpenDC VM format, but over
time the other formats will also have support for writing added.
|
|
This change simplifies the TraceFormat SPI interface by reducing the
number of interfaces that implementors need to implement to only
TraceFormat.
|
|
|
|
This change updates the ComputeWorkloadLoader to use index column
lookups in order to prevent having to lookup the index for every row.
|
|
This change adds support for looking up the column value through the
column index. This enables faster lookup when processing very large
traces.
|
|
This change unifies columns of different tables used by trace formats.
This concretely means that instead of having columns specific per table
(e.g., RESOURCE_ID and RESOURCE_STATE_ID), with this changes these
columns are shared between the tables with a single definition
(RESOURCE_ID).
|
|
This pull request enables re-use of virtual machine workload helpers by extracting the helpers into a
separate module which may be used by other experiments.
- Support workload/machine CPU count mismatch
- Extract common code out of Capelin experiments
- Support flexible topology creation
- Add option for optimizing SimHost simulation
- Support creating CPU-optimized topology
- Make workload sampling model extensible
- Add support for extended Bitbrains trace format
- Add support for Azure VM trace format
- Add support for internal OpenDC VM trace format
- Optimize OpenDC VM trace format
- Add tool for converting workload traces
- Remove dependency on SnakeYaml
**Breaking API Changes**
- `RESOURCE_NCPU` and `RESOURCE_STATE_NCPU` are renamed to `RESOURCE_CPU_COUNT` and `RESOURCE_STATE_CPU_COUNT` respectively.
|
|
This change removes the dependency on SnakeYaml for the simulator. It
was only required for a very small component of the simulator and
therefore does not justify bringing in such a dependency.
|
|
This change adds an initial implementation to the trace library for
converting between workload trace formats. Currently the tool supports
only converting to the OpenDC VM trace format. However, in the future,
we will add support for converting between other formats as well.
|
|
This change optimizes the OpenDC VM trace format by removing
unnecessary columns as well as optimizing the writer settings.
The new implementation still supports reading the old trace format in
case users run OpenDC with older workload traces.
|
|
This change adds official support to the trace library for the internal
VM trace format used by OpenDC for its experiments. This is a compact
format that uses Parquet to store the virtual machine trace data in two
Parquet files.
|
