summaryrefslogtreecommitdiff
path: root/opendc-experiments
AgeCommit message (Collapse)Author
2021-09-11feat(capelin): Implement trace API for Azure VM trace formatFabian Mastenbroek
This change adds a trace API implementation for the Azure VM traces.
2021-09-11fix(capelin): Parse last column in Solvinity trace formatFabian Mastenbroek
This change fixes an issue where the last column in the Solvinity traces is not parsed correctly, due to the last column having no whitespace at the end to seek to.
2021-09-11perf(trace): Keep reader state in own classFabian Mastenbroek
This change removes the external class that holds the state of the reader and instead puts the state in the reader implementation. Maintaining a separate class for the state increases the complexity and has worse performance characteristics due to the bytecode produced by Kotlin for property accesses.
2021-09-10refactor(compute): Integrate fault injection into compute simulatorFabian Mastenbroek
This change moves the fault injection logic directly into the opendc-compute-simulator module, so that it can operate at a higher abstraction. In the future, we might again split the module if we can re-use some of its logic.
2021-09-10refactor(capelin): Terminate servers after reaching deadlineFabian Mastenbroek
This change updates the Capelin experiment helpers to terminate a server when it has reached its end-date.
2021-09-10feat(simulator): Support generic distribution in fault injectorFabian Mastenbroek
This change adds support for specifying the distribution of the failures, group size and duration for the fault injector.
2021-09-07refactor(capelin): Restructure input reading classesFabian Mastenbroek
2021-09-07refactor(capelin): Move metric collection outside Capelin codeFabian Mastenbroek
This change moves the metric collection outside the Capelin codebase in a separate module so other modules can also benefit from the compute metric collection code.
2021-09-07feat(capelin): Report up/downtime metrics in experiment monitorFabian Mastenbroek
2021-09-07refactor(capelin): Make ExperimentMonitor optional for trace processingFabian Mastenbroek
2021-09-02refactor(capelin): Migrate trace reader to new trace APIFabian Mastenbroek
This change updates the trace reading classes in the Capelin experiment to use the new trace API in order to re-use many of the trace reading parts.
2021-09-02refactor(trace): Implement trace API for WTF readerFabian Mastenbroek
This change updates the WTF trace reader to support the new streaming trace API.
2021-09-02refactor(trace): Move Bitbrains format into separate moduleFabian Mastenbroek
This change moves Bitbrains trace support into a separate module and adds support for the new trace api.
2021-09-02refactor(format): Remove environment reader from format libraryFabian Mastenbroek
This change removes the environment reader from the format library since they are highly specific for the particular experiment. In the future, we hope to have a single format to setup the entire datacenter (perhaps similar to the format used by the web runner).
2021-09-02refactor(trace): Extract Parquet helpers into separate moduleFabian Mastenbroek
This change extracts the Parquet helpers outside format module into a new module, in order to improve re-usability of these helpers.
2021-08-25refactor(compute): Measure power draw without PSU overheadFabian Mastenbroek
This change updates the SimHost implementation to measure the power draw of the machine without PSU overhead to make the results more realistic.
2021-08-25refactor(capelin): Simplify metric extraction for monitorFabian Mastenbroek
This change updates the metric exporter logic in the Capelin module and reduces the number of allocations by looping directly over the collection of metrics.
2021-08-25fix(capelin): Eliminate unnecessary double to long conversionsFabian Mastenbroek
This change eliminates the unnecessary conversions from double to long in the Capelin metric processing code.
2021-08-25build: Upgrade to OpenTelemetry 1.5Fabian Mastenbroek
This change upgrades the OpenTelemetry dependency to version 1.5, which contains various breaking changes in the metrics API.
2021-08-24test(capelin): Add tests for interference and failuresFabian Mastenbroek
This change adds tests to the Capelin integration suite for performance interference as well as failures. These test more accurately the experiment setup.
2021-08-24fix(capelin): Update Bitbrains trace testsFabian Mastenbroek
This change updates the Bitbrains trace tests with the updated trace that does not hardcode the duration of the trace fragments.
2021-08-24fix(capelin): Keep trace order after samplingFabian Mastenbroek
This change fixes an issue with the workload sampler where the resulting workload entries would not be ordered properly according to their submission time.
2021-08-24refactor(format): Clean up Bitbrains trace reader to enable re-useFabian Mastenbroek
This change updates the code for the Bitbrains trace reader and upgrades the TraceConverter to re-use existing code of the Bitbrains trace reader.
2021-08-24fix(simulator): Support unaligned trace fragmentsFabian Mastenbroek
2021-08-24fix(simulator): Record overcommit only after deadlineFabian Mastenbroek
This change fixes an issue with the simulator where it would record overcomitted work if the output was updated before the deadline was reached.
2021-08-24refactor(simulator): Execute traces based on timestampsFabian Mastenbroek
This change refactors the trace workload in the OpenDC simulator to track execute a fragment based on the fragment's timestamp. This makes sure that the trace is replayed identically to the original execution.
2021-08-22refactor(compute): Update FilterScheduler to follow OpenStack's NovaFabian Mastenbroek
This change updates the FilterScheduler implementation to follow more closely the scheduler implementation in OpenStack's Nova. We now normalize the weights, support many of the filters and weights in OpenStack and support overcommitting resources.
2021-08-13build: Update Kotlin dependenciesFabian Mastenbroek
This change updates the Kotlin dependencies used by OpenDC to their latest version.
2021-06-24simulator: Re-implement performance interference modelFabian Mastenbroek
This change updates reimplements the performance interference model to work on top of the universal resource model in `opendc-simulator-resources`. This enables us to model interference and performance variability of other resources such as disk or network in the future.
2021-06-24format: Remove performance interference from trace readersFabian Mastenbroek
This change updates the trace reader implementation to remove their dependency on the performance interference model. In a future commit, we will instead pass the performance interference model via the host/hypervisor.
2021-06-21simulator: Re-organize compute simulator moduleFabian Mastenbroek
This change re-organizes the classes of the compute simulator module to make a clearer distinction between the hardware, firmware and software interfaces in this module.
2021-06-21exp: Enable interpreter sharing across hostsFabian Mastenbroek
This change enables the experiments to share the SimResourceInterpreter across multiple hosts, which allows updates to be scheduled efficiently for all machines at the same time. This is especially beneficial if the machines operate on the same time slices.
2021-06-17faas: Rename opendc-serverless to opendc-faasFabian Mastenbroek
This change renames the opendc-serverless module to opendc-faas to better distinguish between the two terms (Serverless and FaaS) and be clearer about the intent of the module. The opendc-faas module holds the code for the FaaS platform on top of OpenDC. Although this is one approach of doing serverless, serverless can also entail other services that will not be covered by this module.
2021-06-16exp: Fix power tracking for energy experimentsFabian Mastenbroek
This change fixes an issue where the power in the energy experiments is always reported as zero due to the changes in commit 652b869.
2021-06-15exp: Fix execution of energy experimentsFabian Mastenbroek
This change fixes the execution of the Energy21 experiments which failed due to various changes in the OpenDC codebase. First, the directory structure is now required to be pre-generated before the writer starts writing the experiment output. Second, we must include the configuration of the Capelin experiment in this experiment in order to workaround an issue with harness filtering.
2021-06-11simulator: Integrate power subsystem with compute subsystemFabian Mastenbroek
This change integrates the power subsystem of the simulator with the compute subsystem by exposing a new field on a SimBareMetalMachine, psu, which provides access to the machine's PSU, which in turn can be connected to a SimPowerOutlet.
2021-06-09build: Eliminate most Hadoop dependenciesFabian Mastenbroek
This change eliminates all Hadoop dependencies that are not necessary for Parquet to work correctly. As a result, the number of dependencies should now be greatly reduced, which in turn leads to less artifacts that need to be retrieved at build time.
2021-06-09exp: Use LocalOutputFile for Parquet writersFabian Mastenbroek
This change updates the Parquet writers used in the Capelin experiments to use our OutputFile implementation for local files, to reduce our dependency on Apache Hadoop.
2021-06-09exp: Use LocalInputFile for Parquet readersFabian Mastenbroek
This change updates the Parquet readers used in the Capelin experiments to use our InputFile implementation for local files, to reduce our dependency on Apache Hadoop.
2021-06-03exp: Remove Jupyter Notebook from Energy experimentsFabian Mastenbroek
This change removes the Jupyter Notebook that was shipped with the initial version of the Energy21 experiments. At the moment, it is not up to date anymore and we will probably move the plotting facility to the web interface.
2021-06-03simulator: Split CPUFreq subsystem in compute simulatorFabian Mastenbroek
This change splits the functionality present in the CPUFreq subsystem of the compute simulation. Currently, the DVFS functionality is embedded in SimBareMetalMachine. However, this functionality should not exist within the firmware layer of a machine. Instead, the operating system should perform this logic (in OpenDC this should be the hypervisor). Furthermore, this change moves the scaling driver into the power package. The power driver is a machine/firmware specific implementation that computes the power consumption of a machine.
2021-06-02simulator: Start consumers directly from workloadFabian Mastenbroek
This change updates the SimWorkload interfaces to allow implementations to start consumers for the machine resource providers directly.
2021-06-02simulator: Add uniform interface for resource metricsFabian Mastenbroek
This change adds a new interface to the resources library for accessing metrics of resources such as work, demand and overcommitted work. With this change, we do not need an implementation specific listener interface in SimResourceSwitchMaxMin anymore. Another benefit of this approach is that updates will be scheduled more efficiently and progress will only be reported once the system has reached a steady-state for that timestamp.
2021-06-01simulator: Centralize resource logic in SimResourceInterpreterFabian Mastenbroek
This change introduces the SimResourceInterpreter which centralizes the logic for scheduling and interpreting the communication between resource consumer and provider. This approach offers better performance due to avoiding invalidating the state of the resource context when not necessary. Benchmarks show in the best case a 5x performance improvement and at worst a 2x improvement.
2021-05-18chore: Address deprecations due to Kotlin 1.5Fabian Mastenbroek
This change addresses the deprecations that were caused by the migration to Kotlin 1.5.
2021-05-18build: Update to Kotlin 1.5.0Fabian Mastenbroek
2021-05-09exp: Add explanation about experimental nature of moduleFabian Mastenbroek
2021-05-09exp: Implement TensorFlow distribution strategiesWenchen Lai
2021-05-09exp: Model TensorFlow compute devicesWenchen Lai
2021-05-09exp: Add simple network model for TensorFlow experimentsWenchen Lai