Master Project 02 · Research case study

Phase 6 · G8 pass

Forecast-free control for a four-source microgrid.

Four cooperative policies stabilise an islanded 5.5 V DC bus, protect its battery and prioritise a critical load — tested against droop and MPC across a frozen Phase 6 release.

Research verdict

No universal winner. Learned control leads the balanced slow case; domain randomisation strengthens default; classical droop remains the better fast reference.

Phase 6 architecture: four learned policies control PV, wind, biogas and battery sources connected to a shared 5.5 volt bus with a 7.5 farad equivalent supercapacitor and two loads. — **Final Phase 6 architecture.** Power paths, shared observation and the 7.5 F equivalent bus buffer in one view.

Role: Solo — model, training & evaluation
Context: Master's project (MP2), HSBI
Supervisor: Prof. Dr. Jörn Loviscach
Headline arm: Multi-Policy PPO · BC + DR ablation
Release: 17 July 2026 · T7 complete
Source: github.com/amarakramali/mp2-microgrid-marl

View source on GitHub

252

released evaluation artifacts

held-out seeds per comparison

693

tests passed at release gate

157h

observed training wall-clock

01 / From hardware to control

The hardware becomes a control problem.

MP1 builds the physical energy node. MP2 turns that setup into a physics-based digital twin and asks a sharper question: where can learned source controllers beat droop and MPC — and where do the classical controllers still deserve respect?

Bus voltage is not clamped by an ideal supply. It moves with the net source current through the 7.5 F buffer, so voltage becomes the visible consequence of every control decision — the DC analogue of frequency in a larger AC grid.

Slow · 30 sLearned balance

MP-PPO leads the balanced case.

Strong voltage control, 98.9% critical supply and healthy terminal SOC.

Best portfolio result

Default · 5 sDR helps

Randomisation improves the raw policy.

Lower RMSE and saturation with better supply and reward across held-out seeds.

Cleanest DR gain

Fast · 1 sClassical lead

Droop remains the stronger reference.

The short coupled scenario resists a simple “faster RL is better” conclusion.

Honest boundary

02 / Plant model

A small grid with real constraints.

The simulation integrates bus physics at one-second substeps, while the controller acts at 30 s, 5 s or 1 s intervals. Slow and default use 48-hour episodes; the fast setting has a shorter horizon and scenario coupling, so cross-configuration differences are reported as configuration effects — not as a pure action-interval causal claim.

Photovoltaic

0–1 A

Passive source, temperature-adjusted and curtailed by its policy. Phase 6 is bypass-only.

Wind

2–12 m/s

Cubic turbine curve from cut-in through rated speed to cut-out.

Biogas

0.3–1 A

Dispatchable source with ramp, minimum runtime and cooldown constraints.

Battery

±1 A

0.5 Ah bidirectional storage with SOC limits and 95% charging efficiency.

Dynamic statedV_bus/dt = (ΣI_sources − ΣI_loads) / 7.5 F5.5 V nominal · 4.5–6.0 V hard model bounds

03 / Learning setup

One policy per source, one shared objective.

The headline controller is best described as Multi-Policy PPO: four PPO policies receive the same 21-dimensional global observation and optimise one cooperative reward. It is not the original centralised-critic MAPPO architecture, so the project names the implementation precisely.

Behaviour cloning from droop provides the warm-start. The Phase 6 matrix then compares it with local-observation IPPO, single-agent PPO, constrained Lagrangian PPO, domain-randomised training and two inference-time shield settings. Classical references include droop, heuristic MPC with three forecast-noise levels and convex MPC.

01Train5 seeds · 200 iterations · 3 action intervals

→

02Freezebest or final checkpoint by declared rule

→

03Evaluate20 disjoint held-out seeds · no train leakage

→

04Release10k paired bootstrap · Holm-corrected reports

Why the gate structure mattered. Single-agent PPO looked attractive during training, then failed to generalise: in the default held-out evaluation its critical supply collapsed. Phase 6 reports the held-out test result, not the flattering training curve.

04 / Experiment scale

The result took 157 run-hours before it became a 252-job release bundle.

The main Phase 6 campaign trained five controller families at three action intervals and five independent seeds: 75 complete jobs, 200 iterations each. Their measured wall-clock spans sum to 156.8 hours; the matrix itself ran from 14 to 25 June 2026.

The final T7/G8 bundle then adds 120/120 regular evaluations, 108/108 stress evaluations and 24/24 behaviour traces. A headline Multi-Policy PPO run took a median 2.41 hours; the faster single-agent PPO was not the better held-out controller.

Phase 6 training campaign and release bundle: 75 completed training runs, 156.8 summed run-hours, 252 released evaluation artifacts and T7/G8 PASS. — **Measured training effort.** Timings are observed wall-clock spans from each run's recorded creation time to its completed results file. They describe this campaign's workflow, not a hardware-normalised algorithm benchmark.

5 × 3 × 575 main runs

five algorithms, three action intervals and five seeds

75 × 20015,000 iterations

the health-checked headline training matrix

27 × 5 × 506,750 more

iterations in the reward-weight sensitivity sweep

120 + 108 + 24252 artifacts

regular evaluation, stress scenarios and behaviour traces all released

Controller familySlow · 30 sDefault · 5 sFast · 1 s

Multi-Policy PPO5 seeds · 2005 seeds · 2005 seeds · 200

Multi-Policy PPO + DR5 seeds · 2005 seeds · 2005 seeds · 200

IPPO · local obs5 seeds · 2005 seeds · 2005 seeds · 200

Single-agent PPO5 seeds · 2005 seeds · 2005 seeds · 200

Lagrangian PPO5 seeds · 2005 seeds · 2005 seeds · 200

05 / Phase 6 results

Slow shows the learned controller’s strength — fast shows why the story stays honest.

In the completed slow configuration, raw Multi-Policy PPO reduces mean voltage RMSE from 0.554 to 0.291, raises critical-load supply from 65.6% to 98.9%, and ends with mean battery SOC 0.786 instead of 0.052. The domain-randomised arm improves RMSE further to 0.286, but slightly lowers critical supply, final SOC and reward — a real trade-off, not free dominance.

In default, domain randomisation improves RMSE, saturation, critical supply and reward over the raw learned arm. In fast, droop is stronger than the DR arm on the main voltage, saturation, supply and reward metrics. The final page therefore presents Phase 6 as a configuration-dependent result instead of a universal “RL wins” claim.

Slow configuration · held-out evaluation

What changed against droop

MP-PPO Droop

01 / Lower is better

Voltage RMSE

−47.5%

MP-PPO

0.291

Droop

0.554

02 / Higher is better

Critical supply

+33.3 pp

MP-PPO

98.9%

Droop

65.6%

03 / Terminal state

Battery SOC

+0.734

MP-PPO

0.786

Droop

0.052

Bars are normalised within each metric. They show the paired slow-case contrast, not one combined score.

Phase 6 slow benchmark comparing voltage RMSE and critical-load supply across Lagrangian PPO, domain-randomised Multi-Policy PPO, Multi-Policy PPO, perfect-forecast MPC, noisy MPC and droop. — **Two objectives, not one leaderboard.** Raw Multi-Policy PPO is the strongest balanced slow result. Lagrangian PPO reaches the lowest voltage RMSE, but pushes final SOC to 0.922 near the upper boundary and earns a worse total reward — an important Phase 6 trade-off.

Voltage0.291vs 0.554 droop

Critical supply98.9%vs 65.6% droop

Final SOC0.786vs 0.052 droop

DR-mild RMSE0.286better RMSE, but not free dominance

Forecast robustness

MPC degrades exactly where the learned policy is independent.

With a perfect forecast, MPC is competitive and in some metrics better. At the Phase 6 15% noise setting, however, its RMSE rises to 0.548 and critical supply falls to 68.4%. The learned controller has no forecast input, so this specific error channel does not exist for it.

Phase 6 chart showing MPC voltage error rising and critical-load supply falling as forecast noise increases, while forecast-free Multi-Policy PPO remains outside that failure mode. — **The forecast-free advantage.** The horizontal MARL reference is not a claim that its performance changes with MPC noise; it shows that the learned policy consumes no forecast, while MPC is evaluated at perfect, 5% and 15% forecast-noise settings.

06 / Original Phase 6 evidence

Eight released plots — organised for exploration, not dumped into a wall of charts.

The summary graphics above explain the result quickly. This browser keeps the original Phase 6 plots available underneath: three Holm-corrected effect-size matrices and five held-out behaviour traces. Every trace uses the same unseen EVAL seed 100, so the controller families face one directly comparable scenario.

Statistical view

Paired effect sizes across the full method family

Slow Phase 6 paired Cohen's d heatmap comparing all controller families with domain-randomised Multi-Policy PPO. — **Slow · 30 s actions · 48 h episode** The reference is MP-PPO + BC + DR. Stars use Holm-adjusted p-values; the sign must be read with the metric direction because lower RMSE is good while higher supply and reward are good.

Behaviour view

What the controllers actually do over time

Slow Phase 6 held-out trace comparing Multi-Policy PPO, domain-randomised Multi-Policy PPO, droop and perfect MPC. — **Slow core · MP-PPO, DR, droop and perfect MPC over 48 h** From top to bottom: bus voltage, battery SOC, source actions and battery action. The plots expose control behaviour that a single aggregate score cannot show.

8 / 11 release figures shown The three original MPC-noise exports are represented by the cleaner forecast-robustness graphic above; the underlying conclusion and caveat remain unchanged.

07 / What Phase 6 established

Strong evidence, with the caveats left visible.

EstablishedReleased evidence

T7 complete / G8 pass on 17 July 2026 for the frozen Phase 6 result bundle.
120/120 regular evaluations, 108/108 stress jobs and 24/24 behaviour traces are valid.
Held-out EVAL seeds 100–119 stay disjoint from demo, training and tuning seeds.
Statistics use one EVAL seed as the unit, 10,000 paired bootstrap resamples and Holm correction.
Final artifacts include 3 regular reports, 9 stress reports, paper tables, 11 PNG/PDF figure pairs and 693 passed tests.

BoundariesClaims kept scoped

There is no universal winner: slow, default and fast support different operational conclusions.
DR improves default clearly, but in slow it trades lower RMSE/saturation for slightly lower supply, SOC and reward.
The fast setting changes horizon/scenario together with action interval, so it is not a pure speed-causality study.
Perfect MPC is an oracle benchmark without voltage feedback; it is not a deployable MPC claim.
Inference-time shields do not establish formal safety and hardware results remain compute-feasibility, not field performance.

08 / Technology

A reproducible simulation and evaluation stack.

PythonPyTorchRay RLlibPettingZooGymnasiumNumPycvxpy · OSQPSciPypytestMQTT · HIL bridge

Source of truthErgebnisse / Phase 6ReleaseT7 complete · G8 passRelease commit15befd5Bundle frozen17 Jul 2026