Sovereign 8s.is mirror of standardgalactic/Aif_Reinforcement_Failing
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2025-09-19 11:41:15 +02:00
auxiliary added Zhang data 2025-05-22 11:00:55 +02:00
data add training data 2025-05-26 11:16:44 +02:00
figures add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
initial_submission_plots added Zhang data 2025-05-22 11:00:55 +02:00
.gitignore iterate scripts 2025-03-25 10:55:46 +01:00
Figure_1_a_zhang_mtd.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
Figure_1_b_zhang_at.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
Figure_1_f_mtd_fit.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
Figure_1_g_at50_fit.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
Figure_3_bc_trajectories_ODE_ABM.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
Figure_3_d_TTF_vs_min_max.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
Figure_3_g_agents_divergence.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
Figure_3_kl_divergence_performance.py NMI submission 2025-09-19 11:41:15 +02:00
Figure_4_bc_ratchet.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
Figure_4_d_kde.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
Figure_5_b_bimodal_growth_all_environments.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
Figure_5_cd_trajectories_AE_TE.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
Figure_5_e_AE_kl_divergence.py modified some si plots for talk 2025-07-10 11:43:09 +02:00
Figure_5_fg_ratchet.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
Figure_5_h_kde.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
Figure_5_ij_probing_cycles.py modified some si plots for talk 2025-07-10 11:43:09 +02:00
old_Figure_5_d_slv_updated_performance.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
old_Figure_5_ratchet.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
old_Figure_uknown_postion_phase_space.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
Readme.md add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
SI_figure_3_ab_res_vs_tot_trajecory.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
SI_figure_3_c_separatrix.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
SI_figure_3_de_heatmap.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
SI_figure_3_fg_trajectories.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
SI_figure_4_b_growth_rate_distribution.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
SI_figure_4_c_velocity.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
SI_figure_5_a_schematic_correct_position_dependece.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
SI_figure_5_b_agents_lv_slv_pc.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
SI_figure_agents.py added Zhang data 2025-05-22 11:00:55 +02:00
SI_figure_at50_mtd.py added Zhang data 2025-05-22 11:00:55 +02:00
SI_figure_bayesian_inference_parameters_distribution.py added Zhang data 2025-05-22 11:00:55 +02:00
SI_figure_extended_data_2_lv_slv_pc.py modified some si plots for talk 2025-07-10 11:43:09 +02:00
SI_figure_extended_data_average_total.py add figure's pngs to readme, update names of script files with current version of panel labels 2025-05-26 11:15:17 +02:00
SI_figure_growth_layer_plot.py added Zhang data 2025-05-22 11:00:55 +02:00
Si_figure_growth_layer_prep.py added Zhang data 2025-05-22 11:00:55 +02:00
SI_figure_many_trajectories.py added Zhang data 2025-05-22 11:00:55 +02:00
SI_figure_probing_comparison.py modified some si plots for talk 2025-07-10 11:43:09 +02:00
Si_figure_spatial_model_fit_growth_rate.py added Zhang data 2025-05-22 11:00:55 +02:00
Si_figure_spatial_model_fit_velocity.py added Zhang data 2025-05-22 11:00:55 +02:00
SI_figure_tensorboards.py added Zhang data 2025-05-22 11:00:55 +02:00
SI_figure_trajectories_agent_treatment.py update scripts 2025-05-26 10:51:15 +02:00
SI_figure_trajectories_augmented_agent_treatment.py update scripts 2025-05-26 10:51:15 +02:00

Reinforcement Failing paper Figure scripts

This repository contains scripts for generating plots for individual panels of figures for the Aif et al. Reinforcement Failing paper.

Figure 1

Figure 1 Figure 1: A physical model for adaptive tumor therapy. a, b Patient data of normalized PSA values for continuous and adaptive therapy AT50 (adapted from Ref. [5]). Solid lines show competition model fits to individual patient data as reported in that study. c, d Schematics of the mathematical c and physical d tumor growth models. e Snapshots of the physical model simulation under Continuous Therapy (CT, Supplementary Movie 2) and Adaptive Therapy (AT50, Supplementary Movie 3). Sensitive cells are shown in blue, resistant cells are shown in orange. Brighter colors indicate higher proliferation rates of the cells. Grey bars correspond to regions of treatment application. f Dynamics of the sensitive (blue) and resistant (orange) population sizes, relative to the total initial population size, in the physical model (triangles) and fitted mathematical model (crosses), under CT. Grey regions indicate treatment application. g Population size dynamics under AT50. Black circles mark the time points shown in the snapshots in e. h Reinforcement learning approach representation. i Schematic of performance degeneracy and degeneracy lifting via larger complexity of the Physical Model.

Figure 2

Figure 2

Figure 2: Schematic of the Reinforcement Failing framework. The Reinforcement Failing framework is constituted of four main blocks. Environment construction: Based on the empirical data and fundamental physical principles, a high-fidelity simulation is designed as target environment (TE). The complex target environment is then abstracted into a mathematical base training training environment (BE). Reinforcement learning: An ensemble of multiple independent agents with distinct policies is trained to optimize outcome based on a predefined reward function. Group-Relative Agent Performance Evaluation (GRAPE): Evaluating all agents in both BE and TE yields a cross-environment performance difference (CEPD) map, using the KL divergence from the least performing agent as policy space coordinate. Failure analysis: Detailed analysis of differentially performing agents by a human researcher guides the refinement of the training environment to yield a more accurate representation in an augmented training environment (AE), followed by the training and evaluation of second-generation agents.

Figure 3

Figure 3

Figure 3: Learning from failure: Systematically analyzing the mechanisms of agent failure in the target environment. a Time to treatment failure (TTF) in BE (green) and TE (violet) for policies trained in BE (n=10). Dots represent averages over n=50 runs, error bars indicate the standard deviation. Policies are spaced on the x-axis based on their Kullback- Leibler divergence from a reference policy. On the right standardized cross-environment performance differences are shown for all (n=10) policies. b, c Examples of the dynamics of the sensitive (blue) and resistant (orange) population sizes, relative to the total initial population size, under the policies that performed best and worst in the target environment, shown in both TE (Supplementary Movies 4 and 5) and BE. Grey regions indicate treatment application. Dashed lines show the tumor burdens at treatment initiation (black) and treatment pause (grey). Arrows point to the application of treatment at low tumor burden by the agent using the worst policy. d Time to treatment failure (TTF) in BE (green) and TE (violet) for n=10 policies trained in BE as a function of the lowest total population size at which treatment was initiated. Dots represent averages over n=50 runs, and error bars indicate the standard deviation.

Figure 4

Figure 4

Figure 4: Position-dependent growth and resulting motion gradients lead to history-dependent effects in the target environment a Selected simulation snapshots of the TE under the worst policy. Sensitive cells in blue; resistant cells in orange. Brighter colors indicate larger growth rates (see Supplementary Movie 6). b Closest distance from any resistant cell to the nearest point on the tumor front over time for the simulation run presented in a. Orange represents treatment holidays, while grey shows the time points when the treatment was applied. c Relationship between the distance to the front and the total population size, relative to the initial tumor burden. Circles of varying brightness mark the time points corresponding to the snapshots in a. d Radial distribution of resistant cells in time for the same simulation run. Brighter lines represent later time points.

Figure 5

Figure 5

Figure 5: Augmenting the training environment to reflect spatial dynamics. a Schematic of the augmented environment (AE). b Resistant population dynamics over time in the three environments, BE - green, AE - blue-green and TE - violet. c, d Relative population size dynamics od sensitive (blue) and resistant (orange) cells in time under a median performing policy trained in the augmented environment evaluated in both AE c and TE d. Grey bars indicate treatment application. Dashed lines show the tumor burdens at treatment initiation (black) and treatment pause (grey). e Time to treatment failure (TTF) in AE (blue-green) and TE (violet) for policies trained in AE (n=10). Dots represent averages over n=50 runs, error bars indicate the standard deviation. f Closest distance from any resistant cell to the nearest point on the tumor front over time for the simulation run presented in d. Grey represents treatment application, orange - treatment holidays. g Relationship between the distance to the front and the total population size, relative to the initial tumor burden. h Radial distribution of resistant cells in time for the same simulation run. Brighter lines represent later time points. i, j Dynamics of sensitive and resistant cells under the probing cycles for 8 generations of AT50 i and AT75 j therapies following with treatments applied by the best performing agent in AE.