Agent-Based Modeling and Simulation - Feature Spaces, Macroscopic Statistical Analysis, and Specification Fidelity
DOI:
https://doi.org/10.19153/cleiej.29.4.7Abstract
Agent-Based Models (ABMs) are widely used to simulate complex systems through emergent behavior. Agent Based Modeling practice is constrained by the lack of systematic methods to evaluate and select frameworks for a given problem, and by the absence of protocols to study differences across computational implementations. We propose a Feature Space Maturity Model (FSMM) to evaluate how well a framework matches the requirements of a modeling task. We present results of four canonical ABM experiments to demonstrate the FSMM applied to five well-known frameworks. Our results suggest that combining the FSMM with the proposed protocol provides valuable information for both framework builders and modelers facing complex technical and research choices.
Ensuring ABM reliability requires specification fidelity -- meaning both the conceptual correctness of the model and its faithful implementation. We study implementation fidelity by analyzing statistical similarity across multiple frameworks. We apply a macroscopic statistical approach to evaluate whether different ABM frameworks yield functionally equivalent results when executing the same specification. We designed a Pareto-based experiment and implemented it in six ABM frameworks. We defined three macroscopic observables and analyzed 10,000 simulation runs per framework using coefficient of variation, Fréchet distance, ANOVA, and F-tests. Five of the six frameworks show statistically indistinguishable behavior, while one exhibits consistent deviations. The results highlight the value of statistical validation to identify implementation inconsistencies and optimize resource allocation in large-scale simulations.
References
F. Li Vigni, “Regimes of evidence in complexity sciences,” Perspectives on Science, vol. 29, no. 1, pp. 62–103, 2021.
J. M. Epstein, Generative social science: Studies in agent-based computational modeling. Princeton University Press, 2012.
C. M. Macal, “Everything you need to know about agent-based modelling and simulation,” Journal of Simulation, vol. 10, no. 2, pp. 144–156, 2016.
H. Sabzian, M. A. Shafia, A. Bonyadi Naeini, G. Jandaghi, and M. J. Sheikh, “A review of agent-based modeling (ABM) concepts and some of its main applications in management science,” Interdisciplinary Journal of Management Studies (Formerly known as Iranian Journal of Management Studies), vol. 11, no. 4, pp. 659–692, 2018.
C. W. Seagren, “Examining social processes with agent-based models,” The Review of Austrian Economics, vol. 24, pp. 1–17, 2011.
D. A. Robertson, “Agent-based models and behavioral operational research,” Behavioral Operational Research: Theory, Methodology and Practice, pp. 137–159, 2016.
J. C. Stevenson, “Agentization of two population-driven models of mathematical biology,” in Conference of the Computational Social Science Society of the Americas. Springer, 2021, pp. 176–189.
S. N´u˜nez-Corrales, M. Friesen, S. Mudigonda, R. Venkatachalapathy, and J. Graham, “In-silico models with greater fidelity to social processes: towards ABM platforms with realistic concurrency,” in Proceedings of the 2020 Conference of The Computational Social Science Society of the Americas. Springer, 2021, pp. 155–169.
A. Antelmi, G. Cordasco, G. D’Ambrosio, D. De Vinco, and C. Spagnuolo, “Experimenting with agentbased model simulation tools,” Applied Sciences, vol. 13, no. 1, p. 13, 2022.
L. C. Lara Lopez, “On the feature space and architecture of ABM frameworks,” M.Sc. Thesis, Tecnol´ogico de Costa Rica, Cartago, Costa Rica, 2024. [Online]. Available: https: //hdl.handle.net/2238/16413
F. L. Bellifemine, G. Caire, and D. Greenwood, Developing Multi-Agent Systems with JADE. John Wiley, 2008.
C. Macal and M. North, “Agent-based modeling and simulation: Desktop ABMS,” Proceedings - Winter Simulation Conference, pp. 95–106, 12 2007.
M. Gardner, “Mathematical games: the fantastic combinations of john conway’s new solitaire game “life”,” Scientific American, vol. 223, no. October, pp. 120–123, 1970.
K. Nygaard and O.-J. Dahl, The Development of the SIMULA Languages. New York, NY, USA: Association for Computing Machinery, 1978, p. 439–480. [Online]. Available: https://doi.org/10.1145/800025.1198392
S. Abar, G. K. Theodoropoulos, P. Lemarinier, and G. M. O’Hare, “Agent based modelling and simulation tools: A review of the state-of-art software,” Computer Science Review, vol. 24, pp. 13–33, 2017. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1574013716301198
N. J. Roese, “Counterfactual thinking.” Psychological Bulletin, vol. 121, pp. 133–148, 1997. [Online].
Available: https://doi.org/10.1037/0033-2909.121.1.133
B. D. L. Marshall and S. Galea, “Formalizing the role of agent-based modeling in causal inference and epidemiology,” Am J Epidemiol, vol. 181, no. 2, pp. 92–99, Dec. 2014.
J. Becker, R. Knackstedt, and J. Poeppelbuss, “Developing maturity models for it management,” Business & Information Systems Engineering, vol. 1, pp. 213–222, 06 2009.
S. N´u˜nez-Corrales and L. Gasser, “Scalable social simulation: Evaluation of current frameworks and a new approach,” in Proceedings of the 2018 International Conference on Social Computing, BehavioralCultural Modeling, & Prediction and Behavior Representation in Modeling and Simulation (SBP-BRiMS 2018), 2018.
IEEE Computer Society, Guide to the Software Engineering Body of Knowledge (SWEBOK Guide), Version 4.0. Los Alamitos, CA, USA: IEEE Computer Society, 2024. [Online]. Available: https://www.computer.org/education/bodies-of-knowledge/software-engineering/v4
Systems and Software Engineering — Systems and Software Quality Requirements and Evaluation (SQuaRE) — Product Quality Model, International Organization for Standardization and International Electrotechnical Commission Std. ISO/IEC 25 010:2023, 2023.
R. Sargent, “Verification, validation and accreditation of simulation models,” in 2000 Winter Simulation Conference Proceedings (Cat. No.00CH37165), vol. 1, 2000, pp. 50–59 vol.1.
O. Balci, “Verification validation and accreditation of simulation models,” in Proceedings of the 29th Conference on Winter Simulation, ser. WSC ’97. USA: IEEE Computer Society, 1997, p. 135–141.
[Online]. Available: https://doi.org/10.1145/268437.268462
E. W. Weisstein, “Moore neighborhood,” n.d., from MathWorld–A Wolfram Web Resource. [Online]. Available: https://mathworld.wolfram.com/MooreNeighborhood.html
——, “von neumann neighborhood,” n.d., from MathWorld–A Wolfram Web Resource. [Online]. Available: https://mathworld.wolfram.com/vonNeumannNeighborhood.html
J. Cao, X. Feng, J. Lu, and S. Das, “Mailbox-based scheme for mobile agent communications,” Computer, vol. 35, no. 9, p. 54–60, 2002.
G. J. Maclay and M. Ahmad, “An agent based force vector model of social influence that predicts strong polarization in a connected world,” PLoS One, vol. 16, no. 11, p. e0259625, Nov. 2021.
Z. Li, C. H. Sim, and M. Y. H. Low, “A survey of emergent behavior and its impacts in agent-based systems,” in 2006 4th IEEE international conference on industrial informatics. IEEE, 2006, pp. 1295–1300.
T. Holland, “Foundations for the modeling and simulation of emergent behavior systems,” in Engineering Emergence. CRC Press, 2018, pp. 217–258.
D. Montgomery, Design and Analysis of Experiments. John Wiley & Sons, 2008. [Online]. Available: http://books.google.de/books?id=kMMJAm5bD34C
H. A. Simon, “Near decomposability and complexity: How,” The Mind, The Brain And Complex Adaptive Systems, p. 25, 2018.
M. A. Janssen and E. Ostrom, “Empirically based, agent-based models,” Ecology and society, vol. 11, no. 2, 2006.
A. Wilhite and E. A. Fong, “Agent-based models and hypothesis testing: an example of innovation and organizational networks,” The Knowledge Engineering Review, vol. 27, no. 2, pp. 221–238, 2012.
A. Namatame and S.-H. Chen, Agent-based modeling and network dynamics. Oxford University Press, 2016.
E. of Mathematics, “Graph theory,” 01 2024. [Online]. Available: http://encyclopediaofmath.org/index.php?title=Graph theory
U. Wilensky and W. Rand, An introduction to agent-based modeling: modeling natural, social, and engineered complex systems with NetLogo. MIT press, 2015.
P. A. Burrough, R. A. McDonnell, and C. D. Lloyd, Principles of geographical information systems, 3rd ed. London, England: Oxford University Press, Apr. 2015.
P. Antosz, T. Szczepanska, L. Bouman, J. G. Polhill, and W. Jager, “Sensemaking of causality in agent-based models,” International Journal of Social Research Methodology, vol. 25, no. 4, pp. 557–567, 2022.
A. Chopra, S. Kumar, N. Giray-Kuru, R. Raskar, and A. Quera-Bofarull, “On the limits of agency in agent-based models,” arXiv preprint arXiv:2409.10568, 2024.
L. Blume, “Agent-based models for policy analysis,” in Assessing the Use of Agent-Based Models for Tobacco Regulation. National Academies Press (US), 2015.
J. M. Epstein and R. Axtell, Growing Artificial Societies: Social Science from the Bottom Up. Brookings Institution Press, 1996.
P. J. Wangersky, “Lotka-volterra population models,” Annual Review of Ecology and Systematics, vol. 9 pp. 189–218, 1978.
F. Hoppensteadt, “Predator-prey model,” Scholarpedia, vol. 1, no. 10, p. 1563, Jan. 2006.
Y. Berman, E. Ben-Jacob, and Y. Shapira, “The dynamics of wealth inequality and the effect of income distribution,” PloS one, vol. 11, no. 4, p. e0154196, 2016.
M. P. Todaro, “Review of Human Development Report 1992 by United Nations Development Programme,” Population and Development Review, vol. 18, no. 2, pp. 359–363, 1992. [Online].
Available: http://www.jstor.org/stable/1973685
V. Pareto, Cours D’´economie Politique. Librairie Droz, 1964.
H. M. Hochman and J. D. Rodgers, “Pareto optimal redistribution,” The American economic review, vol. 59, no. 4, pp. 542–557, 1969.
G. G. Piva, F. L. Ribeiro, and A. S. Mata, “Networks with growth and preferential attachment: modelling and pplications,” Journal of Complex Networks, vol. 9, no. 1, p. cnab008, 2021.
N. Ferguson, “Report 9: Impact of non-pharmaceutical interventions (NPIs) to reduce COVID19 mortality and healthcare demand,” 2020. [Online]. Available: DOI:10.25561/77482
S. N´u˜nez-Corrales and E. Jakobsson, “The epidemiology workbench: a tool for communities to strategize in response to covid-19 and other infectious diseases,” MedRxiv, pp. 2020–07, 2020.
M. M. Ferreira Caceres, J. P. Sosa, J. A. Lawrence, C. Sestacovschi, A. Tidd-Johnson, M. H. U. Rasool, V. K. Gadamidi, S. Ozair, K. Pandav, C. Cuevas-Lou, M. Parrish, I. Rodriguez, and J. P. Fernandez,“The impact of misinformation on the COVID-19 pandemic,” AIMS Public Health, vol. 9, no. 2, pp.262–277, Jan. 2022.
S. P. Mudigonda, S. N´u˜nez-Corrales, R. Venkatachalapathy, and J. Graham, “Scheduler dependencies in agent-based models: A case-study using a contagion model,” in Conference of the Computational Social Science Society of the Americas. Springer, 2021, pp. 56–70.
P. Bevington and D. Robinson, Data Reduction and Error Analysis for the Physical Sciences. McGrawHill Education, 2003. [Online]. Available: https://books.google.co.cr/books?id=0poQAQAAIAAJ
G. Catalin Balan, “MASON: A Java Multi-Agent Simulation Library,” 2009.
U. Wilensky and W. Rand, An Introduction to Agent-Based Modeling: Modeling Natural, Social, and Engineered Complex Systems with NetLogo, ser. The MIT Press. Cambridge: The MIT Press, 2015.
J. Gosling, B. Joy, G. L. Steele, G. Bracha, and A. Buckley, The Java Language Specification, Java SE 8 Edition, 1st ed. Addison-Wesley Professional, 2014.
S. Luke, Multiagent simulation and the MASON library. George Mason University (CC), August 2019.
N. Collier, “Repast: An extensible framework for agent simulation,” The University of Chicago’s social
science research, vol. 36, p. 2003, 2003.
M. J. North, N. T. Collier, J. Ozik, E. R. Tatara, C. M. Macal, M. Bragen, and P. Sydelko, “Complex adaptive systems modeling with Repast Simphony,” Complex Adaptive Systems Modeling, vol. 1, no. 1, p. 3, Mar 2013. [Online]. Available: https://doi.org/10.1186/2194-3206-1-3
K. A. Barclay and W. J. Savage, “Groovy programming : an introduction for Java developers,” Amsterdam, 2007. [Online]. Available: http://www.books24x7.com/marc.asp?bookid=47160
E. Burnette, “Eclipse IDE pocket guide,” Sebastopol, CA, 2005.
G. Van Rossum and F. L. Drake, Python 3 Reference Manual. Scotts Valley, CA: CreateSpace, 2009.
J. Kazil, D. Masad, and A. Crooks, “Utilizing Python for agent-based modeling: The Mesa framework,” in Social, Cultural, and Behavioral Modeling, R. Thomson, H. Bisgin, C. Dancy, A. Hyder, and
M. Hussain, Eds. Cham: Springer International Publishing, 2020, pp. 308–317.
G. Datseris, A. R. Vahdati, and T. C. DuBois, “Agents.jl: a performant and feature-full agent-based modeling software of minimal code complexity,” SIMULATION, vol. 0, no. 0, p. 003754972110688, Jan. 2022. [Online]. Available: https://doi.org/10.1177/00375497211068820
J. Bezanson, S. Karpinski, V. B. Shah, and A. Edelman, “Julia: A fast dynamic language for tecnical computing,” arXiv preprint arXiv:1209.5145, 2012.
P. K. Feyerabend, Against Method, 4th ed. London, England: Verso Books, 2010.
M. Roza, J. Voogd, H. Jense, and P. Gool, “Fidelity requirements specification: A process oriented view,” 10 2011.
R. L. V. Horn, “Validation of simulation results,” Management Science, vol. 17, no. 5, pp. 247–258, 1971. [Online]. Available: http://www.jstor.org/stable/2628978
V. M. Yakovenko and J. B. Rosser, “Colloquium: Statistical mechanics of money, wealth, and income,” Reviews of Modern Physics, vol. 81, no. 4, p. 1703–1725, Dec. 2009. [Online]. Available: http://dx.doi.org/10.1103/RevModPhys.81.1703
C. Gini, “Measurement of inequality of incomes,” The Economic Journal, vol. 31, no. 121, pp. 124–126, 1921. [Online]. Available: http://www.jstor.org/stable/2223319
J. Kleinberg and E. Tardos, Algorithm Design. Upper Saddle River, NJ: Pearson, Jun. 2005.
T. D. Foundation, “Libreoffice calc,” 2020. [Online]. Available: https://www.libreoffice.org/discover/
calc/
T. Eiter and H. Mannila, “Computing discrete Fr´echet distance,” 05 1994.
G. Casella and R. L. Berger, Statistical Inference, 2nd ed. Wadsworth Publishing, Chapman and Hall/CRC, 2002, 2024.
J. Brenneman, “Experimental design,” J. Qual. Technol., vol. 52, no. 4, pp. 423–424, Oct. 2020.
M. Fukuda, K. Wang, S. Panther, N. Wong, I. Dudder, and Q. Shao, “An analysis of three c/c++ cluster-computing libraries for agent-based models,” ACM Transactions on Modeling and Computer Simulation, 2025.
T. McPhillips, C. Willis, M. R. Gryk, S. Nunez-Corrales, and B. Lud¨ascher, “Reproducibility by other means: Transparent research objects,” in 2019 15th International Conference on EScience (EScience).
IEEE, 2019, pp. 502–509.
V. Grimm, S. F. Railsback, C. E. Vincenot, U. Berger, C. Gallagher, D. L. DeAngelis, B. Edmonds, J. Ge, J. Giske, J. Groeneveld et al., “The ODD protocol for describing agent-based and other simulation models: A second update to improve clarity, replication, and structural realism,” Journal of Artificial Societies and Social Simulation, vol. 23, no. 2, 2020.
M. van Steenbergen, R. Bos, S. Brinkkemper, I. van de Weerd, and W. Bekkers, “The design of focus area maturity models,” in Global Perspectives on Design Science Research, R. Winter, J. L. Zhao, and S. Aier, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2010, pp. 317–332.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Luis Carlos Lara-López, Ignacio Trejos-Zelaya, Santiago Núñez-Corrales, José Helo-Guzmán
This work is licensed under a Creative Commons Attribution 4.0 International License.
CLEIej is supported by its home institution, CLEI, and by the contribution of the Latin American and international researchers community, and it does not apply any author charges whatsoever for submitting and publishing. Since its creation in 1998, all contents are made publicly accesibly. The current license being applied is a (CC)-BY license (effective October 2015; between 2011 and 2015 a (CC)-BY-NC license was used).
