Tuesday, July 13, 2021

Kinetic Action and Radicalization

In the past we had posted on models of radicalization, but such models were rather abstract.  However in a recent paper entitled "Kinetic Action and Radicalization: A Case Study of Pakistan" which was presented at the  International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation (or SBP-BRiMS for short) we take such work a step further. 

In the paper, Brandon Shapiro and myself develop and present a simple agent-based model informed by theory and calibrated using empirical data to explore the relationship between kinetic actions (i.e., drone strikes) and terrorist attacks in Pakistan from 2004 through 2018. The data itself came from the Bureau of Investigative Journalism data as our source for Pakistan drone strikes (i.e., kinetic actions) and the National Consortium for the Study of Terrorism and Responses to Terrorism( START)  Global Terrorism Database (GTD) as our source for terrorist incidents. 

Rather than try to pinpoint and define the motivating factors which might influence somebody down a path toward radicalization, our model that incorporated a distributed lag model to characterize the inter-dependencies between drone strikes and terrorist attacks observed in Pakistan. Based on parametric and validation tests, the model simulates a terrorist attack curve which approximates the rate and magnitude observed in Pakistan from 2007 through 2018. If this sounds of interest, below we provide the abstract to the paper, along with some images of model graphical user interface, the model logic and some of the results. The model itself was created in NetLogo and is available at: https://bit.ly/3qLJynv  (along with the data and detailed ODD of the model). At the bottom of the page you can find the full citation and a link to the paper.

Abstract. Drone strikes have been ongoing and there is a debate about their benefits. One major question is what is their role with respect to radicalization. This paper presents a data-driven approach to explore the relationship between drone strikes in Pakistan and subsequent responses, often in the form of terrorist attacks carried out by those in the communities targeted by these counter-terrorism measures. Our analysis of news reports which dis-cussed drone strikes and radicalization suggests that government-sanctioned drone strikes in Pakistan appear to drive terrorist events with a distributed lag that can be determined analytically. We then utilize these news reports to inform and calibrate an agent-based model which is ground-ed in radicalization and opinion dynamics theory. In doing so, we were able to simulate terrorist attacks that approximated the rate and magnitude ob-served in Pakistan from 2007 through 2018. We argue that this research effort advances the field of radicalization and lays the foundation for further work in the area of data-driven modeling and kinetic actions.

Keywords: Radicalization, Data-driven modeling, Drone strikes, Terrorism, Pakistan, Agent-based modeling.

Radicalization model’s graphical user interface.

The agent-based model flow diagram.

Terrorist attacks simulated by radicalization model qualitatively agree with real-world system.

Full Reference:
Shapiro, B. and Crooks, A.T. (2021), Kinetic Action and Radicalization: A Case Study of Pakistan, in Thomson, R., Hussain, M.N., Dancy, C.L. and Pyke, A. (eds), Proceedings of 2021 International Conference on Social Computing, Behavioral-Cultural Modeling & Prediction and Behavior Representation in Modeling and Simulation, Washington DC., pp 321-330. (pdf)

Wednesday, June 30, 2021

Towards Large-Scale Agent-Based Geospatial Simulation

Running large scale spatial agent-based models is often a computational challenge. To address this challenge, at the upcoming International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation (or SBP-BRiMS for short), Umar Manzoor, Hamdi KavakJoon-Seok Kim, Dieter Pfoser, Andreas Zufle, and Carola Wenk and myself have a paper entitled "Towards Large-Scale Agent-Based Geospatial Simulation."

In the paper we propose a scalable and general agent-based modeling and simulation framework for geospatial simulations involving networks. Specifically we propose to a solution for the parallelization of the single-threaded GeoMASON tookit by employing the Java Agent Development Environment (JADE) for the communication between threads (essentially, we divide the space of our agents into partitions, each handled by a separate thread of execution). We evaluate the proposed framework an simple urban model (created in MASON), which simulates simple patterns of life within an urban setting (click here for the blog post). The model has spatial network for agent movement and social network for maintaining social links. We compared the performance of the proposed framework on different settings, and concluded from experimentation that the proposed framework is outperformed by GeoMason when the agent population is small whereas with an increasing agent population, our proposed framework outperforms GeoMason as the complexity and time taken in simulation step increases substantially. If this sounds of interest, below we provide the abstract to the paper, along with some images of the framework and and simulation architecture. At the bottom of the page you can find the full citation and a link to the paper.

Abstract. Agent-based geospatial simulations have become very popular and widely used in examining the social and cultural characteristics of populations. Well-known toolkits such as NetLogo or MASON generally have scalability limitations, especially when the model and underlying spatial infrastructure become complex. This paper presents a framework for simulating large-scale agent-based geospatial systems by integrating the multi-agent systems toolkit JADE with the MASON agent-based modeling framework and its GIS extension, GeoMASON. The proposed Java-based framework can simulate large areas with hundreds of thousands of agents. It allows for the studying the evolution of a population and its environment over time. Such a framework provides the essential first steps for scalable model execution without sacrificing the model generality. 

Keywords: Large-scale geospatial simulation, Agent-based Modeling, MASON, Jade, GIS.

System Architecture of Proposed Framework.
Agent transfer between Zones.
Simulation using Proposed Architecture.

Full Reference:

Manzoor, U., Kavak, H., Kim, J-S., Crooks, A.T., Pfoser, D., Zufle, A. and Wenk, C. (2021), Towards Large-Scale Agent-Based Geospatial Simulation, 2021 International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation, Washington DC. (pdf)

Tuesday, May 25, 2021

Achieving Situational Awareness with Geolocated Social Media

Tuning back to our work on geosocial analysis we (Xiaoyi Yuan, Ron Mahabir, Arie Croitoru and myself) recently had a paper published in GeoJournal entitled "Achieving Situational Awareness of Drug Cartels with Geolocated Social Media." 
The overarching objective of this paper is to develop an approach that would enable the extraction of potentially relevant situational awareness-related information from geolocated raw data streams (in this example we use Twitter). We accomplish this goal by focusing on Named Entities (NEs) related to drug cartels rather than the raw text as a whole. Specifically, our analysis is performed on the NEs by first extracting them and then clustering them to identify relevant concepts/themes (using TextRazor). This approach gives rise to themes that can then be assessed for temporal and spatial patterns based on frequency in order to gain underlying insights into drug cartels. If is of interest to you below we provide the abstract to the paper, a diagram of our workflow and a sample of our results along with the link to the paper. Also the complete code for the analysis and results is available at https://bitbucket.org/xiaoyiyuan/cartel.

Abstract: Using geolocated tweets to achieve situational awareness is an often researched topic in disaster and emergency management. However, little has been done in the area of drug cartels, which, as transnational crime organizations, continue to pose great risk to the stability and safety of our communities. This paper made an initial effort in using geolocated social media (specifically Twitter) to achieve situational awareness of drug cartels through temporal and spatial analysis of derived named entity clusters. The results show that detecting peaks in the time series of frequently occurring entity clusters enabled the tracking of important events in public discourse surrounding drug cartels. Correlations between time series also provided valuable insights into the synchronicity between different events. Further examining the spatial distribution of key events for different countries, we identify thematic hotpots of public discourse on cartel activity. Our methodology also addresses issues of language ambiguity when working with noisy social media data in order to achieve situational awareness on drug cartels.

Keywords: Cartels, Social Media, Situational Awareness \and Temporal and Spatial Analysis.

The workflow of achieving situational awareness of drug cartels using geolocated tweets.

Tweet and entity counts by language and geolocation.

An example of tweets of high frequency on peak day in Venezuela

Heat maps of frequencies of a Cluster for Day 14 and Days 18-21.

Full Reference:
Yuan, X., Mahabir, R., Crooks, A.T. and Croitoru, A. (2021), Achieving Situational Awareness of Drug Cartels with Geolocated Social Media, GeoJournal. DOI: https://doi.org/10.1007/s10708-021-10433-2 (pdf)

Friday, April 09, 2021

Agent-Based Modeling and the City

Turning our attention back to agent-based modeling, in the recently open access edited volume by Wenzhong Shi, Michael Goodchild, Michael Batty, Mei-Po Kwan and Anshu Zhang entitled Urban Informatics, Alison Heppenstall, Nick Malleson, Ed Manley and myself have a chapter entitled "Agent-Based Modeling and the City: A Gallery of Applications.

In the chapter we discuss cities through the lens of complex systems comprised of composed of people, places, flows, and activities. Moreover, we make the argument that as cities contain large numbers of discrete actors interacting within space and with other systems from nature, predicting what might happen in the future is a challenge. We base this argument on the fact that human behavior cannot be understood or predicted in the same way as in the physical sciences such as physics or chemistry. The actions and interactions of the inhabitants of a city, for example, cannot be easily described in a physical science theory such as that of Newton’s Laws of Motion. This notion is captured quite aptly by a quote by Nobel laureate Murray Gell-Mann: “Think how hard physics would be if particles could think.” Building on these arguments we introduce readers to agent-based modeling as it offers a way to explore the processes that lead to patterns we see in cities from the bottom up but also allows us to incorporate ideas from complex systems (e.g., feedbacks, path dependency, emergence) along with providing a gallery of applications of geographically explicit agent-based models. 

We then discuss how agent-based models can incorporate various decision-making processes within them and  how we can integrate data within such models with a specific emphasis on geographical and social information. This leads us to a discussion on how agent-based modelers are utilizing machine learning (such as genetic algorithms, artificial neural networks, Bayesian classifiers, decision trees, reinforcement learning, to name but a few) and data mining (i.e. finding patterns in the data) within their models: from the design of the model, the execution of the model to that of the evaluation of the model. Finally,  we conclude the chapter with a summary and discuss new opportunities with respect to agent-based modeling and the city. One such opportunity is dynamic data assimilation which could be transformative for the ways that some systems, for example “smart” cities, are modeled. Our argument is that agent-based models are often used to simulate the behavior of complex systems, these systems often diverge rapidly from initial starting conditions. One way to prevent a simulation from diverging from reality would be to occasionally incorporate more up-to-date data and adjust the model accordingly (i.e., data assimilation). Data, especially streaming data produced through near real time observational datasets (e.g., social media, vehicle routing counters) could be utilized in such a case. If what we have written above is of interest, below we provide the abstract to chapter along with some figures which we use to illustrate some key points or concepts (such as dynamic data assimilation). Finally at the bottom post, we provide the full reference and a link to the chapter.

Agent-based modeling is a powerful simulation technique that allows one to build artificial worlds and populate these worlds with individual agents. Each agent or actor has unique behaviors and rules which governs their interactions with each other and their environment. It is through these interactions that more macro phenomena emerge: for example, how individual pedestrians lead to the emergence of crowds. Over the last two decades, with the growth of computational power and data, agent-based models have evolved into one of the main modeling paradigms for urban modeling and for understanding the various processes which shape our cities. Agent-based models have been developed to explore a vast range of urban phenomena from that of micro-movement of pedestrians over seconds to that of urban growth over decades and many other issues in between. In this chapter we will introduce readers to agent-based modeling from simple abstract applications to those representing space utilizing geographical data not only for the creation of the artificial worlds but also for the validation and calibration of such models through a series of example applications. We will then discuss how big data, data mining, and machine learning techniques are advancing the field of agent-based modeling and demonstrate how such data and techniques can be leveraged into these models, giving us a new way to explore cities.

Key Words: Agent-based Modeling, Geographical Information Systems, Machine Learning, Urban Simulation.
Using geographical information as a foundation for artificial worlds.
A selection of GeoMason models across various spatial and temporal scales.
Dynamic data assimilation and agent-based modeling.

Full Reference:
Crooks, A.T., Heppenstall, A., Malleson, N. and Manley, E. (2021), Agent-Based Modeling and the City: A Gallery of Applications, in Shi, W., Goodchild, M., Batty, M., Kwan, M.-P., Zhang, A. (eds.), Urban Informatics, Springer, New York, NY, pp. 885-910. (pdf)


Wednesday, March 31, 2021

A Busy Day: A Talk and NetLogo Tutorial

It is not often that I get to give a talk in one country and tutorial in another country, but thanks to COVID and the internet, that was today. First up I was invited to give a talk to the GIScience Research Group (GIScRG) at the Royal Geographical Society with IB, in the UK. The talk was entitled "Analyzing and Modeling Urban Environments Utilizing Computational Social Science: Opportunities, Examples and Challenge" which covered many of the topics that have been blogged here over the last few year. Below is the abstract to the talk and if this peaks your interest, the talk was recorded and is embedded here.

Abstract: The beginning of this century marked a milestone in human history. For the first time, more than half of the world’s population lived in urban areas. This trend is expected to continue into the foreseeable future with 6.7 billion people projected to live in cities by 2050. This rapid urbanization will place unprecedented pressures on urban systems and their ability to provide basic of services. To plan for this future, we need to better understand the inherent complexity of urban systems from social, economic and environmental perspectives. In this talk, I will explore how such understanding can be gained through the lens of computational social science (CSS): the interdisciplinary science of complex social systems and their investigation through computational modeling (e.g. agent-based models) and related techniques. Through a series of example applications, I will demonstrate how new forms of geographical data (e.g. crowdsourced, social media etc.) not only provide us with a novel way of analyzing urban environments but how such data can be integrated into geographically explicit agent-based models. In addition, I will highlight that by focusing on individual, or groups of individuals, leads to more aggregate patterns emerging and show how model outcomes can be validated by such datasets. After these demonstrations, I will outline the challenges associated with this program of research, as using such data is not without its difficulties. Together, this work provides a brief overview of the current state of analyzing and modeling urban environments through the lens of CSS. 

I would like to thanks those who joined this webinar, especially those who asked questions. On a side note, the RGS-IBG GIScience Research Group YouTube Chanel also has a great number of talks relating to GIScience and Geographic Data Science which are well worth watching. 

Later in the day, Sara Metcalf and myself were invited to give a tutorial entitled "Introduction to Agent Based Models" as part of the University at Buffalo's Computational and Data-enabled Science and Engineering (CDSE) day. In this tutorial we introduced agent-based modeling, discussed a variety of applications and ran through a tutorial, that of creating the Schelling Segregation model in NetLogo.

Abstract: This session will introduce the method of agent-based modeling, give a tutorial, and discuss a range of applications. Agent-based models facilitate dynamic simulation of multi-scalar feedback mechanisms and interactions between heterogeneous individual agents and their environments. Agents may represent people, animals, organizations, or other kinds of discrete decision-making entities. Participants who wish to practice developing the agent-based models demonstrated in this session should install the free NetLogo software

For those who are interested, the tutorial as a PDF can be found at https://tinyurl.com/CDSEnetlogo and you can follow along by watching the movie below.

Tuesday, February 23, 2021

Simulating Urban Shrinkage in Detroit via Agent-Based Modeling

While we are witnessing a growth in the world-wide urban population, not all cities are growing equally and some are actually shrinking (e.g., Leipzig in Germany; Urumqi in China; and Detroit in the United States). Such shrinking cities pose a significant challenge to urban sustainability from the urban planning, development and management point of view due to declining populations and changes in land use. To explore such a phenomena from the bottom up, Na (Richard) Jiang, Wenjing Wang, Yichun Xie and myself have a new paper entitled "Simulating Urban Shrinkage in Detroit via Agent-Based Modeling" published in Sustainability

This paper builds on our initial efforts in this area which was presented in a previous post. In that post we showed how a stylized model could not only simulate housing transactions but the aggregate market conditions relating to urban shrinkage (i.e., the contraction of housing markets). In this new paper, we significantly extend our previous work by: 1) enlarging the study area; 2) introducing another type of agent, specially, a bank type agent; 3) enhancing the trade functions by incorporating agents preferences when it comes to buying a house; 4) adding additional household dynamics, such as employment status change. These changes will are discussed extensively in the methodology section of the paper.

If this is of interest to you, below we provide the abstract of the paper along with some figures of the study area, graphical user interface, model logic and results. At the bottom of the post you can see the full reference to the paper along with a link to it. The model itself was created in NetLogo and a similar to our other works, we have a more detailed description of the model following the Overview, Design concepts, and Details (ODD) protocol along with the source code and data needed to run the model at: http://bit.ly/ExploreUrbanShrinkage.


While the world’s total urban population continues to grow, not all cities are witnessing such growth, some are actually shrinking. This shrinkage causes several problems to emerge, including population loss, economic depression, vacant properties and the contraction of housing markets. Such issues challenge efforts to make cities sustainable. While there is a growing body of work on studying shrinking cities, few explore such a phenomenon from the bottom-up using dynamic computational models. To fill this gap, this paper presents a spatially explicit agent-based model stylized on the Detroit Tri-County area, an area witnessing shrinkage. Specifically, the model demonstrates how the buying and selling of houses can lead to urban shrinkage through a bottom-up approach. The results of the model indicate that along with the lower level housing transactions being captured, the aggregated level market conditions relating to urban shrinkage are also denoted (i.e., the contraction of housing markets). As such, the paper demonstrates the potential of simulation to explore urban shrinkage and potentially offers a means to test policies to achieve urban sustainability.

Keywords: Agent-based modeling; housing markets; Urban Shrinkage; cities; Detroit; GIS

Study Area. 

Model graphical user interface, including input parameters, monitors (left) and the study area (middle) and charts recording key model properties.

Unified modeling language (UML) Diagram of the Model.

Household Decision-Making Process for Stay or Leave Current Location.

Heat Maps of Median (A) and Average (B) House Prices at the End of the Simulation where Demand equals Supply.

Full Reference: 

Jiang, N., Crooks, A.T., Wang, W. and Xie, Y. (2021), Simulating Urban Shrinkage in Detroit via Agent-Based Modeling, Sustainability, 13, 2283. Available at https://doi.org/10.3390/su13042283. (pdf)


Thursday, January 28, 2021

Call for Papers: Humans, Societies and Artificial Agents (HSAA)


As part of the Annual Modeling & Simulation Conference (ANNSIM 2021), Philippe Giabbanelli, and myself are organizing a tract entitled "Humans, Societies and Artificial Agents (HSAA)" which now has a call for papers out. 

Track description: Artificial societies have typically relied on agent-based models, Geographical Information Systems (GIS), or cellular automata to capture the decision-making processes of individuals in relation to places and/or social interactions. This has supported a wide range of applications (e.g., in archaeology, economics, geography, psychology, political science, or health) and research tasks (e.g., what-if scenarios or predictive models, models to guide data collection). Several opportunities have recently emerged that augment the capacity of artificial societies at capturing complex human and social behavior. Mixed-methods and hybrid approaches now enable the use of ‘big data’, for instance by combining machine learning with artificial societies to explore the model’s output (i.e., artificial societies as input to machine learning), define the model structure (i.e. machine learning as a preliminary to designing artificial societies), or run a model efficiently (i.e. machine learning as a proxy or surrogate to artificial societies). Datasets are also broader in type since artificial societies can now be built from text or generate textual as well as visual outputs to better engage end-users. 

Authors are encouraged to submit papers in the following areas: 

  • Applications of artificial societies (e.g., modeling group decisions and collective behaviors, emergence of social structures and norms, dynamics of social networks). 
  • Data collection for artificial societies (e.g., using simulations to identify data gaps, population simulations with multiple data sources, use of the Internet-of-Things). 
  • Design and implementation of artificial agents and societies (e.g., case studies, analyses of moral and ethical considerations). 
  • Participatory modeling and simulation. 
  • Policy development and evaluation through simulations. 
  • Predictive models of social behavior. 
  • Simulations of societies as public educational tools.
  • Mixed-methods (e.g., analyzing or generating text data with artificial societies, combining machine learning and artificial societies). 
  • Models of individual decision-making, mobility patterns, or socio-environmental interactions. 
  • Testbeds and environments to facilitate artificial society development. 
  • Tools and methods (e.g., agent-based models, case-based modeling, soft systems).

Key dates:

  • Papers due: March 1, March 22nd 2021. 
    • Accepted papers will be published in the conference proceedings and archived in ACM Digital Library and IEEE Explore. 
  • Conference (hybrid format), July 19 – 22, 2021.

Further information including paper guidelines can be found at: https://scs.org/annsim/

Wednesday, January 06, 2021

Elections and Bots

Continuing our work on botsRoss Schuchard and myself have a new paper in PLOS ONE entitled "Insights into elections: An ensemble bot detection coverage framework applied to the 2018 U.S. midterm elections." Our motivation for the work came from the fact that during elections internet-based technological platforms (e.g., online social networks (OSNs), online political blogs etc.) are gaining more power compared to mainstream media sources (e.g., print, television and radio). While such technologies are reducing the barrier for individuals to actively participate in political dialogue, the relatively unsupervised nature of OSNs increases susceptibility to misinformation campaigns, especially with respect to political and election dialogue. This is especially the case for social bots—automated software agents designed to mimic or impersonate humans which are prevalent actors in OSN platforms and have proven to amplify misinformation.  

The issue however is that no single detection algorithm is able to account for the myriad of social bots operating in OSNs. To overcome this issue, this research incorporates multiple social bot detection services to determine the prevalence and relative importance of social bots within an OSN conversation of tweets. Through the lens of the 2018 U.S. midterm elections, 43.5 million tweets were harvested capturing the election conversation which were then analyzed for evidence of bots using three bot detection platform services: Botometer, DeBot and Bot-hunter.

We found that bot and human accounts contributed temporally to our tweet election corpus at relatively similar cumulative rates. The multi-detection platform comparative analysis of intra-group and cross-group interactions showed that bots detected by DeBot and Bot-hunter persistently engaged humans at rates much higher than bots detected by Botometer. Furthermore, while bots accounted for less than 8% of all unique accounts in the election conversation retweet network, bots accounted for more than 20% of the top-100 and top-25 ranking out-degree centrality, thus suggesting persistent activity to engage with human accounts. Finally, the bot coverage overlap analysis shows that minimal overlap existed among the bots detected by the three bot detection platforms, with only eight total bot accounts detected by all (out of a total of 254,492 unique bots in the overall tweet corpus ).

If this research sounds interesting to you, below we provide the abstract to the paper along with some figures outlining our methodology and some of the results. While at the bottom of the post you can see the full reference and there is a link to the paper were you can read more.


The participation of automated software agents known as social bots within online social network (OSN) engagements continues to grow at an immense pace. Choruses of concern speculate as to the impact social bots have within online communications as evidence shows that an increasing number of individuals are turning to OSNs as a primary source for information. This automated interaction proliferation within OSNs has led to the emergence of social bot detection efforts to better understand the extent and behavior of social bots. While rapidly evolving and continually improving, current social bot detection efforts are quite varied in their design and performance characteristics. Therefore, social bot research efforts that rely upon only a single bot detection source will produce very limited results. Our study expands beyond the limitation of current social bot detection research by introducing an ensemble bot detection coverage framework that harnesses the power of multiple detection sources to detect a wider variety of bots within a given OSN corpus of Twitter data. To test this framework, we focused on identifying social bot activity within OSN interactions taking place on Twitter related to the 2018 U.S. Midterm Election by using three available bot detection sources. This approach clearly showed that minimal overlap existed between the bot accounts detected within the same tweet corpus. Our findings suggest that social bot research efforts must incorporate multiple detection sources to account for the variety of social bots operating in OSNs, while incorporating improved or new detection methods to keep pace with the constant evolution of bot complexity.


Fig 1. Social bot analysis framework employing multiple bot detection platforms. The framework enables the application of ensemble analysis methods to determine the prevalence and relative importance of social bots within Twitter conversations discussing the 2018 U.S. midterm elections.

Fig 3. Cumulative tweet contribution rates for the 2018 U.S. midterm OSN conversation (October 10 – November 6, 2018) from the (a) human (blue) / bot (red) and (b) DeBot (green) / Botometer (pink) / Bot-hunter (orange) account classification perspectives.

Fig 4. Intra-group and cross-group retweet communication patterns of human (blue) and social bot (red) users within the 2018 U.S. midterm election Twitter conversation according to each bot detection classification platform: (a) Combined Bot Sources (b) DeBot (c) Botometer (d) Bot-hunter. The combined bot sources results (shown in gray) classified an account as a bot in aggregate fashion if any of the three detection platforms classified the account as a bot.

Fig 5. Social bot account evidence within the top-N (where, N = 1000 / 500 / 100 / 25) centrality rankings [(a) eigenvector (b) in-degree (c) out-degree (d) PageRank] according to bot classification results from Bot-hunter (orange), Botometer (pink) and DeBot (green).
Fig 7. Bot detection coverage analysis for bots detected within the 2018 U.S. midterm election Twitter conversation using the Botometer, Bot-hunter and DeBot bot detection platforms.


Full reference:

Schuchard, R.J. and Crooks, A.T. (2021), Insights into Elections: An Ensemble Bot Detection Coverage Framework Applied to the 2018 U.S. Midterm Elections, PLoS ONE, 16(1): e0244309. Available at  https://doi.org/10.1371/journal.pone.0244309. (pdf).