Search Process as Transitions Between Neural States

Search is one of the most performed activities on the World Wide Web. Various conceptual models postulate that the search process can be broken down into distinct emotional and cognitive states of searchers while they engage in a search process. These models significantly contribute to our understanding of the search process. However, they are typically based on self-report measures, such as surveys, questionnaire, etc. and therefore, only indirectly monitor the brain activity that supports such a process. With this work, we take one step further and directly measure the brain activity involved in a search process. To do so, we break down a search process into five time periods: a realisation of Information Need, Query Formulation, Query Submission, Relevance Judgment and Satisfaction Judgment. We then investigate the brain activity between these time periods. Using functional Magnetic Resonance Imaging (fMRI), we monitored the brain activity of twenty-four participants during a search process that involved answering questions carefully selected from the TREC-8 and TREC 2001 Q/A Tracks. This novel analysis that focuses on transitions rather than states reveals the contrasting brain activity between time periods - which enables the identification of the distinct parts of the search process as the user moves through them. This work, therefore, provides an important first step in representing the search process based on the transitions between neural states. Discovering more precisely how brain activity relates to different parts of the search process will enable the development of brain-computer interactions that better support search and search interactions, which we believe our study and conclusions advance.


INTRODUCTION
On the Web, search is ubiquitous and has come to form a core and fundamental part of everyday human activity [44]. Research activity over several decades in the information retrieval (IR) community and other related scientific communities has focused on ways to better understand the search process, resulting in seminal theories, models, and findings that have shaped the foundations of current search engine systems. A few examples of such prominent works are Wilson's Information Seeking Behaviour model [45], Kuhlthau's Information Seeking Process (ISP) model [27], Cole's Theory of Information Need [8], Ingwersen's Cognitive IR Theory [22], and Belkin's Anomalous States of Knowledge (ASK) model [2]. One common aspect of all these influential contributions is that they have investigated the phenomenon of search via some mediator and are thus indirect. These approaches are mainly based on behavioural studies of searchers while they engaged in an information retrieval and seeking process, and have often relied upon questionnaires and interviews [27] or by measuring searchers' behavioural data while interacting with IR systems, e.g. via their submitted queries and their reformulation [25] or via their interaction with retrieved results [43] or by measuring their affective and physiological data during such interactions [30].
A common property of works mentioned above has been a divide and conquer strategy that subdivides the search process into subprocesses where each subprocess can be analysed and evaluated. A central aspect of these models is how the search process is subdivided, an issue which has been given different answers according to different approaches [44]. For example, stratified models view search as composed of a set of strata relating the searcher to the system and a searcher's behaviour is determined by interaction at each level of the stratum [40], while process models are generally multi-stage representations of the activities of a searcher during a search process [3,4,26,28,31,36]. Other examples include cognitive models that rely upon representations of the cognitive processes associated with search activity [9,22]. In this paper, we follow the direction of cognitive approaches found in the literature that subdivide search into different cognitive components. However, an essential difference is that our model is based on the brain activity of searchers as they proceed through a search process. This difference allows us to avoid the use of indirect measures based on behaviour to inform the model, which could affect the state of the process itself. To facilitate this examination of search, we rely upon a view of the brain as a set of large-scale networks that subserve different cognitive functions [37]. This is a novel way to describe the search process and allows us for the first time to address two important questions: Track: Web Search and Mining WWW 2018, April 23-27, 2018, Lyon, France • RQ1: "Are there clear, detectable physical manifestations (i.e. neural correlates) of brain activity that identify transitions between different time periods of the search process?"; and • RQ2: "What is the nature of these physical manifestations and how do they functionally combine across different brain areas?".
Answers to these questions will certainly improve our understanding of the search process and will help guide development of more robust definitions of search. They will also open fundamentally new venues for the design and implementation of novel IR techniques to enable better (and even proactive) search. This paper describes our efforts in this direction.
In this paper, we focus on discovering and mapping activity of large-scale functional brain networks of users as they perform search based on a Question Answering (Q/A) retrieval task. Our central aim is to identify how functional brain activity changes during the search task as revealed by differences found between different time periods of search. These time periods include realising an information need, formulating a query, stating a query, determining the relevance of a document returned in the search and determination of satisfaction. The differences between time periods will reveal the dynamic allocation of neural resources to different functions such as attention, task maintenance, self-referential thinking and emotional engagement. Research Hypothesis. Our experiment uses fMRI to measure Blood Oxygenation Level Dependent (BOLD) signals in twentyfour participants engaged in a Q/A search task for a predefined set of questions with respect to a predefined set of relevant and non-relevant documents. Collection of data was achieved using a 3T MRI scanner and utilised a lab-based user study where the data were collected and analysed off-line. Our results rely on the fact that BOLD signals can be analysed to detect significant differences in brain activity across a search task. In particular, we aim to identify how these differences are reflected in changes in large-scale functional brain networks that are thought to broadly subserve particular cognitive functions. We hypothesise that there are brain regions in which the BOLD signals would be different between different time periods and that consideration of these regions in the context of large-scale functional networks would inform our deeper understanding of the basic processes involved in search.

NEUROSCIENCE AND IR
The past decade has seen a keen interest of how neuroscience could inform our understanding of IR [20] and this is reflected in the growing body of literature that relates concepts in IR to neuropsychological processes. This research has employed a variety of brain imaging techniques and has ranged from exploring IR concepts such as relevance and information need to taking steps to demonstrate how these advances in understanding can be exploited. A study by Moshfeghi et al. [32] used functional Magnetic Resonance Imaging (fMRI) to identify brain regions activated by the process of judging relevance of an image. These brain regions include the inferior parietal lobe, inferior temporal gyrus and superior frontal gyrus and their increased activation for relevant items were related to visuospatial working memory. Subsequent fMRI work exploring the neural basis of information need [33] revealed differences in brain activity between when a searcher realised an information need and engaged in a search process and when a searcher realised an information need and simply reported their information need without engaging in a search process. For the case when a searcher performed a search there was evidence of brain activity consistent with a switch from a narrow internal focus to a broad external focus of attention. However, these fMRI studies were focused on particular times during a search and did not extensively study brain activity throughout an entire search process.
Another line of research exploring neuropsychological processes related to IR has employed electroencephalography (EEG). For example, a study conducted by Eugster et al. [12] used EEG to show that the frequency content of the EEG signal as well as Event-Related Potentials (ERPs) can be used to define a set of features that enable decoding whether a text is relevant. This work was later extended to demonstrate how the relevance of a written word could be identified from the EEG activity appearing 200-950 ms following it being read [13]. Consistent with these works, Allegretti et al. [1] used EEG to show that within 500 ms EEG signals begin to appear that differentiate between viewing a relevant and a non-relevant image. Similarly, Kauppi et al. [24] used magnetoencephalography (MEG) to show that the frequency content of the MEG signal, along with eye movement data can be used to decode whether viewed images were relevant. The combination of eye movement data and EEG signal was further explored in the domain of successfully identifying the relevance of reading text [19]. While techniques such as EEG and MEG have a higher temporal resolution, fMRI provides higher resolution in localising where in the brain activity is occurring and therefore provides greater spatial precision. This is particularly important for studies where the brain regions that get activated are unknown.
These previous neuroimaging studies have been successful at finding neural correlates of relevance and information need and show promise that these neural correlates may be used in specific instances such as using brain data to drive automatic recognition of relevance. However, the dynamic process of how brain activity evolves over a search process itself remains unstudied. In this paper, we take an important step towards revealing how brain activity reflects transitions between different states of the search process.

EXPERIMENTAL METHODOLOGY
Several experimental design and data analysis considerations are relevant to the use of fMRI data [10,32], and several factors were important in guiding our research plans. Firstly, the fMRI scanning environment is restrictive, and a participant must lay supine, keeping their head still, and only limited response/interactive devices can be used without causing signal or safety issues. This constraint led to the choice of our task, and we chose to use multiple choice questions since it was possible to provide response using an MRIcompatible button box. To allow users the ability to submit a query we used a noise cancelling microphone attached to the head coil. An additional factor to take into consideration is that although fMRI provides localisation accuracy to within millimetres, the temporal resolution of fMRI (the time to take a single measurement of the entire brain) is around 2 seconds. This relatively slow rate of data acquisition is compounded by the fact that the Blood Oxygenation Level Dependent (BOLD) signal measured is related to the underlying neural signal in a complex way that introduces further  [16]. Due to these constraints on the timing of the data acquisition, it was necessary to time experimental events at a rate that could be resolved by our fMRI measurements.

Design
A "within-subjects" design [7] was used in this study. We organised the search process into five epochs 1 : (1) Information Need (IN), (2) Query Formulation (QF), (3) Query Submission (QS), (4) Relevance Judgment (RJ), and (5) Satisfaction Judgment (SJ). As our independent variable, we then examined the four transitions between each epoch and the next. The dependent variable was brain activity revealed by the BOLD signal. An important aspect of this design is that if a participant responded with an answer to the initial query then they did not have an information need and their data was not considered for this trial. Otherwise, the subsequent search process was entered into the analysis.

Task
Each participant engaged in the task illustrated in Figure 1 where they were first presented with a fixation screen from 4-6 seconds before proceeding to a question for 4 seconds. After this, for 4 seconds four possible responses were provided while the question stayed on the screen. Participants could not make a response until after the 4 seconds of observing the possible responses. This was done so that brain activity related to the motor response of pressing the button would not be contained in the model of brain activity, which only considered these first 8 seconds. Of the four possible responses, one was always the correct answer, and one of them was always "need to search". The position of the four alternatives was randomised for each trial and the response given by pressing one of the four buttons available on the button box that each participant had in their right hand. The time to respond was not restricted so that participants were not under time pressure to respond. The order of the questions was randomised for each participant. After the 8 seconds participants were able to respond and if they responded with an answer, the experiment progressed directly on to the next trial. For the case that they answered the question, the trial was not considered for analysis. However, if the answer "need to search" was provided then this epoch was labelled as IN and participants moved to the next state of the search process. In this next state of query formulation (QF) participants had 4 seconds to consider what their query was before entering the next state where they were given 4 seconds to submit their query (QS). Following this, participants viewed a fixation cross on the screen before being presented with a document to evaluate that was the result of their search. They viewed this document for 16 seconds (RJ) before returning to the original question and multiple choices. When they responded with an answer to the question (as opposed to the "need to search" option) they were presented with a question of whether they were satisfied with the search (SJ) and were given 4 seconds to make this satisfaction judgment. If after seeing the document they still responded with "need to search" they went through another cycle of being presented with another document in response to their query. For any question, there was a limit of 3 attempts. 1 Using a common convention in neuroimaging we will term these conditions that occur over distinct time periods as epochs. [17,23]

Question Answering Dataset
To perform the task scenario mentioned in Section 3.2, we created a Question Answering dataset 2 . To develop this standard set of questions, we used previous runs of TREC Q/A Track, in particular we carefully selected a set of 40 questions from the TREC-8 and TREC-2001 Question Answering Tracks -Main Task 3 . We chose these two Tracks since they were the first and last tracks where the questions presented were (i) independent from one another, in contrast to other Tracks that share a relationship, and (ii) they also provided the correct answer to the questions. We then manually examined all the questions presented in these two tracks and selected a subset of questions that (i) were not longer than one line, and (ii) the correct answer to the question was not longer than five words. This constraint is due to the limitation of presenting the questions and options to the participants in an fMRI settings. An additional constraint was that there were at least two relevant and non-relevant answers in their QRel. We then removed the questions that were ambiguous or were time dependent, e.g. Who is the president of Stanford University? (TREC-8, Topic 51), making the answers provided in the Track not appropriate. The answers of all these questions were then checked by current search engines to make sure that the answers are still valid and correct. We also created two wrong answers for each question that were in the domain of the question, e.g. "What is a supernova?" (TREC-2001, Topic 1067) the correct answer is "An exploding star", and we created two other wrong answers, i.e. "A newborn star" and "A dead star". We also made sure that the questions covered a wide range of topics, e.g. history, politics, science, etc. to reduce any bias that might occur from an emphasis of a particular type of question.
Over this set of questions, two annotators separately judged the difficulty of the questions (i.e. hard or easy) and then chose a subset of questions where both annotators agreed upon question difficulty, i.e. 20 of them were hard, and 20 were easy questions. Since the experiment was divided into two runs, additional care was made to further divide the questions across runs so that each run had ten easy and ten hard questions covering a variety of topics. The goal of this procedure was to control the set of questions such that on average there was an equal chance of replying with a known answer and needing to search since the answer was unknown.
An additional step included preparing the documents that were shown to the subjects once they engaged in a search process. This involved simulating a snippet answer that is returned by current search engines such as Google when a question is submitted. To achieve this, we selected two relevant and two non-relevant documents from QRel. The length of the answers provided in TREC-8 and TREC 2001 were incompatible. In order to keep the size of the results consistent, for those answers that were too short, we found the source file and selected sentences around the answer so that all snippets had the same length. The average length of the answers shown to the participants for the first and second run was 39.47 words (SD of 3.33) and 39.65 words (SD of 3.285) respectively. This was done to reduce any potential confounding effect of snippet size on the brain activity results.

Procedure
This section describes the flow of the study, from start to finish. Ethical permission was obtained from the Ethics Committee of the College of Science and Engineering, University of Glasgow. Participants were instructed of the duration of the experiment, which included approximately 50 minutes to perform all functional tasks examining search processes, and approximately 10 minutes to obtain an anatomical scan of brain structure. Participants were told that they could leave at any point during the experiment and would still receive payment (the payment rate was £6/hr). They were then asked to sign a consent form. Before beginning the experiment participants underwent a safety check to guarantee that they did not possess any metal items inside or outside of their body, or had any contraindications for scanning, such as certain tattoo inks. They were then provided with a gear (similar to a training suit) to wear for the duration of the experiment to avoid potential interference with the fMRI signal from any metal objects in their clothes.
Next, for training, they were given an example task and a corresponding set of example questions to familiarise themselves with the procedure. After successful completion of their training task, participants entered the fMRI scanner, and settings of the machine were adjusted to optimise their comfort and vision. While being scanned, each participant engaged in two separate runs of the task, with each run comprised of 20 questions. Two runs were chosen to give the participants a further break to relax during the scanning and to prevent fatigue on the task, which could extend in time if a participant often needed to search. After the functional runs were complete, the anatomical data of each participant were obtained.
After completion of scanning, participants filled out an exit questionnaire that provided demographic and qualitative descriptions of their experience during the experiment. They also completed the Edinburgh handedness questionnaire [34] that evaluates whether the participant was right-, left-or mixed-handed. Handedness information was obtained since lateralization of brain function is influenced by handedness, and we wished to ensure that our sample of participants approximated the general population. Apparatus. The text was presented using Presentation ® software 4 , and back-projected using an LCD projector onto a translucent screen so that participants could view them in an angled mirror while positioned in the MRI scanner. fMRI Data Acquisition. All fMRI data were collected at the Centre for Cognitive Neuroimaging, University of Glasgow, using a 3T Tim Trio Siemens scanner and 32-channel head coil. A functional 4 Presentation ® software (Neurobehavioral systems, Inc.), http://www.neurobs.com. T2*-weighted MRI run was acquired for two runs of the task (TR 2000ms; TE 30ms; 32 Slices; 3mm 3 voxel; FOV 210, matrix of 70×70). An anatomical scan was performed at the end of the scanning session that comprised a high-resolution T1-weighted anatomical scan using a 3D magnetisation prepared rapid acquisition gradient echo (ADNI-MPRAGE) T1-weighted sequence (192 slices; 1mm 3 voxel; Sagittal Slice; TR 1900ms; TE 2.52; 256 × 256 image resolution).
Questionnaires. When the experiment ended participants took an exit questionnaire that collected background and demographic information as well as general comments about the study. This included questions about previous experience with fMRI type user studies and questions to ascertain participants' subjective experience of performing the experiment. Pilot Studies. Prior to running the actual user study, we performed a pilot study using two participants to confirm that the procedure worked appropriately and to obtain general feedback. As a result of the feedback, we implemented a number of changes to the experimental paradigm. After the pilot, it was determined that the participants were able to complete the user study without problems and that the system correctly logged participants' interaction data.

RESULTS AND DISCUSSION
A study with the procedure explained in Section 3.4 was conducted over 15 days from 7 December, 2015 to 22 December, 2015. Participants consisted of 24 healthy individuals with 11 males and 13 females. All participants were under the age of 44, with the largest group between the ages of 18-23 (54.1%) followed by a group between the ages of 30-35 (20.8%). The handedness survey indicated that 79.1% were right-handed, 12.5% were left-handed, and 8.33% were mixed-handed. Participants tended to have a postgraduate degree (20.8%), bachelors (33.33%) or other qualifications (45.8%). They were primarily students (54.1%), though there were a number of individuals who were self-employed (20.8%), not employed (4.16%) or employed by a company or organisation (20.8%). Participants were primarily native speakers (79.1%) or had an advanced level of English (20.8%). They all had experience in searching, with an average of 11.66 years (SD of 3.58) experience. Log Analysis. The fMRI analysis relied upon a participant's response to the question to code whether a trial was IN or No-IN. If the trial was No-IN, then there was no further analysis of that trial since we were interested in the search process, starting from a realisation of information need. Because the experiment would be too difficult for a participant if they never knew an answer we balanced the number of easy and difficult questions so that there would be approximately an equal response rate for IN (respond "need to search") and fMRI Data Preprocessing. The fMRI data were preprocessed using Brain Voyager QX 5 . A standard pipeline of pre-processing of the data was performed for each participant [18]. This involved slice scan time correction using trilinear interpolation based on  In addition, linear trends in the data were removed and high pass filtering with a cutoff of 0.0025 Hz performed to reduce artefact from low frequency physiological noise. The functional data were then coregistered with the anatomic data and spatially normalised into the common Talairach space [42]. Finally, the functional data of each individual underwent spatial smoothing using a Gaussian kernel of 6mm to facilitate analysis of group data.

General Linear Model (GLM) Analysis
Analysis began with a first-level analysis of individual participants. This used multiple linear regression of the BOLD-response time course in every voxel that modelled the two epochs to be contrasted (IN to QF, QF to QS, QS to RJ, RJ to SJ). Predictors' time courses were adjusted for the hemodynamic response delay by convolution with a hemodynamic response function. Group data were statistically tested with a second-level analysis using a random effects analysis of variance using search epoch as a within-participants factor. Four contrasts were performed between brain activity at different time epochs of the process. These included contrasts of IN vs QF, QF vs QS, QS vs RJ and RJ vs SJ. To address the issue of multiple statistical comparisons across all voxels, activations are reported using False Discovery Rate (FDR) at a threshold of q < 0.01 [5]. Using FDR, we control for the number of false positive voxels among the subset of voxels labelled as significant.

Mapping results of GLM onto large scale functional brain networks
Examining activity at specific areas of the brain has been a particularly effective approach when studying sensory and motor processing where data from individuals with brain damage at specific locations supports the view that particular areas are crucial for particular functions. For example, loss of ability to move a certain limb following a stroke. However, when considering brain organisation more broadly, it is also necessary to think of areas connected through large-scale networks that relate to particular functional activities. Table 5 summarises established brain networks, including those related to the cognitive functions of attention, executive control and self-referential thought. The four transitions involved activation in many brain regions, and we wished to examine how these activations in individual brain regions might reflect patterns of activity in large-scale brain Track: Web Search and Mining WWW 2018, April 23-27, 2018, Lyon, France networks. To achieve this, we needed a means to assign the different clusters to distinct networks. Several different schemes have been proposed to parcellate the brain into different functional areas, and we adapted one such scheme based on resting-state functional connectivity [37]. This approach provides the three-dimensional coordinates of 264 brain locations in MNI coordinates and a network membership for each location. A total of ten networks are used, including dorsal attention, ventral attention, fronto-parietal task control, cingulo-opercular task control, salience, default-mode, auditory, visual, somatomotor-hand and somatomotor-mouth. To assign a particular location to a network we first took the coordinates of our peak activations and used the GingerAle software from BrainMap ® 6 to convert from Talairach to MNI (SPM) coordinates. Following this, we mapped our activations to the closest of the 264 brain location and assigned it the proposed function of that location. This provided us with an estimate of the brain function for each cluster and thus allowed us to view the changes between epochs as changes in the activity of large-scale networks. Main Results. The results can be seen in the evaluation of the four transitions between different epochs (IN to QF, QF to QS, QS to RJ and RJ to SJ) and to broadly address RQ1 are evaluated first in terms of the general areas where activations were found and with an emphasis on subcortical, sensory and motor processing. Following this, we report the differences regarding the changes in large-scale functional brain networks.
Transition 1 -IN to QF: During IN participants were visually presented with the question and then the possible answers, and they provided the response that they needed more information to answer the question. During QF participants followed visual instructions to formulate a query. Thus, while both tasks would require some reading we would expect IN to more robustly activate processes involving vision and eye movements as well as hand movements to make the response on the button box, while for QF we would expect participants to more robustly activate cognitive processes involved in formulating a query as well as preparatory motor activity for issuing the query. The results of the contrast IN versus QF are presented in Table 1 and Figure 2. Inspection of these results shows that of the 26 clusters of activity found, 12 clusters were more active during IN, while 14 were more active during QF. Consistent with the IN condition providing a higher demand of visual processing, areas such as BA18 and 37 were more active during IN. Consistent with ideas of preparatory activity being related to QF, it was found that activations were higher during QF than IN in auditory and somato-motor regions.
Transition 2 -QF to QS: During QF participants were following visually presented instructions to formulate a question while for QS they were following visually presented instructions to verbally state the question. Thus, both tasks would require similar amounts of vision and eye movements, but QF would require greater cognitive activity to formulate the question, while QS would engage speech production and motor areas related to speaking as well as sound processing. The results of the contrast QF versus QS are presented in Table 2 and Figure 3. In these results, we have not reported some activations that were found in the ventricles which contain only cerebrospinal fluid and no neurons. Thus these regions 6 GingerAle software (Brainmap ® ), http://www.brainmap.org.
are not related to neural activity and are known as a potential artefact when people speak while being scanned [14,38]. Inspection of these results shows six clusters of activity, with five clusters more active during QF, while one was more active during QS.
Transition 3 -QS to RJ: During QS participants were following visually presented instructions to state their query and during RJ they were presented with a document that would involve vision and eye movements to read and understand the visually presented document that was given in response to their query. Thus, we would expect motor and speech production areas to be more active during QS and vision, reading and cognitive areas to be more active during RJ. The results of the contrast QS versus RJ are presented in Table 3 and Figure 4. Inspection of these results shows that of the 17 clusters of activity found, nine clusters were more active during QS, while eight were more active during RJ. The cerebellar and subcortical activity during QS appears to reflect processing of movement and potentially also reward processing. For instance, activity in cerebellum can be related to motor production, and the subcortical regions found including the basal ganglia (substantia nigra, globus pallidus) and hypothalamus are not necessarily related to the movement since regions of the basal ganglia can be related to both movement and reward systems. Thus, these activities are consistent with the statement of the query to be seen as initiating reward-seeking behaviour. The activity during RJ can be attributed to language, visual and cognitive processing. For example, the lateralised left hemisphere activity in BA21 is consistent with language processing and the activity in BA18 reflects visual processing.
Transition 4 -RJ to SJ: During RJ participants are using vision and eye movements to read the presented document and are considering whether the information presented in the document will provide them with an answer to the question. During SJ participants are following visually presented instructions to decide whether they were satisfied with the information that had been provided to their query and pressing a response key as appropriate to answer yes or no. Thus, during RJ we would expect there to be greater activity in visual and eye movement areas, as well as cognitive areas related to reading and subsequent semantic encoding of the information. During SJ we would expect motor activity associated with the button press and processes of cognitive evaluation to decide if the information just presented had been relevant. The results of the contrast RJ versus SJ are presented in Table 4 and Figure 5. Inspection of these results shows that of the 22 clusters of activity found, 11 clusters were more active during RJ, while 11 were more active during SJ. A large cluster in BA18 was more active during RJ than SJ, and this is consistent with greater visual processing during the scanning of the document, however, for another higher level visual region in BA37 there was more activation in SJ than RJ, and the reason for this is unclear. However, the large cluster in BA40, associated with the motor activity is consistent with the presence of a key press in the SJ phase.

Large Scale Network Analysis
In this section, we address RQ2 by summarising the results in terms of the activation of large-scale functional brain networks (Table 5). We focus on six networks involved in complex processing (dorsal attention, ventral attention, fronto-parietal task control, cingulo-opercular task control, salience and default mode) as well as     This analysis provides a more holistic view of brain function than consideration of single areas in isolation.

Analysis of IN vs QF network:
In this contrast, four of the ten networks showed a difference between epochs, with no change in activity found for the dorsal attention or default mode network. Greater activity was found during the IN epoch in the cinguloopercular network and two of three clusters in the fronto-parietal task network. This is consistent with the view that IN would require both dynamic restructuring of cognitive resources across the subtasks and sustained effort to maintain task set within the subtasks. Larger activation was found in the ventral attention and salience networks during QF, suggesting that the change from IN to QF involved a switch of attention involving factors that were either emotionally or cognitively relevant to the individual.
Analysis of QF vs QS network: In this contrast, four of the ten networks showed a difference, with no change of activity found in the cingulo-opercular task control and salience networks. Greater    Table 5: Summary of brain networks and regions [37].

Network Brain regions Function dorsal attention
Frontal eye fields (FEF), intraparietal sulcus/superior parietal lobule (IPS/SPL) [11,35] Top-down attention, visuospatial ventral attention Temporoparietal junction (TPJ), ventral frontal cortex (VFC) [11,35] ) Bottom-up reorienting or shifting of attention fronto-parietal task control Dorsolateral prefrontal cortex (dlPFC), Inferior parietal lobule (IPL), dorsal frontal cortex (dFC), inferior parietal sulcus (IPS), precuneus, middle cingulate cortex (mCC) [11,35] Initiation and adjustment for rapid adaptive control cinguloopercular task control Anterior prefrontal cortex (aPFC), anterior insula/frontal operculum (AI/fO), dorsal anterior cingulate cortex/medial superior frontal cortex (dACC/msFC) [11,35] Stable set control for task maintenance salience Anterior Insula (AI), dorsal Anterior Cingulate Cortex (dACC), amygdala, ventral striatum, dorsomedial thalamus, hypothalamus, substantia nigra/ventral tegmental area [29,41] Identify relevant internal and external stimuli and the integration of sensory, emotional and cognitive information default-mode Ventral medial prefrontal cortex (vMPFC), posterior cingulate cortex (PCC)/retrosplenial cortex (Rsp), inferior parietal lobule (IPL), lateral temporal cortex (LTC), dorsal medial prefrontal cortex (dMPFC), hippocampal formation (HF+) [6,39] Self-referential thinking, internally directed or spontaneous selfgenerated thought, mind-wandering visual system spanning most of occipital cortex, often including a small portion of superior parietal cortex and a portion of the postero-lateral thalamus processing of visual information auditory system transverse temporal gyrus, superior temporal gyrus, pre and postcentral gyrus, insula) processing of auditory information somatosensory and motor system S1, M1, and some pre-and postcentral-gyrus cortex processing of touch and control of action activity was found for QF in three networks. Activity in the frontoparietal task control network is consistent with the idea that formulating the question and planning to speak requires more dynamic allocation of cognitive resources compared to saying the query. The finding of both the dorsal and ventral attention networks is somewhat contradictory as it is typically thought that top-down and bottom-up forms of attention are complementary. That both are active might reflect the relatively long time scales used in the experiment, raising the possibility that activation of the ventral attention network is a transient that occurs at the beginning of the epoch that later switches to top-down attention in the process of formulating the question. The default mode network was found to be lower during QF and more active during QS. The reduced activity for QF is consistent with the idea that default mode regions decrease activity when participants are performing a task and would could also possibly suggest that during the QS epoch there was enough time for participants to disengage from the task, allowing an increase of activity in the default mode.
Analysis of QS vs RJ network: In this contrast, four of the ten networks showed a difference, with no change of activity found in either the dorsal or ventral attention networks. Greater activity was found for QS in two networks that included the cingulo-opercular task network and the salience network. These two networks share an overlap of anatomical regions, and their activity suggests that compared to the RJ epoch there were both efforts spent on maintaining task set over the period of stating the question and potentially also an affective evaluation of the question that was posed. Such an affective tagging could include the expectation of success of the question and would be useful for subsequent evaluation of the results of the query. Greater activation was found in the frontoparietal task control network and the default-mode network. Higher activation in the fronto-parietal task control network suggests that dynamic restructuring of cognitive resources was required during RJ. The result of higher activation in the default-mode network for RJ is equivocal, though one possibility is that searching the provided document stimulated self-referential or self-initiated thought.
Network view of RJ vs SJ contrast: In this contrast, nine of the ten networks showed a difference, though the dorsal attention network had a single region equal for both epochs. Greater activity was found for RJ in two networks, including the fronto-parietal task control network and the default-mode network. These results are similar to the contrast between QS and RJ and have a similar explanation that RJ required greater dynamic control of cognitive resources and the task of examining the document and assessing its relevance could lead to self-referential thought. Greater activity was found for SJ on three networks, including the ventral attention, cingulo-opercular task control and salience networks. The ventral attention network activation suggests that the request to provide a satisfaction judgment entailed a transient switch of attention. The cingulo-opercular network and salience networks, like in the contrast of QS versus RJ, both showed greater activation during SJ than RJ, and the explanation would be similar that providing a satisfaction judgment requires maintaining a stable task set and that the outcome is tagged for its emotional or cognitive significance.

Summary of Network Analyses
The results of the network analysis for the ten networks considered are shown in Table 6 for all four transitions. For each transition, we looked at the network membership of each cluster and examined which time epoch produced the larger activation and counted the total number of times each epoch was more active. For instance, in the contrast of IN versus QF, two of the brain areas found in the fronto-parietal task control network had greater activation during IN, while one area had a larger activation during QF. The results show that across the different epochs of the procedure there was differential activation of the different networks. Further exploration of RQ2 revealed complex patterns present in the network analysis that provided several observations worth noting. For example, the fronto-parietal task control network appears to be activated during complex epochs such as IN and RJ when participants must read and evaluate the visually provided material, and this is consistent with the role of this network in dynamically regulating mental processes. Second, when providing a query or satisfaction judgment there is joint activation of the cingulo-opercular and salience networks, which likely reflects both an aspect of maintaining task set while transferring internal information (e.g. question, satisfaction) to an external response and that the affective relevance of this response is being tagged, potentially as part of a learning process. Finally, the default-mode network appeared active while participants were performing RJ which could be due to them accessing autobiographical or other self-referential information that is known to activate the default mode.

CONCLUSION AND FUTURE WORK
Motivated by the ubiquitous nature of search in the modern world, this paper investigated brain activity during a search process. Our current understanding of search as a process composed of distinct steps has been based on indirect measures of brain activity, and it can be greatly refined and extended by the use of more direct measures. We measured the brain activity of twenty-four participants during a search process that involved answering questions carefully selected from the TREC-8 and TREC 2001 Q/A Tracks and used a within-subject design that compared brain activity between distinct epochs during the search process. The key findings which emerged from the results are that the analysis of fMRI brain data revealed differences in brain activity that identified transitions in search (addressing RQ1). Though these differences were distributed throughout the brain, they could be associated with the activity of large-scale functional brain networks that relate to different cognitive functions. (addressing RQ2). These differences appeared to reflect the task demands and an interplay between the brain networks subserving different functions. The brain networks explored in this research as the basis of state transitions are well established in the cognition and cognitive neuroscience literature. Thus, our findings are useful in providing a way to map the unique sequence of information search behaviour into established cognitive and neural systems. This understanding of the neural systems that support search can inform more applied aspects of search behaviour. For example, when considering search by special populations with cognitive difficulties we are able to relate a knowledge of their underlying neural impairments to the neural requirements of search. For typical individuals we gain an understanding of the essential neural information processing and this can be used to understand errors and the basis of why certain search scenarios might be found to be more difficult or fatiguing.
Moreover, the current results provide us with a foundation to further explore and quantify the neural system involved in search. In the present research we used a somewhat fixed search procedure and probed it with a univariate analysis that identified the neural basis of transitions between states. This knowledge stimulates future research using a multivariate pattern classification approach [21] that could potentially identify the states themselves. Advances have been made in this area already for the binary classification of the type of information being processed [15].
In conclusion, the results of this experiment provide insight into how brain activity changes over the different search epochs and how this relates to the activity of large-scale brain networks that subserve fundamental cognitive functions. We believe our study and conclusions constitute an important first step in unravelling the brain functions involved in a search process and therefore help search engines to better assist searchers throughout this process.