The high standard of excellence in medical education today must not be taken for granted. In the early twentieth century, medical education in North America was in a deplorable condition. It was for this reason that the Council on Medical Education (CME) was formed in 1904 by the American Medical Association (AMA) with the primary purpose of reforming the ailing system. Around the same time, The Carnegie Foundation determined to focus its philanthropic attention on the improvement of healthcare in America. In 1908, the AMA CME commissioned The Carnegie Foundation to conduct a survey of medical education in America and Canada in an effort to advance the reform agenda. The person selected to conduct the survey was Abraham Flexner. Flexner was not a physician nor scientist. Flexner was an educator. A former high school teacher, Flexner went on to obtain his MPhil at Harvard and traveled extensively in Europe studying pedagogical methods used at universities there. It was the publication of his exhortation for the American education system in his book, The American College, that brought Flexner’s name to the attention of the then Head of The Carnegie Foundation, Harry Pritchett, who subsequently appointed Flexner to conduct the survey. In his 1910 report, Medical Education in the United States and Canada, Flexner articulated basic recommendations that soon formed the blueprint of American and Canadian medical education, which some have attributed as the catalyst for the most profound reformation of medical education and practice in history – not to mention generous financial gifts from the Rockefeller and Carnegie foundations. Over 100 years later, the core principles by which most medical schools operate by today can find their origin in Flexner’s 1910 report.(1)
Flexner’s report was not perfect – in fact, it was far from it. Besides violating the rights of women and people of color, it majored on scientific knowledge as a prelude for clinical training with such blind passion that it ultimately overlooked two imperatives of the profession: medical ethos and social responsibility. Flexner borrowed this mentality from the German medical education model.(1) “An education in medicine,” wrote Flexner, “involves both learning and learning how; the student cannot effectively know, unless he knows how.” Patients became first and foremost as serving the academic pursuits of the medical professor, and involvement in the arts and humanities of medicine was unimportant for the physician envisioned by Flexner.(1) From this originated the widespread adoption of traditional “2+2” curricular structure consisting of two years instruction in basic sciences (e.g., anatomy, physiology) followed by two years of clinical training in a teaching hospital.(2)(3) In addition, science was to be the animating force in the physician’s life, and the advancement of scientific knowledge was to trump all other commitments of the academic physician. Flexner, an educator whose philosophy was shaped by a pathologist (William Henry Welch) and their shared appreciation for German tradition, can hardly bear all the blame for his apparent oversight of the service and healing role of the physician.(1)
One of the earliest critics of the movement spurred by Flexner’s report was a contemporary of Flexner himself, Dr. William Osler. While Osler paid respect to the central focus of science in medicine, he did so only as this focus served the more important priorities of patient welfare and education of students. Therefore, Osler firmly believed that the Flexnerians had their priorities wrong; however, upon relocating from America to England to assume the Regius Professorship at Oxford, Osler’s voice in the controversy went near silent. As for Osler’s followers who remained in America, their objections were drowned out by the deluge of conflict of interest linked to the implementation of the “full-time” system that provided an adequate salary (made possible by extravagant donations from the aforementioned foundations) enabling medical professors to be freed from any patient care responsibilities and dedicate their lives to research and teaching.(1)
Since Flexner’s report, virtually every aspect of medicine has been shaken by tectonic changes – our understanding of human learning has exploded, novel technologies and medications have revolutionized diagnosis and treatment, the profession expanded its role at the micro-, meso-, and macro-levels while simultaneously specializing and sub-specializing its scope of practice, and the healthcare system has become increasingly complex in terms of delivery, financing, and public policy.(4) Essentially, medicine has come full circle, and nearly every organization within the medical profession – faculties, universities, hospitals, licensing authorities, accreditation bodies, specialty boards, and medical associations – is re-examining the foundation: medical education. Once again, in 2010, The Carnegie Foundation commissioned a team to examine medical education across the USA and make recommendations for the future, which culminated in the book, Educating Physicians: A Call for Reform of Medical School and Residency.(5) The team identified several important gaps where medical education is failing to meet current and future demands. Particular to this study, the team found that medical education is not learner centered, makes poor connections between formal knowledge and experiential learning, and does not make adequate use of the learning sciences. Further, the team found the “2+2” system to be an inadequate approach for today, which is corroborated elsewhere,(6)(7) and called for integration of knowledge and skills in the basic, clinical, and social sciences with clinical experiences both at the medical student and resident level.(4) As such, integration is a strategic priority for the future of medical education.(8)
1.1 Integrated curriculum
The integrated curriculum has been defined as “a fully synchronous, trans-disciplinary delivery of information between the foundational sciences and the applied sciences throughout all years of a medical school curriculum.”(9) The etymology of the term integrate has its root in the Latin, integrat, meaning ‘made whole’ and is closely related to the English term, integer, meaning ‘whole’. More than being ‘made whole’, the modern definition of integration in medical education portrays the thought of making whole by breaking down barriers that traditionally divided the foundational and applied sciences and believes that connections thus made between these subjects will be improved and will enhance learners’ retention of knowledge and development of clinical skills.(9)
The integrated curriculum was first reviewed in general education by Beane in 1977(10) and was later described within the context of medical education by Harden et al. in 1984.(11) The “McMaster approach” to medical education is one of the earliest examples of its implementation, development, and revision.(12) Many learning theories have been explored to understand how adults learn and how integration enhances learning in the context of medical education.(13) Several models and levels (program-, course-, and session-level) of integration have also been described and reviewed.(14) At the program level, integration has been described as horizontal integration (across disciplines but within a finite period of time, e.g., combining courses that were once separate into a year-long introductory course),(15) vertical integration (across time, e.g., the “Z-shaped model” of progressive introduction to clinical practice while maintaining a persistent basic science component throughout all years of the curriculum),(16) or spiral integration (combination of both horizontal and vertical, i.e., learning both basic and clinical sciences across time and subject matter).(17) Course-level integration refers to the contextualization of basic science concept teaching (e.g., Poiseuille’s law and blood flow within a vessel)(18,19) and shared teaching (e.g., basic scientists and clinicians teaching a course together).(20) Session-level integration are those micro-level activities used day to day to teach content and include specific learning interventions such as the causal network where pathologies are taught from a cause-effect perspective and etiology is linked to pathophysiology and clinical presentation.(21–24)
Integration activities are often undertaken with the expectation that organizational change will achieve integration. However, it is clear that adopting an integrated curriculum in a medical school will not automatically produce cognitive integration within the minds of its learners.(14) Of note, Kulasegaram et al. (2013) point out that integration of basic science and clinical practice is likely most effectively achieved at the level where students make direct contact with the content of the formal curriculum, that is, the session level. Whereas integration is usually described as methods and techniques, it is better framed as the action of learning. Thus, our understanding of integration in medical education should emphasize the cognitive activity that takes place within the learner as he or she links clinical concepts with basic science.(14)
1.2 Cognitive integration
Cognitive integration occurs when a conceptual cognitive connection is made between different knowledge sources,(25)(26) but it is more than that. The learner-centered definition of integration in the medical education curriculum refers to the process in which links and interrelations are formed between different types of knowledge (e.g., basic science, clinical science) within the learner in a manner that enhances understanding and performance of clinical activities (e.g., diagnosis, management).(27) Therefore, the scope of cognitive integration extends beyond acquisition, interrelation, and retention of different types of knowledge to the transfer and application of this knowledge to different situations. Teaching basic and clinical sciences in an integrated fashion to novice learners seems to support an organized cognitive conceptual coherence with the learner that has been shown to result in superior knowledge retention and application to solve diagnostic problems when compared with teaching basic and clinical sciences separately.(21–25,28–31)
The end goal of integrated curricular reform is, therefore, not merely the performance of integration activities such as organizing delivery of content in an integrated manner. Rather, the end goal is a cognitive action within the student that enhances acquisition, interrelation, and retention of basic and clinical science knowledge that is borne out in the performance of clinical skills. An objective end goal lends clarity to how evaluating the effectiveness of integration efforts might be achieved. A given integration strategy could be evaluated by assessing how students utilize basic science knowledge in the performance of a specific clinical skill.(14)
Cognitive integration is preceded by knowledge, which is preceded by memory, which is preceded by learning; therefore, a brief review of learning and memory is incumbent. In their book, Make it Stick, authors and cognitive psychologists Peter Brown, Henry Roediger, and Mark McDaniel define learning as acquiring knowledge and skills and having them readily available from memory so one can makes sense of future problems and opportunities. They emphasize three immutable aspects of learning, namely, learning requires memory, individuals need to keep learning and remembering for their entire lives, and learning is an acquired skill (and the most effective strategies are often counterintuitive).(32) Since the late 19th century, the cognitive psychology and education literature has consistently and voluminously yielded credence to two critical components for learning and memory: retrieval practice and distributed practice.(33–35)
1.3 Retrieval practice
Retrieval practice refers to the exercise of actively recalling a previously learned fact. Two key findings central to the discussion of retrieval practice are the testing effect and the retrieval-effort effect.
The testing effect refers to the finding that actively recalling a previously learned fact (re-testing, repeated testing) leads to increased long-term retention of that fact than if it were passively read (re-studying, repeated studying).(36–39) In education, testing is most frequently utilized as a tool to evaluate what the student knows; however, empirical research supports that re-testing substantially enhances retention when compared with re-studying(37,40,41) and potentiates the future learning of new materials.(42) One explanation for the testing effect is the elaborative retrieval hypothesis, which posits that the retrieval process during testing activates elaborative information related to the item being recalled thereby facilitating successful recall of the item in subsequent recall attempts via reactivation of any of this related information.(43,44) As such, testing is a powerful tool not only for assessing learning but also for improving it.(36,45) The act of recalling a memory changes the memory itself by interrupting forgetting and making it easier to recall again later.(32) Attempting to retrieve a memory, even if unsuccessful, appears to enhance encoding for recall in a subsequent retrieval period.(46) Beyond the benefits of long-term retention, the testing effect induces flexibility in recalled knowledge enabling it to be transferred and applied to novel situations. Testing assists the learner in organizing knowledge, forming a coherent knowledge base, and recalling related information.(37,47) The benefits of the testing effect are conveyed by one of the terms often used to describing the concept, “test-enhanced learning”.(36)
The retrieval-effort effect refers to the observation that a memory that is more difficult to recall during retrieval practice is more likely to be retained better in the long-term.(43,44,48) In other words, difficult initial retrieval facilitates subsequent recall,(49) and effortful learning produces durable learning.(50) It is evident that when recall is required after a meaningful amount of time has elapsed and some forgetting has occurred, the retrieval is more difficult and feels less productive; however, the increased effort for retrieval produces a more durable memory by strengthening the retrieval route and enabling more versatile transfer and application in future novel situations.(32) The effect of taking the stairs rather than the elevator on development of strength, balance, coordination, agility, and stamina is a helpful parallel.
1.4 Distributed practice
Distributed practice refers to the exercise of separating review episodes (whether actively recalling or passively reading) for a given item by a measurable time lag.(51) Central to the discussion of distributed practice is the spacing effect.
The spacing effect refers to the finding that the time interval separating different retrieval attempts of the same fact impacts upon acquisition and retention of the fact in subsequent retrieval attempts.(51) If retrieval practice was performed by testing and retesting the learner again and again without permitting any interval forgetting, it would become mindless repetition, and the benefit of the retrieval-effort effect would be lost.(32) Indeed, the evidences supports that when the retrieval practice takes place is as important as the act of retrieval practice itself, and this evidence dates back as far as 1885 to the pioneering work of Hermann Ebbinghaus on the topic of temporal distribution of practice.(51) The spacing effect is thus intimately related to the retrieval-effort effect – after an initial test, spacing out retrieval attempts permits more forgetting, which makes subsequent retrieval more effortful and retention more durable. Research is clear that the timing of retrieval attempts or, in other words, the space of time elapsed between attempts, affects retention. For any given subsequent test, there exists a spacing interval between previous retrieval tests at which retention will be optimized.(51,52) In general, the optimal spacing interval increases as the duration over which the information needs to be retained increases.(51) When retrieval practice attempts for a fact occurs more than twice, there is inherently more than one spacing interval, and in comparison to the first spacing interval, the subsequent spacing intervals may be equal (fixed), progressively longer (expanding), or progressively shorter (contracting). The evidence suggests that expanding spacing intervals either benefit learning(51) or produce effects similar to retrieval practice with fixed spacing.(51,53)
The corollary to an optimum spacing interval is that too short or too long a space between retrieval attempts can reduce retrieval success at a subsequent test,(54) which may explain some conflicting results in the literature on distributed practice.(55) Indeed, when the spacing interval is too short or the learner is not subjected to any interruptions of intervening items or time, the learning is said to be massed in contrast to spaced.(51) Note that massed practice is to massed learning as distributed practice is to spaced learning, and cramming is a form of massed practice. Hundreds of empirical studies and several large systematic reviews and metanalyses have confirmed that distributed practice increases retention (51,56–59) and that the benefits of distributed practice far outweigh the benefits of massed practice.(51,58,59) Several theories attempt to explain how distributed practice accomplishes such results;(60,61) however, no single theory can account for all the effects observed empirically, so distributed practice remains a topic of active research.(51) A neurobiological basis for the superiority of the spacing effect over massed learning has been described through synaptic mechanisms.(62)
Distributed practice has been shown to confer learning benefits, particularly better retention, over massed practice in medical residents.(63–65) Incorporation of distributed practice into the traditional face-to-face lecture format increases time spent self-studying and learners’ satisfaction compared to the traditional format alone in dental students.(66) Distributed practice accelerates learning of surgical skills and protects against performance drop over time when compared with massed practice in medical students.(67) Distributed practice seems to confer benefits across a wide range of ages and has practical implications in that teaching content delivered according to a distributed presentation schedule has learning benefits over a massed or clustered presentation schedule.(68)
In today’s world, where sleep deprivation among students is nearly universal, a further benefit of distributed practice is that, due to the spacing effect, it appears to confer strong protection against the effects of sleep restriction on recall performance when compared to massed practice (i.e., cramming), which often goes hand-in-hand with sleep restriction.(69)
1.5 Learning and memory in the real world
Given what modern cognitive psychology research has discovered about the best methods of learning and memory retention, namely, that retrieval and distributed practice have powerful effects to this regard across an impressive range of practice-test formats, kinds of material, learner ages, outcome measures, and retention intervals,(35) how are these methods viewed and utilized today? Unfortunately, learning how to learn is not a standard component of most educational curricula, including medical school.(33,70)
Cramming is the most commonly employed method of studying – likely because it seems to be the easiest and fastest way to learn. Cramming is the path of least resistance for the learner; however, cramming is not effortful learning and, therefore, not durable. Approximately 75% of content learned by cramming is lost within 48 hours.(32,71) Further, cramming is likely to deceive the learner – rereading material over and over again with a short or no spacing interval in between presentations results in increased familiarity with the content, which can easily be mistaken as mastery.(32,72)
It is counterintuitive that the process of effortful retrieval at spaced intervals, allowing some forgetting to occur, improves learning and strengthens memory. Indeed, decades of research has repeatedly confirmed that the majority of teachers and learners do not appreciate that conditions which promise rapid acquisition often to not promote long-term retention nor do they understand what strategies are most effective for retention.(36,39,72–75) Retrieval and distributed practice are less commonly employed as pedagogical strategies by teachers and as self-regulatory strategies by learners,(41) whereas cramming and massing in the form of rereading is used by the majority of learners in preparation for summative testing.(35,72) Even when students achieve better test scores with retrieval practice then they have with cramming, they still believe that cramming is more effective.(36,76) According to Tullis et al.,(77) all previous laboratory experiments investigating learners’ metacognitive monitoring of their learning strategy effectiveness, participants judged massed practiced items as more likely to be remembered but actually remembered more of the retrieval practiced items. Learners lack cognitive awareness of the mnemonic benefits of retrieval(39,78) and distributed practice.(79)
Instead of using practice testing to improve their memory, students self-test to diagnose what they do and do not remember in order to guide further study, often without realizing that doing so improves their memory.(75) This usage of self-testing solely as a diagnostic tool indicates that most students who use self-testing only recognize its indirect benefits.(75,80) In a recent study, the authors contend that learners’ metacognitive knowledge of the benefits of retrieval practice may be more complex than simple unawareness and that their study method decisions may be risk averse, choosing to take practice tests only when their confidence in successful retrieval is unnecessarily high.(81)
Another explanation is that there is a disconnect between the real-world and the laboratory given that students under experimental settings may be less motivated to retain information than they would be in classroom settings and, therefore, may recognize more benefits of self-testing and use it more widely in their day-to-day study than in the laboratory;(77) however, this is not totally borne out by the evidence. For example, investigations of undergraduate learners’ real-world study behaviours indicate that appreciation and use of retrieval practice is low,(39,72) whereas, in at least one study,(82) medical students have been found to engage extensively in retrieval and distributed practice in the form of practicing multiple choice questions and flashcards in preparation for medical licensing examinations.
1.6 Flashcards as a learning and memory tool in medical education
Based on nearly 135 years of cognitive psychology research and a large body of evidence demonstrating high utility of retrieval and distributed practice for learning and memory, flashcards, which can incorporate both retrieval and distributed practice, are uniquely poised as a powerful tool to enhance learning and memory for learners in the medical profession where knowledge is expanding at an immense and unprecedented rate.(83) While incorporation of retrieval and distributed practice into medical curricula is limited,(82,84) by purchasing, creating, and/or sharing and using digital flashcards autonomously in self-regulated study,(85) health professions students(55,82,86–90) and medical residents(91,92) have found this tool feasible, desirable, and useful and have experienced its benefits over traditional tools.
Learning objects are a group of instructional materials structured to meet an educational objective, and digital learning objects have many advantages over traditional paper-based learning objects.(83) Flashcards are an example of learning objects. Note that traditional paper and modern digital flashcards inherently incorporate retrieval practice. While distributed practice with paper flashcards is possible (e.g., Leitner system), it would be extremely unwieldy as the number of flashcards increased; however, digital flashcards can easily exploit the benefits of distributed practice by building algorithms into the flashcard software that automatically calculate and implement spacing intervals for each flashcard based on the user’s learning progress. One such example of a digital flashcard application is Anki.
Anki is an open-source digital flashcard application that enables users to create, study, and share flashcards on their personal devices (phone, tablet, computer).(93) Anki is multi-platform, meaning that its app runs on Windows, Mac OSX, Linux/FreeBSD, and some mobile devices including Android and iOS systems. Anki is freely available on all these platforms except iOS, and is open source with an impressive library of add-ons created and contributed by its user community. AnkiWeb is a free extension to the application-based version of Anki that allows the user, in the absence of his/her personal device, to study his/her flashcards online. AnkiWeb also enables the user to keep his/her flashcards and study progress synchronized across multiple personal devices. Anki flashcards support text, images, audio, videos, and scientific markup. Anki’s distributed practice capability is via its embedded spaced repetition algorithm, which is based on an open-source algorithm from the proprietary SuperMemo flashcard program called SM-2 and can be traced back to its creator, Piotr Wozniak.(94)
Anki presents flashcards much like the traditional paper flashcard. The “front” of the flashcard containing the question is presented on the user’s screen. The user retrieves the information mentally and then taps or clicks the card to view the “back” of the card, which contains the answer for comparison to that which was mentally recalled. Based on the accuracy of the user’s mental recollection as judged by the user, the user then taps or clicks a selection from a menu of buttons that essentially tells Anki’s algorithm the difficulty level of that flashcard’s retrieval for this particular user at this particular time (e.g., forgot, difficult, good, easy). Anki then uses that information to calculate the spacing interval for the next retrieval of that flashcard, in essence, predicting how long it will take for the user to forget that flashcard and scheduling review of that flashcard immediately prior to this time and, thus, maximizing the retrieval-effort effect. Flashcards that are forgotten by the user are thus assigned shorter spacing intervals and reviewed more frequently, whereas flashcards that are easy for the user to recall are assigned longer spacing intervals and reviewed less frequently. The author explains this process from a constructivism view that can be thought of as Anki building and maintaining a digital representation of the user’s memory by forcing encoding exercises, triggering retrieval routes, and collecting data on the neurobiological process of consolidation occurring within the user in the interim. By modelling the user’s memory, Anki can intelligently and efficiently add to it and assist it in strengthening and improving itself.
Additional advantages of digital flashcards, specifically Anki, includes their portability, allowing the user to create and study flashcards on-the-go via his/her mobile device; their shareability, fostering collaboration and synergy for learning and memory between users; and customizability, permitting each user to personalize flashcard content, formatting, layout, and organization as well as edit content to ensure it remains contemporaneous with the current knowledge and best practices. The main drawbacks of digital flashcards, in the author’s experience, are the steep learning curve required to fully master their use in day-to-day learning and, therefore, reap their full benefits; the challenge of overcoming previously long and deeply engrained study habits, “tried and true” methods, and metacognitive beliefs about “what works best for me”; and the cognitive dissonance created by trading the aforementioned intuitive habits to rely on counterintuitive methods that feel slower and require more cognitive effort, even despite their being backed by nearly 130 years of cognitive psychology research.
Finally, Anki’s open-source design also enables access to user metadata by extraction and query of the user’s review log from Anki’s database. For example, for each flashcard review, one can determine when the review took place, the difficulty level of the flashcard as rated by the user at that review, the spacing interval that occurred previous to that review, the new spacing interval for the subsequent review, and the amount of time the user spent on the “front” and “back” of the flashcard at that review. For those interested in conducting research using Anki as a learning and memory tool, the utility of the metadata access is obvious.
1.7 Theoretical Underpinnings
In order to match current formats of medical education, some medical educators have proposed that Bloom’s taxonomy of learning domains – cognitive, psychomotor, and affective – be redefined as knowledge, skills, and attitudes.(9,95) Note that I focus herein on the knowledge-based domain.(13) I will outline several applicable learning theories and concepts here.
1.7.1 Adult learning theories
Knowles suggested that adult learners learn differently than child learners in several respects and coined the concept of “andragogy” as the art and science of helping adults learn, in contrast to pedagogy.(96–98) In particular, key differences between adult and child learners are that adults have a different readiness and motivation to learn (e.g., I need to learn because my circumstances are changing. I learn because I want to.) and a different orientation to learning (e.g., Learning will help me deal with the situation in which I find myself).(13) Adults are interested in meaningful learning.(99) In general, it is only when adults understand a topic’s relevance that they will be willing to invest time into learning it.(9) Thus, linking basic science to scenarios encountered in clinical practice adds relevance to the basic science for the medical learner while simultaneously reinforcing and expending the cognitive conceptual coherence of the clinical science within the learner.
However, in medical education, to disregard all learning theories related to child learners would be premature – their lessons should be considered carefully. The educational journey through medical school and residency has many similarities to that of learning to read and process a new language. The sheer volume of new knowledge that the medical learner must acquire, retain, and have at-the-ready for transfer to real-world clinical scenarios is staggering. When second-grade children are learning to read, does reading words in context (contextual word training) act to further enhance reading development beyond that achieved by reading words in isolation (isolated word training)? I now review three learning theories from the child cognitive psychology literature related to reading, specifically word training,that may be applicable to the subject under consideration: the focal attention hypothesis, the contextual facilitation effect, and transfer-appropriate-processing approach to reading transfer.
1.7.2 Focal attention hypothesis
In the focal attention hypothesis, Samuels (1967) submitted that learning to read words is facilitated when the greatest amount of attention is focused on the orthographic components of the words.(100) It is thought that the requisite mental effort to convert letters into sounds builds stronger word representations in the memory.(101) It follows that cues received from context, despite facilitating more accurate word identification, actually detract from the print analysis and reduce long-term word retention.(102) In other words, context allows children to read more words but to learn fewer of them.(101) The corollary is that contextual word training, despite seemingly facilitating word acquisition, is more so a short-cut that reduces cognitive decoding effort for acquisition and, thus, does not facilitate long-term retention. Evidence supporting the focal attention hypothesis includes experiments that presented words alongside illustrations, within meaningful sentences, or as part of predictable stories and found lower word retention than when presented in isolation.(102) This seems to be related to the retrieval-effort hypothesis and how effortful learning produces durable learning.
The parallel to isolated word training might be flashcards that present basic science and clinical science separately and making no specific effort to integrate and interrelate these two areas of knowledge.
1.7.3 Contextual facilitation effect
The contextual facilitation effect refers to the observation that context serves as a self-teaching mechanism when performing word training,(102) that is, reading in context allows children to learn significantly more words than when reading words in isolation.(103) Children learning words in context have the greatest gains in accuracy, speed, and comprehension in comparison to children learning words in isolation.(104,105) Additional support for this effect includes evidence that contextual cues can be used by children as an aide to assist subpar decoding ability and, thus, offset deficient word analysis skills.(106)
The parallel to contextual word training might be flashcards that present basic science and clinical science in an integrated manner such that cognitive integration within the mind of the learner is fostered.
While it is clear that contextual word training has a definite advantage over isolated word training in terms of improved word acquisition, proponents of isolated word training have historically argued that the trade-off of contextual word training is inferior long-term retention; however, it is now apparent that the evidence they use to support this argument was flawed in that when traditionally testing retention, the tests administered were presenting words as a list, i.e., in isolation.(101,102,105) Measures of retention were confounded in previous experiments because words learned under both training conditions were only tested in isolation tasks. Indeed, more recent research has found that words learned via contextual word training are no more vulnerable to be forgotten than words learned in isolated word training when the testing task is taken into account.(102) Words learned in isolated word training are given an artificial advantage when recalled isolation because their encoding and recall tasks are similar. On the contrary, words learned in context and recalled isolation are disadvantaged because, in addition to being recalled, they are being transferred between two different tasks: from a contextual encoding task to an isolated recall task.(102)
This led researchers to take a closer look at not only word acquisition and word retention but also word transfer and the nature of the transfer testing task.
Note to self: This is consistent with the zone of proximal development (Taylor, 2013).
Researchers observed that if a context exercise (e.g., reading a sentence) was the method of assessing word retention and transfer, contextual word training was superior, but if an isolation exercise (e.g., reading a list of words) was the method of assessing word retention and transfer, isolated word training was superior.(102) Thus, the transfer-appropriate-processing approach to word training states that words are read most accurately when the same cognitive processes are engaged during training and transfer tasks.(107) In other words, word transfer increases as the congruency between the training and transfer task increases, and word retention is facilitated when the encoding and retrieval processes are similar for word-specific representations in memory.(108) Therefore, words learned in context are less likely to be read accurately when subsequently encountered in isolation, and words learned in isolation are less likely to be read accurately when subsequently encountered in context. Since the end goal, real-world, transfer task for the majority of children is a context exercise, i.e., reading and comprehending a passage, contextual word training is superior in terms of word acquisition, retention, and transfer when compared with isolated word training. Given that words are rarely presented as a list in naturalistic reading, context word training enhances reading skill beyond isolated word training.(102)
The parallel in medical education seems to be that since the end goal of teaching basic and clinical sciences in medical school and residency is the recollection and use of these types of knowledge in clinical practice to solve and manage diagnostic problems (a context exercise), then the transfer-appropriate-processing training task should present, teach, and seek to format these types of knowledge in an integrated and contextual manner in order to enhance acquisition, retention, and transfer.
1.7.5 What makes a good flashcard?
The utility of flashcards as a learning tool, especially those that incorporate the techniques of both retrieval practice and distributed practice, is very high – this is based on almost 135 years of learning and memory research that show these two techniques to be superior than any other known techniques.(35,37) Surprisingly, despite this high utility and widespread use of flashcards in education for many years, the characteristics contributing to a high-quality flashcard has not yet been studied empirically as far as the author is aware. Tips for such characteristics have been proposed based on extrapolations from cognitive psychology research and vision research as follows.(85)
For retrieval-based learning activities, retrieval success is improved by short-answer format questions when compared to multiple choice questions(109) – this observation could translate into a simple maxim for formatting questions on flashcards: prefer short-answer questions and avoid multiple choice questions.
Research on the subject of cognitive loading indicates that when the learner’s working memory is surpassed by the cognitive load of a learning task, performance and learning is impaired. Working memory, limited to processing a finite number of informational elements at one time, presents a bottleneck for learning. Three types of cognitive load have been defined – intrinsic, extraneous, and germane. Drawing on cognitive load theory, various instructional techniques have been developed to minimize intrinsic and extraneous cognitive load in an effort to free up the maximum amount of working memory for the cognitive load most important for learning, that is, germane load.(110) Such instructional techniques could be applied to flashcards.
The expertise-reversal effect refers to the finding that instructional techniques enhancing learning among novice learners may not enhance and may even detract from learning among more advanced learners.(110) An extension of this effect may be that flashcards that are optimized for enhanced learning in novice learners may not be optimized for the same in advanced learners, vice versa.
Reading speed and accuracy is influenced by crowding between words horizontally and vertically.(111,112) Additionally, formatting font size, style, and color and use of bold, italics, and underline may direct and focus the learner’s attention on key concepts and thereby optimize the distribution of cognitive load.(85,113) Again, the application of these ideas to flashcards is feasible.
However, as far as the author is aware, no characteristics, including those that could be extrapolated from the above concepts, have been investigated in the context of flashcards. I submit that the characteristics making up a flashcard may impact knowledge acquisition, retention, and transfer.
The end… cliff hanger!
1. Duffy TP. The Flexner Report–100 years later. Yale J Biol Med [Internet]. 2011 Sep [cited 2019 Oct 3];84(3):269–76. Available from: http://www.ncbi.nlm.nih.gov/pubmed/21966046
2. Anderson J. The continuum of medical education. The role of basic medical sciences. J R Coll Physicians Lond [Internet]. 1993 Oct [cited 2019 Oct 3];27(4):405–7. Available from: http://www.ncbi.nlm.nih.gov/pubmed/8289163
3. Schmidt H. Integrating the Teaching of Basic Sciences, Clinical Sciences, and Biopsychosocial Issues. Acad Med [Internet]. 1998 Sep [cited 2019 Oct 3];73(Supplement):S24–31. Available from: http://www.ncbi.nlm.nih.gov/pubmed/9759115
4. Cooke M, Irby DM, O’Brien BC. A Summary of Educating Physicians: A Call for Reform of Medical School and Residency [Internet]. 2010 [cited 2019 Oct 5]. Available from: http://archive.carnegiefoundation.org/pdfs/elibrary/summary_educating_physicians.html
5. Cooke M, Irby DM, O’Brien BC, Carnegie Foundation for the Advancement of Teaching. Educating physicians : a call for reform of medical school and residency [Internet]. Jossey-Bass; 2010 [cited 2019 Oct 5]. 304 p. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3204956/
6. Cooke M, Irby DM, Sullivan W, Ludmerer KM. American Medical Education 100 Years after the Flexner Report. Cox M, Irby DM, editors. N Engl J Med [Internet]. 2006 Sep 28 [cited 2019 Oct 3];355(13):1339–44. Available from: http://www.nejm.org/doi/10.1056/NEJMra055445
7. Irby DM, Cooke M, OʼBrien BC. Calls for Reform of Medical Education by the Carnegie Foundation for the Advancement of Teaching: 1910 and 2010. Acad Med [Internet]. 2010 Feb [cited 2019 Oct 3];85(2):220–7. Available from: http://www.ncbi.nlm.nih.gov/pubmed/20107346
8. Association of the Faculties of Medicine of Canada. The Future of Medical Education in Canada: A Collective Vision for MD Education [Internet]. 2010 [cited 2019 Oct 3]. Available from: https://afmc.ca/future-of-medical-education-in-canada/medical-doctor-project/index.php
9. Brauer DG, Ferguson KJ. The integrated curriculum in medical education: AMEE Guide No. 96. Med Teach [Internet]. 2015 Apr 3 [cited 2019 Oct 3];37(4):312–22. Available from: http://www.ncbi.nlm.nih.gov/pubmed/25319403
10. Beane JA. Curriculum integration: designing the core of democratic education [Internet]. New York, NY: Teachers College Press; 1977 [cited 2019 Oct 7]. Available from: https://books.google.ca/books/about/Curriculum_Integration.html?id=ZnaBQgAACAAJ&redir_esc=y
11. Harden RM, Sowden S, Dunn WR. Educational strategies in curriculum development: the SPICES model. Med Educ [Internet]. 1984 Jul [cited 2019 Oct 7];18(4):284–97. Available from: http://www.ncbi.nlm.nih.gov/pubmed/6738402
12. Neufeld VR, Woodward CA, MacLeod SM. The McMaster M.D. program. Acad Med [Internet]. 1989 Aug [cited 2019 Oct 7];64(8):423–32. Available from: http://www.ncbi.nlm.nih.gov/pubmed/2751777
13. Taylor DCM, Hamdy H. Adult learning theories: Implications for learning and teaching in medical education: AMEE Guide No. 83. Med Teach [Internet]. 2013 Nov 4 [cited 2019 Oct 7];35(11):e1561–72. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24004029
14. Kulasegaram KM, Martimianakis MA, Mylopoulos M, Whitehead CR, Woods NN. Cognition Before Curriculum. Acad Med [Internet]. 2013 Oct [cited 2019 Oct 3];88(10):1578–85. Available from: http://www.ncbi.nlm.nih.gov/pubmed/23969375
15. Snyman WD, Kroon J. Vertical and horizontal integration of knowledge and skills – a working model. Eur J Dent Educ [Internet]. 2005 Feb [cited 2019 Oct 3];9(1):26–31. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15642020
16. Bradley P, Mattick K. Integration of basic and clinical sciences – AMEE 2008 [Internet]. 2008 [cited 2019 Oct 3]. Available from: https://amee.org/getattachment/Conferences/AMEE-Past-Conferences/AMEE-Conference-2008/Introduction-to-Medical-Education-Bradley-Mattick.pdf
17. Harden RM, Davis MH, Crosby JR. The new Dundee medical curriculum: a whole that is greater than the sum of the parts. Med Educ [Internet]. 1997 Jul [cited 2019 Oct 7];31(4):264–71. Available from: http://www.ncbi.nlm.nih.gov/pubmed/9488841
18. Harden R, Crosby J, Davis MH, Howie PW, Struthers AD. Task-based learning: the answer to integration and problem-based learning in the clinical years. Med Educ [Internet]. 2000 May [cited 2019 Oct 3];34(5):391–7. Available from: http://www.ncbi.nlm.nih.gov/pubmed/10760125
19. Greenberg R, Loyd G, Wesley G. Integrated simulation experiences to enhance clinical education. Med Educ [Internet]. 2002 Nov [cited 2019 Oct 3];36(11):1109–10. Available from: http://www.ncbi.nlm.nih.gov/pubmed/12406301
20. Weiner M, Walker JK. Utilization of basic and clinical health science personnel in a team approach for the achievement of competency in therapeutics. Med Educ [Internet]. 1977 Mar [cited 2019 Oct 3];11(2):114–8. Available from: http://www.ncbi.nlm.nih.gov/pubmed/853966
21. Baghdady MT, Pharoah MJ, Regehr G, Lam EWN, Woods NN. The role of basic sciences in diagnostic oral radiology. J Dent Educ [Internet]. 2009 Oct [cited 2019 Oct 3];73(10):1187–93. Available from: http://www.ncbi.nlm.nih.gov/pubmed/19805783
22. Woods NN, Neville AJ, Levinson AJ, Howey EHA, Oczkowski WJ, Norman GR. The Value of Basic Science in Clinical Diagnosis. Acad Med [Internet]. 2006 Oct [cited 2019 Oct 3];81(Suppl):S124–7. Available from: http://www.ncbi.nlm.nih.gov/pubmed/17001122
23. Cousens S, Blencowe H, Stanton C, Chou D, Ahmed S, Steinhardt L, et al. National, regional, and worldwide estimates of stillbirth rates in 2009 with trends since 1995: a systematic analysis. Lancet (London, England) [Internet]. 2011 Apr 16 [cited 2016 Jul 7];377(9774):1319–30. Available from: http://www.ncbi.nlm.nih.gov/pubmed/21496917
24. Woods NN, Brooks LR, Norman GR. The value of basic science in clinical diagnosis: creating coherence among signs and symptoms. Med Educ [Internet]. 2005 Jan [cited 2019 Oct 3];39(1):107–12. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15612907
25. Woods NN. Science is fundamental: the role of biomedical knowledge in clinical reasoning. Med Educ [Internet]. 2007 Nov 28 [cited 2019 Oct 3];41(12):1173–7. Available from: http://www.ncbi.nlm.nih.gov/pubmed/18045369
26. Norman G. The essential role of basic science in medical education: the perspective from psychology. Clin Invest Med [Internet]. 2000 Feb [cited 2019 Oct 3];23(1):47–51; discussion 52-4. Available from: http://www.ncbi.nlm.nih.gov/pubmed/10782317
27. Regehr G, Norman GR. Issues in cognitive psychology. Acad Med [Internet]. 1996 Sep [cited 2019 Oct 3];71(9):988–1001. Available from: http://www.ncbi.nlm.nih.gov/pubmed/9125988
28. Baghdady MT, Carnahan H, Lam EWN, Woods NN. Integration of basic sciences and clinical sciences in oral radiology education for dental students. J Dent Educ [Internet]. 2013 Jun [cited 2019 Oct 7];77(6):757–63. Available from: http://www.ncbi.nlm.nih.gov/pubmed/23740912
29. Woods NN, Brooks LR, Norman GR. It all make sense: biomedical knowledge, causal connections and memory in the novice diagnostician. Adv Heal Sci Educ [Internet]. 2007 Sep 17 [cited 2019 Oct 7];12(4):405–15. Available from: http://www.ncbi.nlm.nih.gov/pubmed/17318360
30. Goldszmidt M, Minda JP, Devantier SL, Skye AL, Woods NN. Expanding the basic science debate: the role of physics knowledge in interpreting clinical findings. Adv Heal Sci Educ [Internet]. 2012 Oct 15 [cited 2019 Oct 7];17(4):547–55. Available from: http://www.ncbi.nlm.nih.gov/pubmed/22002858
31. Balla JI, Biggs JB, Gibson M, Chang AM. The application of basic science concepts to clinical problem-solving. Med Educ [Internet]. 1990 Mar [cited 2019 Oct 7];24(2):137–47. Available from: http://www.ncbi.nlm.nih.gov/pubmed/2319972
32. Brown PC, Roediger HL, McDaniel MA. Make it Stick. Cambridge, Massachusetts: The Belknap Press of Harvard University Press; 2014.
33. Augustin M. How to learn effectively in medical school: test yourself, learn actively, and repeat in intervals. Yale J Biol Med [Internet]. 2014 Jun [cited 2019 Oct 4];87(2):207–12. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24910566
34. Roediger HL, Pyc MA. Inexpensive techniques to improve education: Applying cognitive psychology to enhance educational practice. J Appl Res Mem Cogn [Internet]. 2012 Dec 1 [cited 2019 Oct 4];1(4):242–8. Available from: https://www.sciencedirect.com/science/article/pii/S2211368112000915
35. Dunlosky J, Rawson KA, Marsh EJ, Nathan MJ, Willingham DT. Improving Students’ Learning With Effective Learning Techniques: Promising Directions From Cognitive and Educational Psychology. Psychol Sci Public Interes [Internet]. [cited 2019 Oct 4];14(1):4–58. Available from: http://pspi.sagepub.com
36. Roediger HL, Karpicke JD. Test-Enhanced Learning. Psychol Sci [Internet]. 2006 Mar 6 [cited 2019 Oct 3];17(3):249–55. Available from: http://www.ncbi.nlm.nih.gov/pubmed/16507066
37. Roediger III HL, Putnam AL, Smith MA. Ten Benefits of Testing and Their Applications to Educational Practice. In 2011 [cited 2019 Oct 3]. p. 1–36. Available from: https://linkinghub.elsevier.com/retrieve/pii/B9780123876911000016
38. Roediger HL, Karpicke JD. The Power of Testing Memory: Basic Research and Implications for Educational Practice. Perspect Psychol Sci [Internet]. 2006 Sep 24 [cited 2019 Oct 3];1(3):181–210. Available from: http://journals.sagepub.com/doi/10.1111/j.1745-6916.2006.00012.x
39. Karpicke JD, Roediger HL. The Critical Importance of Retrieval for Learning. Science (80- ) [Internet]. 2008 Feb 15 [cited 2019 Oct 4];319(5865):966–8. Available from: http://www.ncbi.nlm.nih.gov/pubmed/18276894
40. Larsen DP, Butler AC, Roediger III HL. Repeated testing improves long-term retention relative to repeated study: a randomised controlled trial. Med Educ [Internet]. 2009 Dec [cited 2019 Oct 4];43(12):1174–81. Available from: http://www.ncbi.nlm.nih.gov/pubmed/19930508
41. Binks S. Testing enhances learning: A review of the literature. J Prof Nurs [Internet]. 2018 May [cited 2019 Oct 5];34(3):205–10. Available from: http://www.ncbi.nlm.nih.gov/pubmed/29929801
42. Chan JCK, Meissner CA, Davis SD. Retrieval potentiates new learning: A theoretical and meta-analytic review. Psychol Bull [Internet]. 2018 Nov [cited 2019 Oct 5];144(11):1111–46. Available from: http://www.ncbi.nlm.nih.gov/pubmed/30265011
43. Carpenter SK. Cue strength as a moderator of the testing effect: The benefits of elaborative retrieval. J Exp Psychol Learn Mem Cogn [Internet]. 2009 Nov [cited 2019 Oct 3];35(6):1563–9. Available from: http://www.ncbi.nlm.nih.gov/pubmed/19857026
44. Carpenter SK, Delosh EL. Impoverished cue support enhances subsequent retention: Support for the elaborative retrieval explanation of the testing effect. Mem Cognit [Internet]. 2006 Mar [cited 2019 Oct 3];34(2):268–76. Available from: http://www.springerlink.com/index/10.3758/BF03193405
45. Cook DA, Thompson WG, Thomas KG, Thomas MR, Pankratz VS. Impact of Self-Assessment Questions and Learning Styles in Web-Based Learning: A Randomized, Controlled, Crossover Trial. Acad Med [Internet]. 2006 Mar [cited 2019 Oct 4];81(3):231–8. Available from: http://www.ncbi.nlm.nih.gov/pubmed/16501263
46. Kornell N, Hays MJ, Bjork RA. Unsuccessful retrieval attempts enhance subsequent learning. J Exp Psychol Learn Mem Cogn [Internet]. 2009 [cited 2019 Oct 3];35(4):989–98. Available from: http://doi.apa.org/getdoi.cfm?doi=10.1037/a0015729
47. Roediger HL, Butler AC. The critical role of retrieval practice in long-term retention. Trends Cogn Sci [Internet]. 2011 Jan [cited 2019 Oct 4];15(1):20–7. Available from: http://www.ncbi.nlm.nih.gov/pubmed/20951630
48. Bjork RA. Retrieval as a memory modifier. In: Solso R, editor. Information Processing and Cognition: The Loyola Symposium. Hillsdale, NJ: Lawrence Erlbaum Associates; 1975. p. 123–44.
49. Gardiner FM, Craik FIM, Bleasdale FA. Retrieval difficulty and subsequent recall. Mem Cognit [Internet]. 1973 Sep [cited 2019 Oct 4];1(3):213–6. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24214547
50. Cuddy LJ, Jacoby LL. When forgetting helps memory: an analysis of repetition effects. J Verbal Learning Verbal Behav [Internet]. 1982 Aug [cited 2019 Oct 5];21(4):451–67. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0022537182907277
51. Cepeda NJ, Pashler H, Vul E, Wixted JT, Rohrer D. Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychol Bull [Internet]. 2006 May [cited 2019 Oct 4];132(3):354–80. Available from: http://www.ncbi.nlm.nih.gov/pubmed/16719566
52. Rawson KA, Dunlosky J. Optimizing schedules of retrieval practice for durable and efficient learning: How much is enough? J Exp Psychol Gen [Internet]. 2011 [cited 2019 Oct 5];140(3):283–302. Available from: http://doi.apa.org/getdoi.cfm?doi=10.1037/a0023956
53. Karpicke JD, Bauernschmidt A. Spaced retrieval: Absolute spacing enhances learning regardless of relative spacing. J Exp Psychol Learn Mem Cogn [Internet]. 2011 Sep [cited 2019 Oct 5];37(5):1250–7. Available from: http://www.ncbi.nlm.nih.gov/pubmed/21574747
54. Cepeda NJ, Vul E, Rohrer D, Wixted JT, Pashler H. Spacing Effects in Learning. Psychol Sci [Internet]. 2008 Nov [cited 2019 Oct 4];19(11):1095–102. Available from: http://www.ncbi.nlm.nih.gov/pubmed/19076480
55. Schmidmaier R, Ebersbach R, Schiller M, Hege I, Holzer M, Fischer MR. Using electronic flashcards to promote learning in medical students: retesting versus restudying. Med Educ [Internet]. 2011 Nov [cited 2019 Oct 5];45(11):1101–10. Available from: http://www.ncbi.nlm.nih.gov/pubmed/21988625
56. Janiszewski C, Noel H, Sawyer AG. A Meta-analysis of the Spacing Effect in Verbal Learning: Implications for Research on Advertising Repetition and Consumer Memory. J Consum Res [Internet]. 2003 Jun 1 [cited 2019 Oct 4];30(1):138–49. Available from: https://academic.oup.com/jcr/article-lookup/doi/10.1086/374692
57. Moss V. The Efficacy of Massed Versus Distributed Practice As a Function of Desired Learning Outcomes and Grade Level of the Student [Internet]. Utah State University; 1995 [cited 2019 Oct 4]. Available from: https://digitalcommons.usu.edu/etd/3552
58. Donovan JJ, Radosevich DJ. A meta-analytic review of the distribution of practice effect: Now you see it, now you don’t. J Appl Psychol [Internet]. 1999 [cited 2019 Oct 4];84(5):795–805. Available from: http://doi.apa.org/getdoi.cfm?doi=10.1037/0021-9010.84.5.795
59. Lee TD, Genovese ED. Distribution of Practice in Motor Skill Acquisition: Learning and Performance Effects Reconsidered. Res Q Exerc Sport [Internet]. 1988 Dec [cited 2019 Oct 4];59(4):277–87. Available from: http://www.tandfonline.com/doi/abs/10.1080/02701367.1988.10609373
60. Glenberg AM. Component-levels theory of the effects of spacing of repetitions on recall and recognition. Mem Cognit [Internet]. 1979 Mar [cited 2019 Oct 4];7(2):95–112. Available from: http://www.springerlink.com/index/10.3758/BF03197590
61. Hintzman DL. Theoretical implications of the spacing effect. In: Solso RL, editor. Theories in cognitive psychology: The Loyola Symposium [Internet]. Oxford, England; 1974 [cited 2019 Oct 4]. Available from: https://psycnet.apa.org/record/1975-00291-001
62. Kramar EA, Babayan AH, Gavin CF, Cox CD, Jafari M, Gall CM, et al. Synaptic evidence for the efficacy of spaced learning. Proc Natl Acad Sci [Internet]. 2012 Mar 27 [cited 2019 Oct 4];109(13):5121–6. Available from: http://www.ncbi.nlm.nih.gov/pubmed/22411798
63. Kerfoot BP. Learning Benefits of On-Line Spaced Education Persist for 2 Years. J Urol [Internet]. 2009 Jun [cited 2019 Oct 5];181(6):2671–3. Available from: http://www.ncbi.nlm.nih.gov/pubmed/19375095
64. Kerfoot BP, Fu Y, Baker H, Connelly D, Ritchey ML, Genega EM. Online Spaced Education Generates Transfer and Improves Long-Term Retention of Diagnostic Skills: A Randomized Controlled Trial. J Am Coll Surg [Internet]. 2010 Sep [cited 2019 Oct 5];211(3):331-337.e1. Available from: http://www.ncbi.nlm.nih.gov/pubmed/20800189
65. Matos J, Petri CR, Mukamal KJ, Vanka A. Spaced education in medical residents: An electronic intervention to improve competency and retention of medical knowledge. Vrana KE, editor. PLoS One [Internet]. 2017 Jul 31 [cited 2019 Oct 5];12(7):e0181418. Available from: http://www.ncbi.nlm.nih.gov/pubmed/28759606
66. Nkenke E, Vairaktaris E, Bauersachs A, Eitner S, Budach A, Knipfer C, et al. Spaced education activates students in a theoretical radiological science course: a pilot study. BMC Med Educ [Internet]. 2012 Dec 23 [cited 2019 Oct 5];12(1):32. Available from: http://www.ncbi.nlm.nih.gov/pubmed/22621409
67. Andersen SAW, Konge L, Cayé-Thomasen P, Sørensen MS. Learning Curves of Virtual Mastoidectomy in Distributed and Massed Practice. JAMA Otolaryngol Neck Surg [Internet]. 2015 Sep 3 [cited 2019 Oct 5];141(10):1. Available from: http://www.ncbi.nlm.nih.gov/pubmed/26334610
68. Seabrook R, Brown GDA, Solity JE. Distributed and massed practice: from laboratory to classroom. Appl Cogn Psychol [Internet]. 2005 Jan [cited 2019 Oct 5];19(1):107–22. Available from: http://doi.wiley.com/10.1002/acp.1066
69. Huang S, Deshpande A, Yeo S-C, Lo JC, Chee MWL, Gooley JJ. Sleep Restriction Impairs Vocabulary Learning when Adolescents Cram for Exams: The Need for Sleep Study. Sleep [Internet]. 2016 Sep 1 [cited 2019 Oct 4];39(9):1681–90. Available from: http://www.ncbi.nlm.nih.gov/pubmed/27253768
70. Patel VL, Yoskowitz NA, Arocha JF. Towards effective evaluation and reform in medical education: a cognitive and learning sciences perspective. Adv Heal Sci Educ [Internet]. 2009 Dec 24 [cited 2019 Oct 7];14(5):791–812. Available from: http://www.ncbi.nlm.nih.gov/pubmed/18214707
71. Ebbinghaus H. Memory: A contribution to experimental psychology [Internet]. New York: Teachers College Press; 1913 [cited 2019 Oct 8]. Available from: http://content.apa.org/books/10011-000
72. Karpicke JD. Metacognitive control and strategy selection: Deciding to practice retrieval during learning. J Exp Psychol Gen [Internet]. 2009 [cited 2019 Oct 7];138(4):469–86. Available from: http://doi.apa.org/getdoi.cfm?doi=10.1037/a0017341
73. Yan VX, Clark CM, Bjork RA. Memory and metamemory considerations in the instruction of human beings revisited: Implications for optimizing online learning. In: Horvath JC, Lodge JM, Hattie J, editors. From the laboratory to the classroom: Translating science of learning for teachers [Internet]. New York, NY: Routledge/Taylor & Francis Group; 2017 [cited 2019 Oct 5]. p. 61–78. Available from: https://psycnet.apa.org/record/2016-21706-003
74. Agarwal PK, Karpicke JD, Kang SHK, Roediger HL, McDermott KB. Examining the testing effect with open- and closed-book tests. Appl Cogn Psychol [Internet]. 2008 Nov [cited 2019 Oct 7];22(7):861–76. Available from: http://doi.wiley.com/10.1002/acp.1391
75. Kornell N, Son LK. Learners’ choices and beliefs about self-testing. Memory [Internet]. 2009 Jul [cited 2019 Oct 3];17(5):493–501. Available from: http://www.ncbi.nlm.nih.gov/pubmed/19468957
76. Kornell N. Optimising learning using flashcards: Spacing is more effective than cramming. Appl Cogn Psychol [Internet]. 2009 Dec [cited 2019 Oct 5];23(9):1297–317. Available from: http://doi.wiley.com/10.1002/acp.1537
77. Tullis JG, Finley JR, Benjamin AS. Metacognition of the testing effect: Guiding learners to predict the benefits of retrieval. Mem Cognit [Internet]. 2013 Apr 14 [cited 2019 Oct 5];41(3):429–42. Available from: http://www.ncbi.nlm.nih.gov/pubmed/23242770
78. Karpicke JD, Butler AC, Roediger III HL. Metacognitive strategies in student learning: Do students practise retrieval when they study on their own? Memory [Internet]. 2009 Apr [cited 2019 Oct 7];17(4):471–9. Available from: http://www.ncbi.nlm.nih.gov/pubmed/19358016
79. Wissman KT, Rawson KA, Pyc MA. How and when do students use flashcards? Memory [Internet]. 2012 Aug [cited 2019 Oct 7];20(6):568–79. Available from: http://www.ncbi.nlm.nih.gov/pubmed/22671698
80. Kornell N, Bjork RA. The promise and perils of self-regulated study. Psychon Bull Rev [Internet]. 2007 Apr [cited 2019 Oct 7];14(2):219–24. Available from: http://www.springerlink.com/index/10.3758/BF03194055
81. Toppino TC, LaVan MH, Iaconelli RT. Metacognitive control in self-regulated learning: Conditions affecting the choice of restudying versus retrieval practice. Mem Cognit [Internet]. 2018 Oct 29 [cited 2019 Oct 8];46(7):1164–77. Available from: http://www.ncbi.nlm.nih.gov/pubmed/29845590
82. Deng F, Gluckstein JA, Larsen DP. Student-directed retrieval practice is a predictor of medical licensing examination performance. Perspect Med Educ [Internet]. 2015 Dec 23 [cited 2019 Oct 7];4(6):308–13. Available from: http://www.ncbi.nlm.nih.gov/pubmed/26498443
83. Ruiz JG, Mintzer MJ, Issenberg SB. Learning objects in medical education. Med Teach [Internet]. 2006 Jan 3 [cited 2019 Oct 10];28(7):599–605. Available from: http://www.ncbi.nlm.nih.gov/pubmed/17594550
84. D’Antoni A V., Mtui EP, Loukas M, Tubbs RS, Zipp GP, Dunlosky J. An evidence-based approach to learning clinical anatomy: A guide for medical students, educators, and administrators. Clin Anat [Internet]. 2019 Jan [cited 2019 Oct 7];32(1):156–63. Available from: http://www.ncbi.nlm.nih.gov/pubmed/30307063
85. Hart-Matyas M, Taylor A, Lee HJ, Maclean MA, Hui A, Macleod A. Twelve tips for medical students to establish a collaborative flashcard project. Med Teach [Internet]. 2019 May 4 [cited 2019 Oct 7];41(5):505–9. Available from: http://www.ncbi.nlm.nih.gov/pubmed/29378455
86. Al-Rawi W, Easterling L, Edwards PC. Development of a mobile device optimized cross platform-compatible oral pathology and radiology spaced repetition system for dental education. J Dent Educ [Internet]. 2015 Apr [cited 2019 Oct 7];79(4):439–47. Available from: http://www.ncbi.nlm.nih.gov/pubmed/25838016
87. Hudder A, Tackett S, Moscatello K. First-Year Experience Implementing an Adaptive Learning Platform for First- and Second-Year Medical Students at the Lake Erie College of Osteopathic Medicine. J Am Osteopath Assoc [Internet]. 2019 Jan 1 [cited 2019 Oct 7];119(1):51. Available from: http://www.ncbi.nlm.nih.gov/pubmed/30615042
88. Bow HC, Dattilo JR, Jonas AM, Lehmann CU. A Crowdsourcing Model for Creating Preclinical Medical Education Study Tools. Acad Med [Internet]. 2013 Jun [cited 2019 Oct 7];88(6):766–70. Available from: http://www.ncbi.nlm.nih.gov/pubmed/23619061
89. Taveira-Gomes T, Prado-Costa R, Severo M, Ferreira MA. Characterization of medical students recall of factual knowledge using learning objects and repeated testing in a novel e-learning system. BMC Med Educ [Internet]. 2015 Dec 24 [cited 2019 Oct 7];15(1):4. Available from: http://www.ncbi.nlm.nih.gov/pubmed/25616353
90. Taveira-Gomes T, Saffarzadeh A, Severo M, Guimarães MJ, Ferreira MA. A novel collaborative e-learning platform for medical students – ALERT STUDENT. BMC Med Educ [Internet]. 2014 Dec 14 [cited 2019 Oct 7];14(1):143. Available from: http://www.ncbi.nlm.nih.gov/pubmed/25017028
91. Kuperstock JE, Horný M, Platt MP. Mobile app technology is associated with improved otolaryngology resident in-service performance. Laryngoscope [Internet]. 2019 Jan [cited 2019 Oct 7];129(1):E15–20. Available from: http://www.ncbi.nlm.nih.gov/pubmed/30151970
92. Morin CE, Hostetter JM, Jeudy J, Kim WG, McCabe JA, Merrow AC, et al. Spaced radiology: encouraging durable memory using spaced testing in pediatric radiology. Pediatr Radiol [Internet]. 2019 Jul 16 [cited 2019 Oct 8];49(8):990–9. Available from: http://link.springer.com/10.1007/s00247-019-04415-3
93. Anki 2.1 User Manual [Internet]. 2019 [cited 2019 Oct 8]. Available from: https://apps.ankiweb.net/docs/manual.html
94. Wozniak P. Optimization of learning: a new approach and computer application. University of Technology in Poznan; 1990.
95. Bloom BS, Engelhart MD, Furst EJ, Kratwhol DR. Taxonomy of educational objectives: The classification of educational goals. New York, NY: Longman; 1956.
96. Knowles M. The adult learner: A neglected species. In Houston, TX: Gulf Publishing Company; 1973.
97. Knowles M, Holton EI, Swanson R. The adult learner: the definitive classic in adult education and human resource development. Burlington, MA: Elsevier; 2005.
98. Knowles M. The modern practice of adult education: From pedagogy to andragogy. Chicago, IL: Follett; 1980.
99. Kaufman DM, Mann KV. Teaching and Learning in Medical Education: How Theory can Inform Practice. In: Swanwick S, editor. Understanding Medical Education: Evidence, Theory and Practice. West Sussex, UK: Wiley-Blackwell; 2010. p. 7–30.
100. Samuels SJ. Attentional progress in reading: The effect of pictures on the acquisition of reading responses. J Educ Psychol. 1967;(58):337–342.
101. Landi N, Perfetti CA, Bolger DJ, Dunlap S, Foorman BR. The role of discourse context in developing word form representations: A paradoxical relation between reading and learning. J Exp Child Psychol [Internet]. 2006 Jun [cited 2019 Oct 9];94(2):114–33. Available from: http://www.ncbi.nlm.nih.gov/pubmed/16600285
102. Martin-Chang SL, Levy BA, O’Neil S. Word acquisition, retention, and transfer: Findings from contextual and isolated word training. J Exp Child Psychol [Internet]. 2007 Jan [cited 2019 Oct 9];96(1):37–56. Available from: http://www.ncbi.nlm.nih.gov/pubmed/17034812
103. Goodman KS. A Linguistic Study of Cues and Miscues in Reading. Elem English [Internet]. [cited 2019 Oct 10];42:639–43. Available from: https://www.jstor.org/stable/41387554
104. Dahl PR. An experimental program for teaching high speed word recognition and comprehension skills. In: Button JE, Lovitt TC, Rowland T., editors. Communications research in learning disabilities and mental retardation. Baltimore, MD: University Park Press; 1979. p. 33–65.
105. Archer N, Bryant P. Investigating the role of context in learning to read: A direct test of Goodman’s model. Br J Psychol [Internet]. 2001 Nov 1 [cited 2019 Oct 10];92(4):579–91. Available from: http://doi.wiley.com/10.1348/000712601162356
106. Stanovich KE. Toward an Interactive-Compensatory Model of Individual Differences in the Development of Reading Fluency. Read Res Q [Internet]. 1980 [cited 2019 Oct 10];16(1):32–71. Available from: https://eric.ed.gov/?id=EJ234111
107. Martin-Chang SL, Levy BA. Word reading fluency: A transfer appropriate processing account of fluency transfer. Read Writ [Internet]. 2006 Jul 29 [cited 2019 Oct 10];19(5):517–42. Available from: http://link.springer.com/10.1007/s11145-006-9007-0
108. Rajaram S, Srinivas K, Roediger HL. I. A transfer-appropriate processing account of context effects in word-fragment completion. J Exp Psychol Learn Mem Cogn [Internet]. 1998 [cited 2019 Oct 10];24(4):993–1004. Available from: http://doi.apa.org/getdoi.cfm?doi=10.1037/0278-73188.8.131.523
109. Smith MA, Karpicke JD. Retrieval practice with short-answer, multiple-choice, and hybrid tests. Memory [Internet]. 2014 Oct 3 [cited 2019 Oct 10];22(7):784–802. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24059563
110. Young JQ, Van Merrienboer J, Durning S, Ten Cate O. Cognitive Load Theory: Implications for medical education: AMEE Guide No. 86. Med Teach [Internet]. 2014 May 4 [cited 2019 Oct 10];36(5):371–84. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24593808
111. Bernard J-B, Anne-Catherine S, Eric C. Page mode reading with simulated scotomas: A modest effect of interline spacing on reading speed. Vision Res [Internet]. 2007 Dec 1 [cited 2019 Oct 10];47(28):3447–59. Available from: https://www.sciencedirect.com/science/article/pii/S0042698907004580
112. Chung STL. Reading speed benefits from increased vertical word spacing in normal peripheral vision. Optom Vis Sci [Internet]. 2004 Jul [cited 2019 Oct 10];81(7):525–35. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15252352
113. Kahneman D. Thinking, Fast and Slow. Toronto, ON: Anchor Canada; 2011.