April 2, 2016

Testing: How much could I learn on Rosalind.info?

Hello, internet!

If you're curious what I get up to with my time, observe! I ran a little experiment today. This post reports it in a hodgepodge mix of "personal notes" and "terse scientific report".

If you take thirty seconds out of your day to comment, I will owe you one. Please encourage the norm of testing everyday ideas empirically; Be my miniature peer review.

How much would rosalind.info teach me?

I want to know the holes in my knowledge. If Rosalind actually helps, I want to train my internal motivation to prioritize working on it.

Hypothesis:

I currently can't do ~70% of problems. My learning trajectory looks unlikely to increase my capability beyond ~%50.

Method:

Experiment as proposed Mar 28 - count how many rosalind.info project list I can't do and how many I likely won't learn

Resources:

Rosalind Problems Lists
glimpses at other references for algorithmic and data structure fundamentals

Results:

Apr 2: I ignored Biology Content Knowledge, except where the lack of deep understanding seriously impedes my ability to program the solution.

PYTHON VILLAGE 1:0:0
  • 6 understand
BIOINFORMATICS STRONGHOLD 1:1:5
  • 13 understand
  • 14 don't understand yet
  • 78 doubt I'll cover
BIOINFORMATICS ARMORY 2:1:4
  • 4 understand
  • 2 don't understand yet
  • 9 doubt I'll cover
BIOINFORMATICS TEXTBOOK TRACK 1:2:4
  • 16 understand
  • 39 don't understand yet
  • 70 doubt I'll cover
ALGORITHMIC HEIGHTS 3:4:1
  • 12 understand
  • 18 don't understand yet
  • 4 doubt I'll cover
TOTALS 2:3:6
  • 51 understand
  • 73 don't understand yet
  • 161 doubt I'll cover

Conclusions:

Technically speaking, Rosalind wouldn't teach me anything. It has practice problems and a minor amount of scaffolding; designing the solution remains up to me.

Rosalind can sharpen my current skill set some, and do a little more to stretch my limits (mostly in graphs). The vast majority of problems deal in bioinformatics-specific knowledge. If I want to focus on that, the "Bioinformatics Armory" set works would shore up relevant foundations while not wasting time on general skills. This accords with its stated purpose.

Python Village and Algorithmic heights make okay sources for general practice.

Sanity-checks:


  • talk to an actual bioinformatics researcher, ask what they do and what tools they're using
  • try this experiment on an actual programming practice book, or a fundamentals checklist for comparison
  • google search for "waste of time rosalind.info" and "helped me rosalind.info", see if I can get some user experiences for how much it has helped other people
  • what does ONET say about the career outlook if I did decide to work in bioinformatics?
  • spend 5 minutes thinking about ways I could leverage bioinformatics domain knowledge in my other goals

Followups:


  • Trawl the Rosalind problems for Anki cards
  • Throw the glossary into Anki
  • Share this feedback with rosalind.info

Checklist for a Good Anki Card


This is my personal Checklist for making a Good Anki Card. Refer to as it helps, ignore it as it doesn't.

Better version here.

TLDR:

☐ I know why I want to make this card
☐ I know what I need to know for this card
☐ I know how to recognize when/where I need this
☐ There are no almost or partially right answers to this card
☐ There are no technically correct answers which comically miss the point
☐ There is no way to cheat to the right answer
☐ I cannot get marked wrong for being smarter than the question writer

DO NOT MAKE A CARD IF YOU DO NOT FULLY UNDERSTAND THE INFORMATION INVOLVED.

I know why I want to make this card

Always ask yourself why you want to make a card. Ask yourself before you do anything else. Ask so before you start on a chunk of cards and ask before you make each card. The answer to this question tells you what information you need.
  • If you want to remember things, you have to know what exactly you're remembering. 
  • If you want vocabulary, you have to understand what the words mean and when you'd use them 
  • If you want to know things, you have to understand what it is you know. 
  • If you want to do things, you have to understand how and when to do them 
  • If you want to follow advice and principles, you have to understand what you're following, what to do, and when it's relevant.
Do not make cards if you do not totally understand the information involved. Really, don't. I put it first on the checklist because it matters the most.

If you want to, you can justify memorizing anything on the basis that you totally understand how to turn "This Question" into "This Answer". You may feel tempted to throw a bunch of information into your deck and hope that some of it will stick. Most stuff added this way will waste your review time and dead-end your actual learning.

Anki can indeed help you can bootstrap your way up to big ideas very quickly, if you take the time to solidify the foundation of knowledge you're building on top of. The throw-it-all-in-and-hope-something-sticks method does a poor job of prioritizing the right things to reinforce, leaving you with a treacherously swiss-holed foundation.

I know what I need to know

When you have a particular piece of information you want to include, ask yourself what exactly you need to know. This tells you what kind of note template to use:
  • Question & Answer card: a certain response for a specific situation
  • Reversible card: two ideas that lead directly to each other.
    • Commonly used for an idea and its representation, like vocabulary or abbreviations.
  • Cloze deletion: a big hunk of information that needs to be understood in context of itself
    • Fore example, remembering the specific wording of a phrase. 
  • Many examples, one answer: a principle that spreads across many domains or has many different ways of instantiating itself
  • Generated cards: a skill to train. Anki doesn't have much of this built in, but you can cheat a bit with templates or excel. Hacks exist for making truely dynamic cards.

I know how to recognize when/where I need this

Ask yourself when the information becomes relevant and how you'd know. This will tell you what needs to go on the front of the card.

Invoke signs and senses and emotions, hint at what context you need to remember this in. Wording the question right will help you find the quickest and most reliable mental pathway between noticing a need for more information and remembering the exact information you need. If there's a lot of situations in which you need to remember the information, you may need to make more cards.

You may feel tempted to shortcut this process by declaring the obvious answer—"when I see the card question!" and just try to come up with a question that the information answers. That won't help you if the card question looks nothing like the specific moment when that information should pop up in your head.

I cannot get an almost or partially right answer to this card.

Ask yourself if you notice any answers that could conceivably get half-credit for. This will tell you if you need to split an idea into more cards.

Anki and other spaced repetition software wins out over good old fashioned flash cards because it uses smart scheduling; if you review all the information at the same interval anyways because you crammed it all into a single card, you lose that benefit. Long answers also take longer to learn, longer bring to mind fully, and longer to judge. You have to remember them as a group, so improvement comes all-or-nothing. You might remember pieces in a different order each time too, never practicing one mental path often enough to improve substantially at it.

You may feel tempted to throw a few glossaries into a deck. However, definitions actually cover a lot of information! Rather than try to judge whether you covered *enough* of the definition every time you answer the card, split it up. Take a step back, ask yourself when you need each part of the definition, and make cards for each case.

You may also make the mistake of writing questions that require you to list all the members of a collection, such as the names of your cousin's kids. You would get better practice out if you split it into things like:
  • recognize a name as one of your cousin's kids' names
  • given an incomplete list of the names, remember the ones blanked out
  • know how many kids your cousin has total.
  • recall a name to start from (youngest or oldest probably)

I cannot get a technically correctly answer that completely misses the point of this card

Ask yourself how a determined idiot or smartass will completely miss the point of your card. This will tell you if you need to work in details and get vividly descriptive.

People make the most egregiously technically-correct-but-useless-card-answers in response to Skill cards. You can make a card asking what to do in an emergency situation, and could easily get the answer "right" without having any idea how to put the answer into practice. To avoid this, you need to use vivid details—mental narrative, personal examples, pictures even—that get your imagination running wild and *put* you, mentally, into the situation where you need a real and useful answer.

It happens more subtly in vocabulary cards. You may find that you could define a term at multiple levels of detail, that the same word takes on different meanings in different contexts, or that the same idea goes by many different names. What answer are you looking for? How do ensure that *exactly that answer* is the first thing that pops into mind?

I cannot get a right answer for stupid, unrelated reasons

Ask yourself if it's possible to construct the right answer for the wrong reason. This will tell you if you need to make things more simple and consistent.

The way you word the question matters. You can give leading questions, full of hints that nearly obviate the answer. If you phrase all of your "No" questions the same way, you might start giving "No" answers based on the phrasing instead of thinking through the card's prompt. If you put a whole bunch of extra information in the question, you might start associating the answers with those details instead of the central point.

So think carefully when you add related cards: add both positive and negative examples, keep a relatively consistent question style for both, and put extraneous details in their own cards.

I cannot get the wrong answer for being smarter or more knowledgeable than the question writer

Ask yourself how what you learn now could interfere with what you'd learn later. This will tell you if you need to put in more precision and distinctions.

When you learn about similar things you often confuse them. Two cards with exceedingly similar questions and differing answers will screw up your practice of both; you need to either pick one answer to be the "proper" response or rewrite the questions to make the distinction clear.

Information can also get out of date, and or conflict with other sources sources. Where you know this is likely to happen, you need for the cards to be either more precise (not "this is true", but "X said this", or "This as of 2016") or for there to be an easy way to find and update them all quickly (Tags, Decks, Card types).

Also, in case I wasn't clear before,

DO NOT MAKE A CARD IF YOU DO NOT FULLY UNDERSTAND THE INFORMATION INVOLVED.

Thus concludes my checklist. I'll go into more detail and provide resources in later posts.