Opinionated Reference on Doing Research

Personal Experience

• Use all leverages:
  • LLM
    • Kickstart labor tasks: “Can you write a Python script to find the most similar images dir1 for each image in dir2 using L2?”
    • Brainstorm: “What are the best metrics for crafting a good food nutrition dataset? Can you give pseudocode for each?”
• Direction is more important than speed, but need both.

Experience at VLL

• Get more ideas from reading or walking.
• Capture ideas to any form of writing.
• Design experiments with clear purpose and hypothesis.
• Make good visualizations.
  • Bad visualizations hide findings.
  • Rearrange visualizations when comparing two things.
  • There are always more ways to visualize data.
• Always do anything that seems worth doing. Don’t be lazy.
  • It’s okay if it’s time-consuming the first time.
  • When it happens repeatedly, automate.
• Always be aware of the process and optimize the bottleneck.
  • Need to rearrange images for research update for the third time -> Ask ChatGPT for a Python script to automate it.
  • Difficult to run experiments on different machines -> Set up a container.
• Have multi-level perspectives, debug (smallest) to experiment design (higher); know when to zoom in and out.
• Fast feedback loop is very important.
  • Set up environments, tools as much as possible to eliminate brain blocks.
  • Ideas slip away while waiting to load a model.
• Introduction is more like an extended abstract.
  • Clear and concise.
  • Easy to understand.
  • Easy to see the contributions.

Examples

• Visualization
  • spatial; reorganize to be close (horizontal > vertical) > far apart
  • scale; normalize/standardize > different scales
• Note-taking
  • bad: note once and forget; because it is not good enough to be a reference -> make it YOUR own note.
  • good: always refine and update

John Schulman

• most solid, high-level advices¹ on doing ML research.
• Three rough forms of ultimate research goals
  • groundbreaking result that changed perspective on some problem
  • an algorithmic idea that’s reusable
  • a deep insight about some recurring questions
• research taste is important to be developed
• goal-driven to develop unique perspective; ask questions that nobody else is asking
• constrain search to solutions that seem general and can be applied to other problems, e.g., avoid incorporating domain information into the solution–achieve locomotion in simulation, in a general way that could be applied to other problems.
• use notebook
  • review every 1 or 2 weeks
  • read all daily entries
  • condense the information into a summary
  • Usually they contains sections for experimental findings, insights (which might come from yourself, colleagues, or things you read), code progress (what did you implement), and next steps / future work.
• switching problems too frequently (and giving up on promising ideas) is a more common failure mode than not switching enough
• one untested strategy is to devote some fixed time budget to trying out new ideas that diverge from main line of work.
  • one day per week on something totally different
  • This would constitute a kind of epsilon-greedy exploration, and it would also help to broaden knowledge.
• The main ways to build ML knowledge are to read textbooks, theses and papers; and to reimplement algorithms from these sources.
• in early career, it is recommended to split time about evenly between textbooks and papers.
  • choose a small set of relevant textbooks and theses to gradually work through
  • and you should reimplement the models and algorithms from your favorite papers.
  • A couple of John’s favorites were Numerical Optimization by Nocedal & Wright, and Elements of Information Theory by Cover & Thomas.
  • Recent theses are often the best place to find a literature review of an active field, but older theses also often contain valuable gems of insight.
  • Textbooks and theses are good for building up your foundational knowledge, but you’ll also need to read a lot of papers to bring your knowledge up to the frontier.

Graham Neubig

• Noted from “CMU Advanced NLP Fall 2024 (9): Experimental Design and Data Annotation” ^{2 3}.
• Rough research categories:
  • Applications-driven: make a better, useful system
  • Curiosity: understand something
  • The ratio from ACL is about 95% applications-driven, 5% curiosity.

Hypothesis

• “Yes-no” > “how to.”
• A good hypothesis is explicit, precise, and falsifiable.
  • Certain experiment result can validate or disprove the hypothesis.
  • “Does X make Y better?” is not precise.
  • “Do pre-trained embeddings help more when the size of the training data is small?”

Data annotation

• Statistically significant difference needs a certain amount of data.
• Given effect size and significance threshold, “Power analysis” can estimate the amount, e.g., effect size is the expected accuracy difference between tested models.

Workflow

• Directory can be used in experiment steps modularization: data/, model/, result/, log/, script/, note/
• Name directories by parameters, e.g., transformer-layer8-node512-dropout0.5-labelsmooth0.02
• Planning results section in advance helps identify unjustified experimental claims; create main table, use TBD placeholder
• Result reporting: generate paper latex directly from log files

Simon Peyton Jones

• A classic on how to write a paper. ⁴ (Graham: useful, timeless piece.)
• Seven simple, actionable suggestions

Don’t wait: write

• Act as a lazy evaluation.
• Forces to be clear, focused, and crystallises unknown.
• Makes it easy to share.
• Makes writing part of the research process.

Identify key idea

• don’t need to be the best idea, refine it.
• paper should have one clear, sharp idea and be explicit
  • “The main idea of this paper is …”
  • “In this section we present the main contributions of the paper.”
• review: “I believe the main idea is …”
• fool-proof the writing

Tell a story

• imagine explaining at a whiteboard to a friend
  • here’s a problem. it’s interesting and unsolved.
  • here’s my idea
  • it works. compare with others.
• out of 1000 readers on the title, only 3 readers will read the details.

Nail your contributions to the mast

• don’t waste time. state what’s interesting. don’t state the obvious.
  • bad: “computer programs often have bugs. it is very important to eliminate these bugs …”
  • good: “consider this program, which has an interesting bug … We will debug this.”
• should be refutable.
• bulleted list
• “We give the syntax and semantics … (Section 3)” > “rest of this paper is … Section 3 provides …”
• page one is very important.
  • is every section referred to from the first page?
  • evidence supports claims from page one

Related work: later

• it’s tiring since it’s very compressed.
• be generous to the competition. “in his inspiring paper […]… We develop his foundation …”
• acknowledge help from people, and also acknowledge weaknesses in our approach.
• provide value judgement, not just a list of references.

Put your readers first

• don’t send them to sleep or stupidity; if the idea is clever, readers will find it.
• explain with examples, and generalize
• get help, each reader can read for the first time once.

Listen to your readers

• explain desired feedback (e.g. “i got lost here” > “jarva is mis-spelt”)
• “could you help me ensure that I describe your work fairly?”
• treat every review like gold dust.
• read every criticism as a positive suggestion for something we could explain more clearly.
• fix the paper so that X is apparent even to the stupidest reader. > “you stupid person, I meant X”
• thank the reviewers for their time.

Short Opinions

1. How to publish a paper at CVPR ⁵ gives a reviewer perspective.
   • Given 70% rejection rate, reviewer’s job is to find ANY reason to reject the paper
   • “The Cockroach” is a bland paper that is hard to kill.
   • “The Puppy with 6 toes” is a delightful paper that is easy to kill.
2. The Craft of Writing Effectively ⁶ is radical, unconventional, but very useful
   • Write to change ideas, not just explain.
   • Write VALUES. Useless pieces won’t be read.
   • 50% of PhD time is used to know the readers in the field.
   • Problem + solution is better than background + thesis.
   • Learn the language code
   • Notice what construes values in the papers.
3. How to do Research At the MIT AI Lab ⁷ is outdated but contains timeless advices.
   • It paints the setting of good interactions within a lab.
   • Connection helps stay informed on the state-of-the-art.
   • Alan Lakien’s “How to Get Control of Your Time and Your Life” is a good book.
4. Writing Advice for Fledging Machine Learning Researchers⁸ and The Last Mile of Creating Publication-Ready Plots ⁹ - small details, solid advice on writing.

Recommended Resources in Order

1. An Opinionated Guide to ML Research ¹⁰ (#john-schulman)
2. How to Read a Paper ¹¹
3. How to write a great research paper ¹² (#simon-peyton-jones)

Last edited 4 hours, 43 minutes ago.