We welcome papers presenting surprising empirical results that do not match a priori intuition nor theoretical expectations, or that highlight failure modes of current approaches. Code sharing is strongly encouraged. Submissions will concentrate on anomalies in:

  • Data: idiosyncratic aspects of a dataset that affect the outcome in an unexpected way
  • Modeling: simplistic or misspecified model assumptions
  • Inference: challenging situations where inference algorithms fail
  • Validation: unsatisfactory evaluation metrics.

Two tracks will be considered:

“I Can’t Believe It’s Not Better!” track: Papers in this track show unexpected failure modes of new approaches. Although it is preferable that papers explain why the approach performs poorly, this is not essential if the paper demonstrates why the negative result is of interest to the community. Papers in this track should clearly describe the analysis, include a self-critique of the method, and show rigorous results. Authors are encouraged to provide ablation studies to isolate the cause of a method’s performance.

“Questioning default practices” track: Papers in this track highlight undesired effects or anomalous behaviors in practices that are widespread or taken for granted in the community. Some examples are questionable performance metrics, historic baseline methods, or datasets that are unsuitable in certain settings. This also includes flawed intuitions or unreasonable assumptions that are commonly made. Ideally, papers should suggest potential fixes or research directions to address such bad practices.

If you think your paper falls within the remit of the workshop but does not clearly fit one of these tracks, please contact the organizers at cant.believe.it.is.not.better@gmail.com

Formatting instructions

Important note: please use the latest Latex template in this webpage. For submission, it is fine if you used the NeurIPS template directly, but please note that you should switch to the Latex style files below for the camera-ready version.

We will follow the same formatting instructions as in the main NeurIPS conference. In short, all submissions must be in PDF format and at most 8 pages long (shorter is fine and even better, we encourage authors to be concise); the appendix can have unlimited length. Submissions will be double-blind, so you should submit an anonymous preprint.

You must format your submission using the provided LaTeX style files.

The maximum file size for submissions is 50MB. Submissions that violate the NeurIPS style (e.g., by decreasing margins or font sizes) or page limits may be rejected without further review.

How to submit

Submissions to the workshop will be handled through our OpenReview site.

Main deadline: October 14th October 16th 23:59 Anywhere on Earth. Accept/reject notification will be sent out by October 31st.

Reviewing criteria

We plan to create an open-minded and diverse space for promising works regardless of the final outcome by providing reviewers with concise guidelines that reward thorough understanding, transparency, and reproducibility rather than high performance. Reviewers will nominate papers with exemplary scientific rigor for publication in PMLR. Papers should clearly convey the interest of the unexpected behavior and demonstrate a high-quality research process.

We will be selecting for submissions that provide interesting insights on unexpected results. Morevover, we expect all submissions to follow the guidelines of basic courtesy and respect.

Submissions will be evaluated on the following criteria:

  • Clarity of writing
  • Rigor and transparency in the scientific process
  • Vulnerability and honesty in discussion, particularly if the submission is by the original author
  • Quality of discussion of limitations
  • Significance of new insights