Statistics help, tutorials and a shoulder to cry on…

Menu & Search

Scrivener, Zotero & Pandoc integration for Mac Users

Changing pace today, statfans. I tweeted about my winning workflow for academic writing involving Scrivener, Zotero and Pandoc, and a few people asked for a detailed explanation – so here it is.

Credit for this whole setup goes to Dave Smith at Nottingham University.  Dave’s how-to is perfectly clear if you’re a very tech savvy Scrivener user. If you’re a comfortable bash-scripter then go ahead and use Dave’s concise tutorial. The instructions below are aimed at everyone else.

Note: This tutorial is for Mac users only. If you’re a Microsoft user, I’m sorry to hear that, and wish you the best of luck on your Scrievner/Zotero journey 😉

Note: Within scrivener, you need to write in markdown for this to work. What does this mean? Instead of using scrivener to put in all of the rich-text formatting you want (italics, bold, underlined, headings, etc) you need to use markdown syntax to embed the formatting into plain text. You can still write using any font you like, but unless you use markdown to express your formatting needs you will end up with an unformatted mess at the other end. Do not panic, this is not a big deal. Mostly, you need to remember to put *one asterisk around italic text* and **two asterisks around bold text**. And specify your headings properly. Click here to learn the basics.

An added bonus of doing academic writing in markdown is that you can write tables in markdown, and don’t have to deal with Scrivener’s table functionality. This is almost worth doing even without the awesome reference integration I’m about to demonstrate.

Better BibTeX

First, we need to get Zotero to produce a unique identifier for all of the items in your Zotero library.  The reference also needs to be in a format which other programs can understand. Without an exportable unique reference for each library item there is no hope of encoding the references in Scrivener. The consensus from people who know more about this than me is that the best way to do this is via the Better BibTeX Zotero plugin.

To install the plugin,

  1.  visit this page and download the xpi file. Then open Zotero, and under the Tools menu, click “Add-Ons”.
  2. Click the little cog-wheel, and select “Install add-on from file…” and select the xpi file you just downloaded.

The two steps below are absolutely crucial, double check you’ve got this next part right or nothing is going to work properly. You want Zotero to pass all of its quick-copy citations through the BetterBibTeX plugin, and you want BetterBibTeX to output citation keys in a format which Pandoc can understand.  

To set this up, once BetterBibTeX installs, click “preferences” in the extension window and change the Quick Copy Format to “Pandoc”.

Now click “Export” at the top of this window (or navigate via Zotero > Preferences) and change Zotero’s default Quick Copy format to “Better BibTeX citation Key Quick Copy”. 

Now click the Better BibTex icon at the top of the preferences window and double check you’ve got “Pandoc” selected as the quick copy format.

Export a copy of your Zotero library

Now we need to export your whole Zotero library, into a format that Pandoc can understand. In Zotero, Navigate through File > Export Library…

Select the Better BibLatex format, and tick the keep-updated box. Save your library as something easy like My_Library.bib, somewhere you can easily find it (we’re going to move it later).

Accessing citekeys within Scrivener

This is where it gets difficult if you’re not a tech-y person. Hang in there!

Zotero is now ready to talk to Scrivener, and we have an updated library ready for Pandoc to use once we install it. In this step we’ll set up a simple method for grabbing the BetterBibTex citekeys from within Scrivener.

Dave Smith has written an application which makes this whole process incredibly easy.  In your Mac’s Applications folder is a “Utilities” folder. Go there and open Script Editor (or search for Script Editor using the Spotlight Search)

Copy the 50 lines of code from here and paste them into the script editor. Save the script as something like “zotpick-pandoc_Scrivener”, and select “Application” from the File Format drop-down menu before you save. Visit this very clear tutorial if you are confused. 

Now we need to point scrivener in the direction of the application. We will do so by setting it as our reference manager. Navigate through Scrivener > Preferences > Citations. Click “choose” and select the application you just saved a moment ago:

Once you’ve done this, when writing in Scrivener you can use the keyboard shortcut (⌘ – Y) to pull up a fabulous little red box which will grab citekeys for you!  Type the name of an author or keyword to search your library. Once you’ve selected the reference(s) you want, hit return and the applet will type the citekey within Scrivener, wherever your cursor is placed. This works in any word-processing program or text editor. Visit this page if you need help with the applet.

Now we need to setup Pandoc, which will be able to convert these weird citekeys to proper references.

Install Pandoc

Pandoc is an amazing program which can convert between a huge number of text file formats. It can also scan text files and replace BibTeX citekeys with correctly formatted references, which is why we’re going to use it.  Pandoc does not have a user interface, you use it via the command line, by writing computer code and not by clicking a mouse. Do not panic, Dave Smith has already written all the code you need. Thanks Dave!

Download and install the appropriate version of Pandoc for your machine (probably the MacOS.pkg one) current version at time of writing is here.

Hidden Files…

If you search for “Pandoc” using the Finder or Spotlight Search, you won’t find it on your machine. This is odd, because you just installed it! Pandoc lives in a hidden directory, and before we can ask Pandoc to scan our scrivener files and populate the references, we need to get into that hidden directory and put some things in.

We need to add:

  • Some .csl files which tell Pandoc how to format the references in various styles
  • The My_Library.bib file we created earlier
  • A reference.docx file (If your end goal is to export to MS Word)

Open a finder window and use ⌘+ shift + H to navigate to your home directory. Now use ⌘ + shift + . to reveal the hidden directories, and open .pandoc 

 

Edit: If you can’t see it, you may need to create the .pandoc folder. (You can also use use an existing folder)

Move My_Library.bib (from earlier) into this directory.

If like me you want to ultimately export into MS word because your co-authors use it, create a new word document and call it reference.docx. The contents of this document do not matter, Pandoc will use the Styles specified for this document to format all of your output. To change the defaults, open reference.docx in word, click “Styles Pane” and modify any or all of the following styles recommended by the Pandoc Manual:

[paragraph] Normal, Body Text, First Paragraph, Compact, Title, Subtitle, Author, Date, Abstract, Bibliography, Heading 1, Heading 2, Heading 3, Heading 4, Heading 5, Heading 6, Heading 7, Heading 8, Heading 9, Block Text, Footnote Text, Definition Term, Definition, Caption, Table Caption, Image Caption, Figure, Captioned Figure, TOC Heading; [character] Default Paragraph Font, Body Text Char, Verbatim Char, Footnote Reference, Hyperlink; [table] Table.

Once you’ve set all these styles in the way you want, save reference.docx and close Word.

Liftoff

Ok, let’s bake this turkey! Now all this hard work is going to pay off. I’ve prepared some text below, go ahead and put it into a new scrivener document:

Abstract 
============ 
Here is the abstract! The text above is a top-level header. 
Introduction 
---------------- 
Here is the introduction! The text above is a second-level header. 
Here is some text which requires a reference. 
Here is some text which requires multiple references. 
Here is some *italics*. Here is some **bold text**. Pandoc will populate the reference list at the end of the document, so here is a heading for that. 

*References* 
----------------

Within Scrivener, use the keyboard shortcut (⌘ – Y) to pull up the applet, and add some references from your library to this test document (if you lose the red box just hit F3 to find it). I’ve highlighted my citekeys in yellow, below:

 

Now export this document as a plain text file (.txt). Don’t put any spaces in the file name because you’ll need to point to it through the command line in the next step, call it “pandoc_test.txt” or something similar.

Now open a terminal window (Applications > Utilities > Terminal)

The code you need to run (thanks again to Dave Smith) is below. Copy the code below and paste it into a text editor like textEdit:

pandoc –bibliography \
~/.pandoc/My_Library.bib \
–csl ~/.pandoc/vancouver.csl \
-f markdown -t docx \
-o ~/Documents/Folder1/Folder2/Output.docx ~
/Documents/SomeFolder/SomeOtherFolder/pandoc_test 

Now edit all of the coloured bits. Type the name of the relevant csl style which corresponds to the kind of reference you want (note: you need to have copied the csl file in the hidden .pandoc folder for this to work!). Next adjust the file path so it points to the folder in which you’d like the final word document to appear. And finally, point pandoc to your text file you want it to add the references to.  

Once you think your code is ready, paste it all into the terminal and hit ‘Return’. Pandoc should think for a second, and if no error is returned your word document is ready! Close your terminal and go take a look! The result of my test is below. Here is the code I used:

And the final result is here. Notice that the heading style specified in reference.docx has over-ruled the formatting indicated by the asterisks in the “References” heading.

 

If you want to go even further, you can preview your fully-formatted document while you write using Brett Terpstra’s Marked 2. I haven’t explored this yet, but intend to in the future.

 

I hope you enjoy this workflow. And, if you don’t work with code often, I hope you enjoy feeling like a programming ninja whenever you run your pandoc script 🙂

Taya

Taya
Related article
Sampling Distribution & Central Limit Theorem

Sampling Distribution & Central Limit Theorem

Understanding sampling distributions is really important, but very few researchers…

9 Discussion to this post

  1. Bartosz Gradecki says:

    Thanks for the helpful and much needed post. I got stuck though at the point where you’re supposed to move the files to the hidden folder, for the folder isn’t there. Any ideas how to solve this issue?

    • Taya says:

      You’re very welcome! Did you use the keyboard shortcut (⌘ + shift + .) to reveal the hidden directories in your home folder? Can you see other hidden directories such as .cache and .config?

      • BG says:

        Yes, of course. It’s not there. Though .cache isn’t either – but .config is (together with .cups, .local, and a few others). I’ve done a system files search too, and there’s no such folder, though there is a couple of .pandoc files scattered here and there.

        • Taya says:

          Right. To confirm the directory doesn’t exist, try navigating to it in the terminal (cd ~/.pandoc) and then listing the contents (ls). If you can’t find the .pandoc directory you can skip this whole step, and modify the code at the end to point to some other directory which you know exists. As long as you point pandoc to the csl file, .bib file and reference docx you should be fine. In fact, this is probably easier than having to move things in and out of hidden directories… I hope that is helpful!

  2. BG says:

    After a few hours of tinkering I finally managed to make it work. You’re right, what matters is that files are in the same folder, not its name. Perhaps it’s worth pointing out that you’re actually giving two instructions in here: one on how to integrate Scrivener with Zotero through Pandoc, and the other one how to make citekeys into actual citations – which is valid regardless of where you’ve created your .txt file with citekeys. I’ve tinkered with Atom and it works great too (and integrating it with Zotero is even easier than in case of Scrivener). In any case, thanks a lot for your help!

  3. Sarah says:

    Hi Taya, thank you so much for this extremely clear explanation — I had been trying to follow Dave Smith’s instructions with zero success until I found your page!!

    I had the same problem as BG though. Stupidly I didn’t read down the comments until just now however I posted an item on the Pandoc discussion board yesterday (and got some very helpful responses, I highly recommend it for troubleshooting problems). John MacFarlane, the developer of Pandoc responded that you have to create the ~/.pandoc directory yourself, stating “As a matter of principle, pandoc never creates any file on your file system (except perhaps a temp file which it will delete later) unless you ask it to.”

    So you just go to your home directory (the base folder of your username), create a new folder and name it .pandoc

  4. Sarah says:

    A quick tip for anyone who needs to use in-text referencing, i.e. (Author, Year) rather than (1), I couldn’t work out why my references were all being processed by pandoc as Smith (2018) rather than (Smith, 2018). After pouring through the pandoc manual it turns out that for this to occur, square brackets need to be placed around references in Scrivener.

    Thus, my zotpicked entries need to look like “…blah blah blah [@AuthorTitleYear].”

  5. Sarah says:

    Ok one more tip — I found a blog post by Raphael Kabo on this workflow the other day which I couldn’t get to work, however after following your wonderfully clear instructions with success, I went back and implemented his strategy for using Automator to create a custom app to replace having to type terminal commands each time you want to compile a document.

    See: http://raphaelkabo.com/blog/posts/markdown-to-word

    In the first line of code I substituted his .md file extension for .txt and it works perfectly!!

Leave a Reply

Your email address will not be published. Required fields are marked *

Type your search keyword, and press enter to search