TwoFold: Glorified curly bubbly templates

⏱ 13 min

tl;dr; TwoFold (2✂︎f) is a small Node.js library and command line app that allows plain text files to behave like dynamic files. Use-cases include: auto updating documentation, text based dashboards, text virtual assistants, text spreadsheets, turn based games, etc.

If that sounds interesting, read ⬇️ below. In this article, I try to address the basic questions of: who, what, when, where, why, in what way.

How it all started

I am super lazy when I can afford to. If I knew about a tool that was doing what I’m trying to accomplish with TwoFold, I would have immediately used that, instead of spending so much time to implement and test a thing from scratch…

It all started at the end of 2017, but what I wanted to do back then was kinda fuzzy (like many projects are, in the beggining). And the idea has changed so many times that it’s hard to remember for sure which one was the original concept.

But these are the main reason why this project exists:

  1. The idea that excited me the most was to have an interactive text-based dashboard. I imagined you could see the status of some running apps, but also to interact with the dashboard by writing commands AND to see the effect of your commands, ALL in the same file!
    Initially this was supposed to be a part of another project: Spinal 🌀, but eventually I realized this idea makes more sense as a separate project.
    I mean, you can always have a command line application with a dashboard and interact with it, but the command line output is not persistent, you lose everything once you close the terminal. A dashboard as text is just easy to edit anywhere, easy to share and can be saved into Git to track the history of changes.

  2. I wanted to write structured information as text, or add some numbers in a table and once I save the file, the app immediately responds by collaborating with me in the same place, in the same file, by validating the information, or calculating some min, max, sum or statistics, like a Spreadsheet application.
    But it doesn’t have to be a table, it could be a list of contacts, or bookmarks, and a relation between them.
    It’s a very simple idea, but I searched for apps that are working with text like this and didn’t find anything.

  3. Then the previous ideas merged with much older needs and desires. I work with plain text files a lot, but I miss advanced features available in document editors, things like: autocomplete, check for spelling, or grammar errors. And also I don’t want to be forced to use a specific application like: Word, or Pages, or GDocs.

So, I searched and searched and searched for applications or libraries that allow all these things and I found a few, but just… somehow… I didn’t feel very excited about using them.

The closest idea is Ned Batchelder’s Cog. There’s also MosJS.
And of course, Jupyter Notebooks, where you can mix documentation and code and see your code and the result of the execution in the same place.

Initially I was thinking to have some code inside “windows” and you can see the code and a big “-----” line and the app would inject the result of the code execution. But as I started to think more about the implementation, I decided that it’s a better and easier for user experience to have re-usable tags, instead of writing code in “windows” all the time.

I thought: What if this app would be React for Text files ??
But instead of generating HTML, it would render pure text and write it on the disk ?? The most interesting effect is that the template could evolve with every cycle, every render, like a game of life

I started to imagine how I could use React/ XML-like tags to call different functionalities and build them together, like LEGO bricks.

I wrote the first lines of code mid 2018. I published version 0.1 in 🗓 September 2018.
This version was using regular expressions to parse the XML-like tags and it was very limited, because it couldn’t easily parse props/ attributes and it couldn’t parse deeply nested tags.

Then I didn’t write any code for a whole year, because I was terrified? or too lazy? to start implementing a custom parser… I checked a lot of Node.js parsing libraries, a few worth mentioning are: Parse5, HtmlParser2, HtmlJs-Parser. And Snapdragon and Nearley.js.

I started working on version 0.2 in 🗓 July 2019 and at the end of September, the new version, written from scratch, was finished. I didn’t use any parsing libraries, I wrote all the code myself.

I’m also working on an article to describe the amazing adventure of writing my first parser 😅 and when it’s done, I’ll add the link here.

What is TwoFold now

TwoFold (2✂︎f) is a small Node.js library and command line application that allows plain text files to behave like dynamic files.

This can done by writing React/ XML-like <whatever /> tags in your text files and calling 2fold CLI to convert all known tags into useful responses. 2fold CLI can watch your files for changes and colaborate with you in the same file and the same place in the file, so you end up with a “live text file”. A “reactive” static file.
The open tag <, close tag > and the stopper character / are customizabie, so the tags could also look like { whatever /}, or [ whatever !].

It can be used for auto-updating documentation, if you inject the tags into your README files. Auto-updating source code, if you inject the tags into your comments. A virtual assistent inside your personal journal, if you inject tags into your private journals, like I do.

Use-cases include: personal notes, dream diary, recording feelings, todo lists, text spreadsheet, documentation and science papers, writing code (because code is a plain-text file too).

The goal of TwoFold is to make working with plain text files more fun, productive, interactive and intelligent. As much as any other word/ office like application out there, even more… because it’s independent of the application and the operating system.
It’s designed to be easily customizable and hackable, with Javascript code, to give users the power to implement their own use-cases.

If you’re editing your file with Visual Studio Code, Atom editor, Sublime text (and others), you’ll see the changes instantly, because they automatically reload the text when the file changes.

Very important to understand: TwoFold provides only the framework and tools for implementing the use-cases mentioned above, but they might not be immediately available.
I’m adding more tags like: “cmd”, “httpGet”, “weather”, “moonPhase”, “zodiacSign”, etc, in the TwoFold-extras repo.

More tags and plugins can be written in Javascript. The NPM ecosystem is huge, more than 1.1 million packages.

Why text files

So, I keep mentioning plain text files. I’m not trying to sell you the advantages of using text files, it’s not the goal of this article, but I can briefly mention what it means NOT to use them:

  • vendor lock-in - when you use a proprietary file format like: .doc, .docx, .pages, .psd, etc, you are forced to continue using the same tool to view and edit your documents. Sometimes you are forced to use the same Operating System just because the documents you have can only be edited with tools that only work in Windows, or only in macOS.
  • silos - can be much worse than vendor lock-in, because the company hosting your data can shut-down the application, or can go bankrupt. Check the cyber cemetery of site deaths to see a huge list of examples (including: Google+, del.icio.us, Readability). And Google cemetery for Google-only products.

Keeping your data in simple text files allows you the freedom to use any operating system, have a lot of free tools to choose from and make them work together easily.
There’s a bit of R&D involved and sometimes you just can’t find tools that do exactly what you need, but I find that the advantages surpass the confort of using all-in-one applications.

This is nothing new, there’s a ton of GUI, or CLI applications that adhere to this principle. Worth mentioning:

  • Standard Notes (We believe in building software that lasts. The notes you write now should be there for you in a 100 years)
  • Notable (No Vendor Lock-In. Notes and attachments are simply stored on your disk, this is extremely portable and powerful)
  • jrnl (simple journal application for the command line. Journals are stored as human readable plain text files)
  • Taskwarrior (manage your TODO list from the command line. Uses human-readable text files for storage, imports and exports JSON, so your data is never held captive)
  • Plain text accounting (CLI reporting tool and a plain text data format and for efficient double-entry-style accounting)

Any file is a plain text file in fact (even binary files)… But if an app expects a very strict or specific structure in that file and the structure is lost (XML, JSON, etc), the application won’t be able to open the file anymore.
So the point of plain text files is to have enough structure to make sense, but as little as possible, because we are humans, not machines.

I use Markdown a lot, for everything, and the default config for TwoFold makes the tags look like HTML, which is awesome because the Markdown parsers ignore custom HTML tags, so they are totally invisible.

TwoFold (2✂︎f) is designed to work with any readable file like: *.txt, Markdown, AsciiDoc, reStructuredText, Creole, HTML, other templating libraries like Django, Liquid, Mustache, etc.
It could also work with Python/ Javascript/ Go-lang/ whatever programming language you use, probably it makes sense to call the tags as comments inside the code.
It WON’T work with binary files like: .doc, .pages, .xls, .numbers, .pdf, images, audio, or video. Running TwoFold on binary files MIGHT break them (with the default config), because media files contain XML-like tags Exif or IPTC.

Current state

In 🗓 November 2019, before publishing this article, I pushed versions 0.3, then 0.4 and 0.5.
The new versions don’t add any major features on top of v0.2, but they make TwoFold much more user friendly and is in a state that makes me feel more confident to make it public and talk about it 😇

Currrent features:

  • 2fold CLI allows to scan a file or many files in a folder matching a pattern, to see a list of tags that will be affected by a render
  • render a file or many files in a folder matching a pattern
  • watch a file or many files in a folder matching a pattern and render the files that are changed, or created
  • can parse different tag styles, instead of <whatever /> it could be {whatever !} or [whatever ?], etc.
  • can load extra Node.js functions from folders, and they will become available as tags.

Roadmap

TwoFold tags are the only way for a user to actually interact with the framework, they are the building blocks. If there aren’t enough useful tags available, the framework is totally useless for a user.

Below there’s a list of possible tags, the majority are NOT YET implemented, but they could be implemented more or less easily, using already existing Node.js libraries.
I will implement the tags that are useful for me, but I won’t have time to implement all of them.

  • list (and automatically refresh) all contribuitors from a Github, or Gitlab repository (useful for Readme files)
  • fetch info about a Github, or Gitlab repository: programming lang, number of commits, last version, last commit date, etc. (useful for awesome lists and documentation)
  • list all files from a folder (to keep a documentation always up to date with the actual file list)
  • fetch data from a database and return the result as JSON/ YAML/ whatever (to create dashboards)
  • fetch data from an API (this can be used in millions of ways by using APIs from rapidApi.com or programmableWeb.com) (already implemented ✅)
  • ping all the URLs from a text, to check if the links are not broken
  • fetch latest currency conversion rates ($USD to €EUR, whatever) if you’re into that
  • convert currencies 💰 (probably more useful with self destructing tags)
  • execute a command line app and use the output (for integrating with different command line apps: wc, grep, ps, etc) (already implemented ✅)
  • run Javascript tests and output the result
  • read a text file and write parts of it, or all the file (already implemented ✅)
  • random shuffle some values, separated by space
  • math calculations for the current line
  • convert km to=miles, pounds to=kilograms, etc
  • show your public IP (in case you’re in a Coffee place)
  • show your location City, Country, GPS coords (in case you’re blogging while travelling)
  • show weather in your location, or some specified location (already implemented ✅)
  • show an emoji clock (useful for private notes, so you don’t have to type the full time)
  • show the moon phase (in case you’re a werewolf, vampire, or witch) (already implemented ✅ because I’m born in Romania)
  • show the current zodiac sign (already implemented ✅)
  • synonyms and antonyms for words
  • show text stats (sentences, words, characters, readability score, basic NLP)
  • show text spelling errors, and suggestions for improving the grammar
  • read section out loud (using Say, Speak, whatever)
  • keep lines of text sorted alphabetically (for awesome lists) (already implemented ✅)
  • calculate number of calories for some foods 🥘 (for writing about recipes)
  • turn based games 👾 (cards, chess, interactive quests, game-of-life)

Why this article

Why I’m writing this article now: because TwoFold (2✂︎f) is stable and usable enough to be used by other developers.
It’s not quite ready for end-users because there are not enough tags available yet…

And this is also a call to action to you, the reader. If you’re excited about the idea, write your thoughts in the Github repo, or on /r/ShinyTrinkets.
You can share this article on Mastodon, or Twitter.

Hope to hear from you soon!

PS: It took more than a month to write this article. Writing english is much harder than writing code 😓…

🏷 @articles #software #programming #trinkets