cmpxchg8b

I miss the uMatrix Chrome extension.

Tavis Ormandy

Replacing uMatrix.

Tavis Ormandy

$Id: f0cc50a6fc6a5dd652c2b96ca9c1779f763fd6b1 $

Introduction

There used to be a fantastic chrome extension called uMatrix, written by Raymond Hill, the uBlock Origin developer. uMatrix was an intuitive way to control site permissions and subresource requests.

It looked like this:

This isn’t something you’ll care about unless you’re a nerd – but it let you limit what third parties could serve subresource requests, and controlled access to features like frames, scripts, video, fonts and so on. You can do some of that manually with browser settings, but uMatrix made it quick and easy. This meant that what would otherwise be quite a laborious and fiddly way to browse the web became simple. Okay, not simple, but simpler.

Anyway, I enjoyed all the extra knobs to control website permissions.

The features in uMatrix were – more or less – a subset of the features available in uBlock Origin, so rather than maintain both, Hill deprecated uMatrix and users were encouraged to migrate to uBO.

This actually worked okay for me – uBO added some features I quite liked (e.g. cosmetic filters). and if you didn’t mind writing the more complex rules by hand, it could basically do everything uMatrix could do.

The bad news is that uBO is a legacy MV2 extension, and the successor – uBO Lite – removed this feature.

I really don’t want to give that up – is there a solution?

Options

My question is could uMatrix even be possible under mv3?

declarativeNetRequest

The main difference between mv2 and mv3 here is that it is no longer possible to block requests while you execute a callback, a so-called “blocking” web request.

If you want to intercept a request, it now has to be done declaratively. That just means instead of running some javascript on every single request, you have to declare what actions you want applied in advance.

Here is the controversial bit you’ve probably read about: You can’t do everything that was possible in a callback declaratively, so the capabilities have been nerfed. That is technically true, but practically the rules are flexible enough for everything I would ever want.

Policies

The web already has a system to control features and subresources, it’s called Content Security Policy.

The obvious design is to use declarativeNetRequest to add our own Content-Security-Policy, then the browser will do all the work for us. All we need to do is provide a convenient interface to manage it.

Okay, but how will we offer a list of subresources to allow and deny? When you used uMatrix, it would show you a list of subresource types and origins, and let you manually approve or deny them.

This was the core thing uMatrix did well, so it’s essential. Well, I think CSP already provides a solution! There is a report-to directive that instructs the browser to post any security violations to a reporting endpoint.

All we have to do is enable that feature, add another declarativeNetRequest rule to catch those reports, then populate a list from the reports the browser sent. That’s another task successfuly offloaded 😂

Of course, this will take some plumbing, but the pieces are all there and it seems completely viable to me.

Outline

So now that we have a rough outline for how it could work… someone just has to write a proof of concept. Ah fine, I’ll give it a shot.

Here is a first attempt, matrix³.

It is clearly in a prototype state, but it basically works. I thought I’d post about this on my blog and see if there’s any interest!

The code is minimal – no building or dependencies required, I didn’t use any frameworks. You can just git clone and then “Load Unpacked” in Chrome.

I am not a web developer, and I clearly have zero UX skills 🦎 Please let me know if this is a bad idea, you know a better solution, or you want to help!

Hey… quick question, why are anime catgirls blocking access to the Linux kernel?

Tavis Ormandy

Anubis.

Tavis Ormandy

$Id: f0cc50a6fc6a5dd652c2b96ca9c1779f763fd6b1 $

Hey… quick question, why are anime catgirls blocking my access to the Linux kernel?

Intro

I’ve started running into more sites recently that deploy Anubis, a sort of hybrid art project slash network countermeasure. The project “weighs the souls” of HTTP requests to help protect the web from AI crawlers.

If you’ve seen anime catgirl avatars when visiting a new website, that’s Anubis.

I’m sympathetic to the cause – I host this blog on a single core 128MB VPS, I can tell you some stories about aggressive crawlers!

Anubis recently started blocking how I access git.kernel.org and lore.kernel.org. Those sites host the Linux Kernel Mailing List archive and the kernel git repositories. As far as I know I do have a soul, I just wasn’t using a desktop browser… so how exactly is my soul being weighed?

Note: Linux has Tux 🐧, OpenBSD has Puffy 🐡, SuSE has Geeko 🦎 and Microsoft has Bob 🤓… nothing wrong with mascots! 😸

Problem

The traditional solution to blocking nuisance crawlers is to use a combination of rate limiting and CAPTCHAs. The CAPTCHA forces vistors to solve a problem designed to be very difficult for computers but trivial for humans. This isn’t perfect of course, we can debate the accessibility tradeoffs and weaknesses, but conceptually the idea makes some sense.

Anubis – confusingly – inverts this idea. It insists visitors solve a problem trivial for computers, but impossible for humans. Visitors are asked to brute force a value that when appended to a challenge string, causes its SHA-256 to begin with a few zero nibbles.

    calcString := fmt.Sprintf("%s%d", challenge, nonce)
    calculated := internal.SHA256sum(calcString)

    if subtle.ConstantTimeCompare([]byte(response), []byte(calculated)) != 1 {
        // ...
    }

    // compare the leading zeroes
    if !strings.HasPrefix(response, strings.Repeat("0", rule.Challenge.Difficulty)) {
        // ...
    }

source: lib/challenge/proofofwork/proofofwork.go#L66

If that sounds familiar, it’s because it’s similar to how bitcoin mining works. Anubis is not literally mining cryptocurrency, but it is similar in concept to other projects that do exactly that, perhaps most famously Coinhive and JSECoin.

So how do some useless SHA-256 operations prove you’re not a bot? The argument goes that this simply makes it too expensive to crawl your website.

This… makes no sense to me. Almost by definition, an AI vendor will have a datacenter full of compute capacity. It feels like this solution has the problem backwards, effectively only limiting access to those without resources or trying to conserve them.

Numbers

Let’s assume the argument has some merit and math out the claims.

We can see that with the default Anubis configuration, a typical website visitor will have to solve a challenge with a difficulty of 4.

// DefaultDifficulty is the default "difficulty" (number of leading zeroes)
// that must be met by the client in order to pass the challenge.
const DefaultDifficulty = 4

source: anubis.go

This means that a visitor must make the first 4 hex digits of the challenge hash be zero, so 16 bits (4 digits, one nibble each). Therefore, you can expect to mine a suitable nonce within 2^16 SHA-256 operations.

If every single github star on the anubis project represents a website that has deployed Anubis, how much would the cloud services bill be to mine enough tokens to crawl every single website?

// CookieDefaultExpirationTime is the amount of time before the cookie/JWT expires.
const CookieDefaultExpirationTime = 7 * 24 * time.Hour

source: https://github.com/TecharoHQ/anubis/blob/main/anubis.go#L32

At the time of writing, Anubis has 11,508 github stars.

The default configuration means mining one token gets you access for 7 days (although I think this expiration check is broken, see below), so we need 11,508 * 2^16 SHA-256 operations per week, how expensive is that?

To get some numbers, I started an e2-micro vm on Google Compute Engine, and ran openssl speed. This is what you get in the free tier.

$ openssl speed sha256
Doing sha256 for 3s on 16 size blocks: 6915549 sha256's in 3.00s
Doing sha256 for 3s on 64 size blocks: 4631718 sha256's in 3.00s
Doing sha256 for 3s on 256 size blocks: 393694 sha256's in 3.21s
Doing sha256 for 3s on 1024 size blocks: 100123 sha256's in 3.00s
Doing sha256 for 3s on 8192 size blocks: 13300 sha256's in 2.98s
Doing sha256 for 3s on 16384 size blocks: 7137 sha256's in 2.99s
version: 3.0.17
built on: Tue Aug  5 07:09:41 2025 UTC
options: bn(64,64)
compiler: gcc -fPIC -pthread -m64 ...
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
sha256           36882.93k    98809.98k    31397.40k    34175.32k    36561.61k    39107.90k

It looks like we can test about 2^21 every second, perhaps a bit more if we used both SMT sibling cores. This amount of compute is simply too cheap to even be worth billing for.

So (11508 websites * 2^16 sha256 operations) / 2^21, that’s about 6 minutes to mine enough tokens for every single Anubis deployment in the world. That means the cost of unrestricted crawler access to the internet for a week is approximately $0.

In fact, I don’t think we reach a single cent per month in compute costs until several million sites have deployed Anubis.

I’m just not convinced this math works… this is literally nothing for a souless AI vendor with a monthly cloud services budget in the 8 figures. However, the cost for real soul-owning humans with limited access to compute is high – the Anubis forums are full of complaints like these:

Alternatives

Anubis cites hashcash as the primary inspiration for their design, an anti-spam solution from the 90s that was never widely adopted.

The idea of “weighing souls” reminded me of another anti-spam solution from the 90s… believe it or not, there was once a company that used poetry to block spam!

Habeas would license short haikus to companies to embed in email headers. They would then aggressively sue anyone who reproduced their poetry without a license. The idea was you can safely deliver any email with their header, because it was too legally risky to use it in spam.

Here’s a sample haiku:

winter into spring
brightly anticipated
like Habeas SWE (tm)

Was this a good idea? I don’t know, but they really did sue a few spammers!

Workarounds

So you’re trying to read LKML, but catgirl says no… is there a solution?

My issue is I don’t want to use a desktop browser to mine the required value, so how can I get the auth cookie?

If we look at the response with curl, we can see the challenge in the HTTP headers:

$ curl -I https://lore.kernel.org/
HTTP/2 200
server: nginx
set-cookie: techaro.lol-anubis-auth=; Path=/
set-cookie: techaro.lol-anubis-cookie-test-if-you-block-this-anubis-wont-work=5d737f0600ff2dd; Path=/

That techaro.lol-anubis-cookie is the challenge, here is a quick C program to mine an acceptable token:

#include <stdio.h>
#include <string.h>
#include <openssl/sha.h>

#pragma GCC diagnostic ignored "-Wdeprecated-declarations"

int main(int argc, char **argv) {
    unsigned char hash[SHA256_DIGEST_LENGTH];
    int difficulty = 4;
    char *message;
    SHA256_CTX base;

    if (argc < 2) {
        fprintf(stderr, "usage: %s challenge\n", *argv);
        return 1;
    }

    message = argv[1];
    SHA256_Init(&base);
    SHA256_Update(&base, message, strlen(message));

    for (int i = 0; i < 1 << 18; i++) {
        char nonce[16];

        SHA256_CTX ctx = base;
        SHA256_Update(&ctx, nonce, snprintf(nonce, sizeof(nonce), "%d", i));
        SHA256_Final(hash, &ctx);

        for (int j = 0; j < SHA256_DIGEST_LENGTH; j++) {
            if (hash[j] != 0)
                break;
            if (j < (difficulty >> 1) - 1)
                continue;

            // Looks good
            puts(nonce);
            return 0;
        }
    }
    return 1;
}

Let’s run that and see what it says…

$ gcc -Ofast -march=native anubis-miner.c -lcrypto -o anubis-miner
$ time ./anubis-miner 5d737f0600ff2dd
47224

real    0m0.017s
user    0m0.016s
sys     0m0.000s

Looks okay, let’s verify that solution is correct:

$ printf "5d737f0600ff2dd%d" 47224 | sha256sum
000043f7c4392a781a04419a7cb503089ebcf3164e2b1d4258b3e6c15b8b07f1  -

It seems valid, so now we can get a signed auth cookie by sending back the value we mined:

$ curl -I --cookie "techaro.lol-anubis-cookie-test-if-you-block-this-anubis-wont-work=5d737f0600ff2dd" \
    'https://lore.kernel.org/.within.website/x/cmd/anubis/api/pass-challenge?response=000043f7c4392a781a04419a7
cb503089ebcf3164e2b1d4258b3e6c15b8b07f1&nonce=47224&redir=/&elapsedTime=0'
HTTP/2 302
server: nginx
location: /
set-cookie: techaro.lol-anubis-auth=eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJhY3Rpb24iO...OTYifQ...;

Success, this cookie is now valid for 1 week of access. Let’s validate what it sent us, the actual schema is visible in the code:

    claims["iat"] = time.Now().Unix()
    claims["nbf"] = time.Now().Add(-1 * time.Minute).Unix()
    claims["exp"] = time.Now().Add(s.opts.CookieExpiration).Unix()

source: lib/http.go

We can examine this auth token and see what Anubis gave us…

$ base64 -d <<< eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9 | jq
{
  "alg": "EdDSA",
  "typ": "JWT"
}
$ base64 -d <<< eyJhY3Rpb24iO...OTYifQ== | jq
{
  "action": "CHALLENGE",
  "challenge": "5d737f0600ff2dd",
  "exp": 1756185722,
  "iat": 1755580922,
  "method": "fast",
  "nbf": 1755580862,
  "policyRule": "dbf942088788cc96"
}

It looks like exp is the expiry date, so 1756185722, which is…

$ date --date @1756185722
Mon Aug 25 22:22:02 PDT 2025

Yep, about 7 days from the date I requested it. You can now place that into a cookie file for curl,lynx etc.

Interestingly, sending the same request the next day got me a new signed cookie!?

This seems like a bug – exchanging a mined token for an auth cookie should immediately remove the challenge from the store, or there is a double spend vulnerability.

This error benefits me, I have to mine less tokens, but I’ll open an issue 😇

Update: wow, fixed just a few minutes after opening an issue by the maintainer!

This dance to get access is just a minor annoyance for me, but I question how it proves I’m not a bot. These steps can be trivially and cheaply automated.

I think the end result is just an internet resource I need is a little harder to access, and we have to waste a small amount of energy.

Notes

I wrote this article with my own puny brain, I didn’t use any AI. I know there are endashes (they’re not emdashes!) – I have a habit of writing two consecutive dashes, which pandoc converts into U+2013.
This post is a bit critical of a small well-intentioned project, so I felt obliged to email the maintainer to discuss it before posting it online. I didn’t hear back.

Troubleshooting a slow terminal startup issue.

Tavis Ormandy

Troubleshooting: Terminal Lag

Tavis Ormandy

$Id: a07cf90837a3c4373b82d6724b97593810766af7 $

I think I should blog more about random troubleshooting sessions, if nothing else it will remind me what steps I took when it inevitably happens again!

Okay, here is the first one – why is my xterm opening so slowly?

Background

I have two similarly specced machines at my desk – my primary workstation running Fedora Linux, and a Windows 11 machine. They share the same monitor and input devices, and I switch between them with an iogear KVM.

I do the bulk of my work in either a browser or a terminal. This is true even on Windows, where I rely heavily on WSL.

This works well for me, and I’m happy enough with the setup.

Issue

I have the shortcut Super+1 bound to xterm on both machines, and I probably use this hundreds of times per day.

Here is how that looks on Fedora:

It takes about 300ms from key activation to a terminal being ready. This is fine, I’ve never noticed any problem.

However, let’s compare that to Windows:

That’s about 1600ms before I can type, over 5 times slower! This is slow enough that it bothers me, and I use this shortcut so often that I want to solve it.

I don’t think many people care about xterm performance on Windows, so I guess that means it’s up to me to solve this 🙂

Window Effects

Hey, wait a minute… ENHANCE! 👀

Why does the window fade in like that? It looks like the Window is ready when the effect starts, but I can’t interact with it until it completes. If I count those frames, this animation must be costing me ~200ms! 🤬

I always disable anything like animation or compositing effects, so I’m confused where this is coming from.

I tested with some native windows programs like notepad and calc – they just appear instantly… so what is causing that?

I experimented with it a bit, I can see other windows behind it as it fades in, so I think something must be changing the opacity. Searching msdn, it looks like the function for that is SetLayeredWindowAttributes().

Could something be calling that, is my X server betraying me?

$ dumpbin /imports X410.exe  | grep SetLayeredWindowAttributes
                         33A SetLayeredWindowAttributes

This looks like the culprit!! I’m using a server called X410, it seems like it’s adding it’s own animation effects, and doesn’t have any way to disable it. I’m reluctant to switch to an alternative – that could just replace this issue with a different issue to troubleshoot.

Is it possible I can just stop it from doing that with a debugger?

$ cdb -p 6624
(19e0.2ad0): Break instruction exception - code 80000003 (first chance)
ntdll!DbgBreakPoint:
00007ff9`1f9b3c90 cc              int     3
0:014> eb win32u!NtUserSetLayeredWindowAttributes c3
0:014> .detach
Detached
NoTarget> q
quit:

Ah-ha, that actually worked!!!

I’ve added that cdb command into my xinit initialization, and it looks a lot snappier. That saved nearly 300ms, so we’re down to just 4 times slower than Fedora!

Profiling

Okay, let’s try get some real numbers. I like the tool hyperfine for this, here are the initial results:

If we run it under optimal conditions, it takes about 900ms on Windows, and about 100ms on Fedora.

Now that I can reproduce the delay reliably, I can start exploring some theories…

Filesystem

My first thought is that filesystem performance under WSL can be very slow, could that explain the difference?

taviso@WORKSTATION:~$ strace -wc -efile xterm -e true
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 43.93    0.014870          56       261        56 openat
 23.79    0.008053          31       257        33 access
 20.48    0.006932          32       211        12 newfstatat
  7.07    0.002394          23       100        54 readlink
  4.72    0.001596        1596         1           execve
------ ----------- ----------- --------- --------- ----------------
100.00    0.033845          40       830       155 total

Nope, it’s actually a bit faster on Windows! If I browse the logs, it looks related to fonts, and I do have fewer fonts installed on Windows. I suspect that causes fontconfig to query less files on initialization.

Whatever the reason, I concluded it wasn’t a big proportion of startup time, so it doesn’t seem worth worrying about.

X Server

The issue must be the X server, how fast is a very simple X client?

taviso@fedora:~$ hyperfine xdpyinfo
Benchmark 1: xdpyinfo
  Time (mean ± σ):       4.6 ms ±   0.8 ms    [User: 1.9 ms, System: 1.6 ms]
  Range (min … max):     3.1 ms …   9.4 ms    317 runs

That does run slower on Windows, but not significantly slower – perhaps I actually need to create a window to see a difference?

taviso@fedora:~$ x11perf -repeat 1 -subs 8 -popup
    5600000 reps @   0.0010 msec (1040000.0/sec): Hide/expose window via popup (8 kids)

That is also slower on Windows – this is understandable, it has to translate from X11 to win32 – but not so slow that it adequately explains the problem alone.

FreeType

Could it be a FreeType or FontConfig issue?

taviso@WORKSTATION:~$ ftbench -p consola.ttf
executing tests:
  Load                           2.491 us/op     809010 done
  Load_Advances (Normal)         2.437 us/op     827190 done
  Load_Advances (Fast)           0.022 us/op   88575990 done
  Load_Advances (Unscaled)       0.013 us/op  147448890 done
  Render                         2.039 us/op     390870 done
  Get_Glyph                      0.921 us/op     509040 done
  Get_Char_Index                 0.018 us/op  111551968 done
  Iterate CMap                  21.539 us/op      88503 done
  New_Face                       6.271 us/op     284029 done
  Embolden                       2.491 us/op     357540 done
  Stroke                        24.663 us/op      69690 done
  Get_BBox                       0.865 us/op     487830 done
  Get_CBox                       0.679 us/op     506010 done

Loading fonts is slightly slower, but the other numbers seem fine, and it’s not that much slower. I don’t think it’s this.

Features

Maybe I can get some clues from ltrace?

$ ltrace -c xterm -e true
% time     seconds  usecs/call     calls      function
------ ----------- ----------- --------- --------------------
 26.74    1.977389      988694         2 XtRealizeWidget
 22.18    1.640132       14139       116 XtSetValues
 13.39    0.989966      197993         5 XtVaCreateManagedWidget
  6.42    0.474410         200      2361 strlen
  5.51    0.407617      203808         2 read
  4.21    0.311086         197      1572 FcCharSetHasChar
  2.92    0.215975        2571        84 XtCreateManagedWidget
  2.80    0.207397       69132         3 XtVaCreatePopupShell
  2.39    0.176458         218       808 XftTextExtents32
  1.51    0.111306      111306         1 XpmReadFileToPixmap
  1.48    0.109486      109486         1 XtOpenApplication
  1.44    0.106786         203       524 malloc

This XtRealizeWidget call does seem slow, and I don’t see that on Fedora, what is calling that?

$ gdb --args ./xterm -e true
Reading symbols from ./xterm...
(gdb) b XtRealizeWidget
Breakpoint 1 at 0x2db20
(gdb) r
Breakpoint 1, 0x00007fffff43d940 in XtRealizeWidget () from /lib/x86_64-linux-gnu/libXt.so.6
(gdb) bt
#0  0x00007fffff43d940 in XtRealizeWidget () from /lib/x86_64-linux-gnu/libXt.so.6
...
#4  0x00007fffff44c4f6 in XtSetValues () from /lib/x86_64-linux-gnu/libXt.so.6
#5  0x0000000008086683 in UpdateMenuItem (menu=0x811c5c0 <mainMenuEntries>, which=0, val=1)
    at ./menu.c:1026
#6  0x000000000808a763 in update_toolbar () at ./menu.c:3366

Ah-ha – it’s the toolbar feature. It’s disabled at compile time on Fedora, but I quite like it and enable it on Windows.

If I disable that, startup is a little faster, I wonder if there are any other features that are slowing down initialization…?

Parameter Scanning

The hyperfine utility has a feature called parameter scan, where it it will try a bunch of settings for you and tell you which one is fastest.

Let’s ask XTerm what features are available, and toggle each one on and off.

$ xterm -report-xres -e true
activeIcon              : default
allowBoldFonts          : true
allowC1Printable        : false
allowColorOps           : true
allowFontOps            : false
allowMouseOps           : true
...

I’ll start by extracting all the settings that are booleans.

$ xterm -report-xres -e true | grep -Po '^\S+(?=\s+: (true|false))' | tr '\n' ','
allowBoldFonts,allowColorOps,allowMouseOps...

Note: xterm has a lot of features, I’m truncating the list for brevity!

Now we can give each of those to hyperfine, and let it figure out which settings have the most noticable effect.

That took about 20 minutes to run, and reports:

$ hyperfine --parameter-list res allowBoldFonts,allow... \
            --parameter-list bool true,false \
            "xterm -xrm 'XTerm*{res}: {bool}' -e true"
...
Benchmark 240: xterm -xrm 'XTerm*xftTrackMemUsage: false' -e true
  Time (mean ± σ):     140.1 ms ±   7.4 ms    [User: 30.8 ms, System: 23.4 ms]
  Range (min … max):   129.2 ms … 153.4 ms    21 runs
 
Summary
  xterm -xrm 'XTerm*tekInhibit: true' -e true ran
    1.01 ± 0.08 times faster than xterm -xrm 'XTerm*allowSendEvents: false' -e true
    ...

This helped a little, I found a combination of options that saved around 200ms total. One example was tekInhibit, which disables the Tektronix emulation. That’s usually used as a graphing mode – it’s actually pretty cool.

Still, it isn’t a big enough difference, and this is as far as I was able to get through tweaking settings.

I’m starting to think that this is just death by a thousand cuts, everything just has some small overhead on Windows and it adds up…

Caching

There’s a simple generic solution to slow startup performance: server mode.

The idea is to cache a few processes in the background, then all the slow stuff will already be done, ready for you to start working immediately.

XTerm doesn’t have this feature natively, but it’s not complicated, I can add it.

To do this, I will use deferred mapping – that just means that a program is running, but the window is not visible yet.

I tried a few solutions and found one that works well, an LD_PRELOAD library. All it does is intercept any toplevel XMapWindow() calls, then pause execution until it receives a signal.

It’s a bit hacky, but my code is here, if you’re interested.

To use it, you need something to manage the cache for you in the background, xargs will work!

$ xargs --null --arg-file=/dev/zero --max-procs=3 --replace -- \
     env LD_PRELOAD=defermap.so xterm -display :0 [PARAMS...]

This will keep three xterms running in the background.

Note: If you often rapidly start terminals in quick succession, increase max-procs.

When you want a new terminal, instead of running xterm as you normally would, do this instead:

$ pkill --oldest --signal SIGUSR1 xtermserver

A terminal should appear near-instantly. You can now execute that instead of xterm, and startup performance should be solved.

Conclusion

This whole process took a while! Now I need to adjust my shortcuts to run pkill instead of xterm, and I can compare the results.

Counting the frames in that video, it’s down to about 366ms, just 60ms slower than Fedora, this is totally acceptable!

I’ve been using this configuration for a few days, so far it’s working great. I haven’t noticed any issues running it this way.

I highly doubt anyone else will find this useful, who else is using XTerm on Windows? 😆

Nevertheless, if you have a better solution, or can think of something else I can try, let me know!

I wrote a brief introduction to the hiew hex editor.I started using ledger to track my finances.

Tavis Ormandy

Hiew Hex Editor

Tavis Ormandy

$Id: a07cf90837a3c4373b82d6724b97593810766af7 $

Introduction

A big part of my daily work is digging around in binary blobs. I’ve been using Hiew almost exclusively for a few years, and I’m a big fan – more people should use it!

The problem is Hiew has a steep learning curve, so if you literally use a hex editor once a year, then it’s probably just not worth it. However, if you live in a hex editor, then it might be more interesting than you realized!

This article assumes you’re already familiar with another hex editor, and are curious if it’s worth your time learning Hiew.

Let me walk you through a few common hex editing tasks and show you how they look… maybe 0.01% of the people reading this will be intrigued 😂

Background

Let’s get this out of the way - if you’ve never seen Hiew before, it’s a console hex editor!

It has basic mouse support, but you primarily operate it with the keyboard. If you think vim looks arcane, you’re not going to enjoy using Hiew – that’s okay!

Hiew has been around a long time, first released in 1991. It’s still actively maintained – version 8.80 was just released in November – but it comes from an era where powerful software was usually designed around the idea of keyboard macros.

Like other tools built this way, Hiew has an emphasis on chaining together simple commands to perform complex tasks. That means there’s a steep learning curve before you feel proficient, but as you build up your command vocabulary you’ll be able to perform operations that look like magic.

Okay, Let’s dive right in!

← Psst…this symbol means there’s a figure available… just click it!

🎈

Scenarios

Scenario #1

You’re trying to understand an undocumented proprietary file format.

A good place to start is usually looking at the list of strings that appear in the file, that’s alt+f6.

Hopefully that will give you some clues about what fields might be in the header. In Hiew, you mark fields using blocks, and you start a block with *.

I see there’s what looks like a filename near the top, so let me mark that.

All the usual movement commands work while you’re defining a block, including searching and jumping. Vim users will recognize this as very similar to visual mode.

Now that it’s marked, you can assign it a color with alt+m, or just give it a random color with alt+shift+m.

I like to just keep hitting alt+shift+m until I get a color I like.

Okay, now you can watch me repeat that process for a few more fields…

Tip: Feeling shortcut overload? Don’t worry, I made hiewdocs, an easy online shortcut reference!

Now we probably want to add some notes to remind us what we think a field is. In Hiew, you enter a comment with ;.

Just like with a dissasembler, each location can have a name and a comment, you enter a name with shift+f12. They’re both optional, skip one or both if you like!

Once some fields are marked, you can move between them with alt+n (forward) and alt+shift+n (backward).

Hiew will display any names and comments as you navigate around as floating tips. You can also open a list with f12 and jump straight to the one you’re interested in!

I think we’re making progress. Look at this field, I think it’s the length of the data block:

If I examine it in the calculator, as a DWORD that would be 203008. Okay, Let’s extract a block of that length and save it to a file.

So we need to…

Hit * to select a block.
Press f5 to jump to an offset.
Enter +203008t to move forward that many bytes.

Now we can use f2 to save the current block to a file.

You can hit backspace to jump back to your last location when done.

Tip: Hiew has about a million ways to jump around a file! Don’t worry, you don’t need to memorize them!

Let’s take a look at this block we saved. I think I know what this is, if I add a deflate header…

Ah-ha, we got back the uncompressed data!

You probably spotted the PK header, and realized this is not an undocumented file format at all, it’s a ZIP archive. Yep, you got me 🙂

You don’t need to go through this process for common formats like Zip, you can just use a standard template that will populate the fields for you. If I do that, the names list f12 looks like this.

Scenario #2

You’re trying to patch an executable you don’t have the source code for.

The first thing you need to know is that Hiew has three main views. They are text, code and data. You can quickly flip between them without losing your place, just hit enter. Most Hiew users get into the habit of doing this frequently!

Let’s take a closer look at the code view.

As you can see, Hiew has a builtin disassembler, very handy if you work in security and need to extract shellcode!

Do you see those little arrow annotations? Those are shortcuts to follow a branch, you just hit the corresponding key to jump to the branch target. This lets you browse the code effectively without leaving Hiew.

If you see ↓5, that means it’s a forward branch and you can press 5 to jump to the destination.

While we’re browsing around we can enter comments with ;, just like in data view. The only difference is that your comments get added to the disassembly, rather than as floating tooltips!

Here’s a neat feature, see these dashes?

That’s your bookmark stack! If you want to go explore somewhere else, you can hit +, then you return later by hitting -.

Tip: Take a look at hiewdocs if you want to master the bookmark stack!

Hiew understands most major executable formats, and will populate names automatically. You can easily browse and patch headers, sections, tables, and so on.

Let’s take a look at the imports table.

Look, there’s a call to IsDebuggerPresent().

Let’s patch it out just for fun. Hiew not only has a dissasembler built in, it also has an assembler. I think this is a unique feature among major hex editors!

There’s a handy shortcut to just nop an instruction alt+f2.

Now let’s change this nop into xor eax, eax, and we’re done!

I challenge you to find me a hex editor that makes patching executables easier than Hiew 😂

Scenario #3

You’re trying to repair a damaged file.

Want to see a picture of my pup 🐶? His name is Deputy Dexter!

$ display dexter.png
display-im6.q16: improper image header `dexter.png' @ error/png.c/ReadPNGImage/4107.

Huh, that didn’t work! Maybe we can fix this problem with Hiew.

Okay, this doesn’t look right, there’s an extra 0D in the header.

I’d bet this was probably broken by someone trying to convert from UNIX to DOS line endings!¹ We need to go through the file and replace every occurrence of 0D 0A with 0A.

A repetitive task like this is a good opportunity for me to show you how keyboard macros work. All we need to do is fix one of these errors, then we can tell Hiew to repeat everything we just did.

So, hit alt+. to begin recording a macro. This recording symbol should appear.

Now we need to search for the next occurrence of 0D 0A, so f7, and enter 0D0A.

To delete a byte, you mark it then press shift+f2.

That’s it! Now hit alt+. to end the macro.

To tell Hiew to repeat this for every occurrence in the file, open the macro manager with alt+=, and then make sure the L (loop) and S (search failure) flags are set. This just means that Hiew should repeat exection until a search failure occurs.

You can give it a name if you want to save it for future use, but we’re only going to use it once. So hit enter to run it!

Tip: The default execution speed is 200ms, which is slow enough that you can watch it run. It’s kinda cool to watch, but you can turn it all the way down to 0ms if you like!

Let’s check if that worked, can we see Dexter!

There’s my little guy!

Tip: You can abort a running macro with esc. For some reason it will say “aborted by hacker” lol?

What happens when a CPU goes wrong?

Tavis Ormandy

Reptar

Tavis Ormandy

We have a CPU mystery! We found a way to cause some processors to enter a glitch state where the normal rules don’t apply, but what does that mean…?

If you’re interested what can go wrong inside modern CPUs, read on!

Introduction

If you’ve ever written any x86 assembly at all, you’ve probably used rep movsb. It’s the idiomatic way of moving memory around on x86. You set the source, destination, direction and the count - then just let the processor handle all the details!

lea rdi, [rel dst]
lea rsi, [rel src]
std
mov rcx, 32
rep movsb

The actual instruction here is movsb, the rep is simply a prefix that changes how the instruction works. In this case, it indicates that you want this operation repeated multiple times.

There are lots of other prefixes too, but they don’t all apply to every instruction.

Prefix Decoding

An interesting feature of x86 is that the instruction decoding is generally quite relaxed. If you use a prefix that doesn’t make sense or conflicts with other prefixes nothing much will happen, it will usually just be ignored.

This fact is sometimes useful; compilers can use redundant prefixes to pad a single instruction to a desirable alignment boundary.

Take a look at this snippet, this is exactly the same code as above, just a bunch of useless or redundant prefixes have been added:

            rep lea rdi, [rel dst]
             cs lea rsi, [rel src]
       gs gs gs std
          repnz mov rcx, 32
rep rep rep rep movsb

Perhaps the most interesting prefixes are rex, vex and evex, all of which change how subsequent instructions are decoded.

Let’s take a look at how they work.

The REX prefix

The i386 only had 8 general purpose registers, so you could specify which register you want to use in just 3 bits (because 2^3 is 8).

The way that instructions were encoded took advantage of this fact, and reserved just enough bits to specify any of those registers.

Simple 2-byte instructions that use modr/m might be encoded like this, for example mov eax, ebx.

This is an 8-bit opcode, 2 bit addressing mode (labeled m), and 3 bits each for the source (s) and destination (d).

Well, this is a problem, because x86-64 added 8 additional general purpose registers. We now have sixteen possible registers..that’s 2^4, so we’re going to need another bit! 😆

The solution to this is the rex prefix, which gives us some spare bits that the next instruction can borrow.

When we’re talking about rex, we usually write it like this:

rex.rxb

rex is a single-byte prefix, the first four bits are mandatory and the remaining four bits called b, x, r and w are all optional. If you see rex.rb that means only the r and b bits are set, all the others are unset.

These optional bits give us room to encode more general purpose registers in the following instruction.

The rex prefix can lend the next instruction extra bits to use for operands, so now we can encode all 16 possible general purpose registers!

Now we’re fine until someone adds another register! 😂

Encoding Rules

So now we know that rex increases the available space for encoding operands, and that useless or redundant prefixes are usually ignored on x86. So… what should this instruction do?

rex.rxb rep movsb

The movsb instruction doesn’t have any operands - they’re all implicit - so any rex bits are meaningless, right?

If you guessed that the processor will just silently ignore the rex prefix, you would be correct!

Well… except on machines that support a new feature called fast short repeat move! We discovered that a bug with redundant rex prefixes could interact with this feature in an unexpected way and introduce a serious vulnerability, oops 🙂

Fast Short Repeat Move

FSRM is a new feature introduced in Ice Lake that fixes some of the shortcomings of ERMS. Hopefully that clears up any confusion. 😆

Just kidding, let’s quickly look at ERMS.

The hard part of moving strings around efficiently is getting all the buffers aligned so you can use the widest possible stores available. You could do this in software, but if we do it in microcode then the processor can just transparently make your existing code faster for you.

This requires some expensive setup, but once that’s done you get vastly improved throughput. This feature is known as enhanced repeat move/store, ERMS.

If you have a processor with ERMS support, simple rep movsb operations can sometimes perform comparably with more complicated hand-tuned vector move operations.

However, there is a problem with ERMS. That initial setup is so expensive that it just isn’t worth it for very short strings. This is what FSRM is designed to solve, it handles the case of only moving 128 bytes or less and makes that faster too!

I’m not aware of any documentation that explains exactly how FSRM works, but you can check if you have a processor that supports it by looking at the flags line in /proc/cpuinfo:

flags       : fpu vme de pse tsc msr pae mce cx8 [...] fsrm

Some of the processors that have this feature include:

Ice Lake
Rocket Lake
Tiger Lake
Raptor Lake
Alder Lake
Sapphire Rapids

Note: This list may not be comprehensive, please see Intel advisory INTEL-SA-00950 for a complete list.

Discovery

I’ve written previously about a processor validation technique called Oracle Serialization that we’ve been using. The idea is to generate two forms of the same randomly generated program and verify their final state is identical.

You can read more about Oracle Serialization in my previous writeup.

In August, our validation pipeline produced an interesting assertion. It had found a case where adding redundant rex.r prefixes to an FSRM optimized rep movs operation seemed to cause unpredictable results.

We observed some very strange behavior while testing. For example, branches to unexpected locations, unconditional branches being ignored and the processor no longer accurately recording the instruction pointer in xsave or call instructions.

Oddly, when trying to understand what was happening we would see a debugger reporting impossible states!

This already seemed like it could be indicative of a serious problem, but within a few days of experimenting we found that when multiple cores were triggering the same bug, the processor would begin to report machine check exceptions and halt.

We verified this worked even inside an unprivileged guest VM, so this already has serious security implications for cloud providers. Naturally, we reported this to Intel as soon as we confirmed this was a security issue.

Reproduce

We’re publishing all of our research today to our security research repository. If you want to reproduce the vulnerability you can use our icebreak tool, I’ve also made a local mirror available here.

$ ./icebreak -h
usage: ./icebreak [OPTIONS]
    -c N,M      Run repro threads on core N and M.
    -d N        Sleep N usecs between repro attempts.
    -H N        Spawn a hammer thread on core N.
icebreak: you must at least specify a core pair with -c! (see -h for help)

The testcase enters what should be an infinite loop, and unaffected systems should see no output at all. On affected systems, a . is printed on each successful reproduction.

$ ./icebreak -c 0,4
starting repro on cores 0 and 4
.........................................................................
.........................................................................
.........................................................................
.........................................................................
.........................................................................

In general, if the cores are SMT siblings then you may observe random branches and if they’re SMP siblings from the same package then you may observe machine checks.

If you do not specify two different cores, then you might need to use a hammer thread to trigger a reproduction.

Analysis

We know something strange is happening, but how microcode works in modern systems is a closely guarded secret. We can only theorize about the root cause based on observations.

μops

The CPU is split in two major components, the frontend and the backend. The frontend is responsible for fetching instructions, decoding them and generating μops to send to the backend for execution.

The backend executes instructions out of order, and uses a unit called the ROB, reorder buffer, to store and organize results.

We believe this bug causes the frontend to miscalculate the size of the movsb instruction, causing subsequent entries in the ROB to be associated with incorrect addresses. When this happens, the CPU enters a confused state that causes the instruction pointer to be miscalculated.

The machine can eventually recover from this state, perhaps with incorrect intermediate results, but becoming internally consistent again. However, if we cause multiple SMT or SMP cores to enter the state simultaneously, we can cause enough microarchitectural state corruption to force a machine check.

Questions

I’m sure some readers will have questions about what is possible in this unexpected “glitch” state. Well, so do we!

We know that we can corrupt the system state badly enough to cause machine check errors, and we’ve also observed threads interfere with execution of processes scheduled on SMT siblings.

However, we simply don’t know if we can control the corruption precisely enough to achieve privilege escalation. I suspect that it is possible, but we don’t have any way to debug μop execution!

If you’re interested in studying this, then we would love to get your input!

Credit

This bug was independently discovered by multiple research teams within Google, including the silifuzz team and Google Information Security Engineering. The bug was analyzed by Tavis Ormandy, Josh Eads, Eduardo Vela Nava, Alexandra Sandulescu and Daniel Moghimi.

Solution

Intel have published updated microcode for all affected processors. Your operating system or BIOS vendor may already have an update available!

Workaround

If you can’t update for some reason, you could disable fast strings via the IA32_MISC_ENABLE model specific register.

This will cause a significant performance penalty, and should not be used unless absolutely necessary.

Notes

If you’re interested in more CPU bugs, we publish everything we find!

Not all the bugs we discover have security consequences, but they’re usually worth reading! For example, did you know that sometimes movlps just doesn’t work? or that registers can sometimes roll back to previous values?

An overview of hacking the Timex m851.

Tavis Ormandy

Hacking the Timex m851

Tavis Ormandy

$Id: a07cf90837a3c4373b82d6724b97593810766af7 $

This post is part of a series on my quest to find the perfect watch.

Intro

Take a look at this watch, it’s just some boring watch for runners, right?

Nope, I think this might be the best ultra-low power consumer digital watch ever produced!

Let me explain…

Specifications

This is the Timex m851. It uses an 8-bit Seiko SC188 CPU, has 48KB of ROM, 2KB of RAM and a 42x11 dot matrix main display.

The cpu is designed for ultra-low power operation - a single battery can last 3 years!

This is a big selling point for me, I don’t think they’re making consumer watches like this any more.

Those pins on the side? That’s USB. Naturally, you can synchronize things like appointments, and so on (there’s a neat linux library for that).

Here’s where it get’s interesting… this thing has an SDK, and it is surprisingly good!

You can just $ cc helloworld.c, and upload it to your watch.

Timex provided a Windows XP-era wizard to develop applications, and it is painful. However, it turned out that was just driving a UNIX-like toolchain behind the scenes. After pulling that out, hacking this started to become pretty fun!

WatchApps

Just want to see some code? No problem, I’ve uploaded a hello world and a Makefile to build it to github:

https://github.com/taviso/timex/

If you want to browse the full manual, there are some links below.

You can do some fun things with this, people wrote all kinds of games, utilities, and tools.

If you want a TL;DR of development, I’ll try!

Design

States and Modes

You need to split your application into states.

A state is just a way to swap code in and out as needed. When you switch states, the active code is discarded and the new code is swapped in.

You could write an app that uses up to ~30kb of code and/or data, but there is only ~2kb of RAM. This is why you need to use states, there is no paging and you couldn’t fit that in!

There is room for some common code that all your states share, and of course there is space for variables, along with a database API for persistent data.

The first few states are reserved for handling common events. The rest you can use for anything you want - you just ask the kernel to switch you whenever you need it!

if (foo > bar) {
    coreRequestStateChange(FOOBAR_STATE);
}

A mode is just a foreground application.

Your app is (probably) providing a mode, but you can also add background tasks, periodic tasks, and so on.

Requesting a mode change is sort of like exit(), the next app will gain control.

coreRequestModeChangeNext();

Events

Each state needs an event handler. A super simple one would be like this:

void default_state_manager(void)
{
    switch (CORECurrentEvent) {
        // This is called when this state becomes active.
        case COREEVENT_STATEENTRY:
            coreEnableSwitchReleaseEvents();
            coreAllowKeys(COREALLOWALLSWITCHES);
            break;
        // This is called when the user pulls the crown out.
        // When they push it back in, you get a CROWN_HOME event.
        // It can be rotated in either home or set positions.
        case COREEVENT_CROWN_SET:
            coreRequestStateChange(CORESETBANNERSTATE);
            break;
        // You can get button events too, like start/split:
        case COREEVENT_STARTSPLITDEPRESS:
            show_message();
            break;
        // The mode button probably means the user wants to exit...
        case COREEVENT_MODEDEPRESS:
            coreRequestModeChangeNext();
            break;
    }

    return;
}

In general, you can ignore any event you don’t care about.

Services

The kernel takes care of handling the hardware, dispatching events to you and provides various services.

Things like timers, generating tones (beeps), scrolling the display, access to database records, etc.

uint8_t banner[] = {
    LCDBANNER_COL10,
    DM5_H, DM5_E, DM5_L, DM5_L, DM5_O, DM5_EXCLAMATION,
    LCD_END_BANNER
};

lcdClearDisplay();

// Draw "hello!" using kernel services.
lcdDispBannerMsg(&banner);

The watch has a crown - and you can ask the kernel to configure it in a few different ways. For example, you might want pulse mode so you can monitor how quickly the user is turning it.

It has a backlight, you can control that from software too.

Debugging

There’s no memory protection or anything, you can just clobber the kernel if you want… so what happens if you crash?

Well, one thing the kernel does is notify a watchdog every 2 seconds that it’s still alive. If the watchdog doesn’t see that notification, it will just reset the watch… so try not to do that! 😀

If you’re doing some heavy computation you could accidentally trigger the watchdog. Just remember to let the watchdog know you’re still alive:

 // Let the watchdog know we're not dead.
 hwResetWatchdog();

Simulator

Luckily there’s a really good simulator available, so you don’t have to sit through tedious resets everytime you make a typo. It’s actually an open source 3rd party tool, Virtual Datalink (source).

Sadly it’s for Windows only. I would port it to Linux, but it’s written in Delphi… is it feasible to port it to free pascal? Let me know!

It has conditional breakpoints, a dissasembler, save states, resource and power analysis. I’ve found it pretty capable.

Conclusion

The main app I’m working on right now is integrating my watch with remind.

Hacking this little watch has been a super fun project!

Availability

Now that I’ve told you how much I love this thing - the bad news.

They’re long out of production, and getting hard to come by.

I bought a set of two on eBay, they just needed new batteries and were as good as new.

If this is the sort of thing that appeals to you, setup an alert and then help me hack on it 😀

Tips

Found a watch but missing the cable? It uses the standard USB type-a pinout with an odd connector, you can probably just use an old USB cable and alligator clips.
Timex used the same connector on a much less desirable device called the T5G751 (probably other model variants too). These are easy to find at reasonable prices (often < $10 at the time of writing), so you can simply toss the device and keep the cable (example).
There was a m851 variant called the “dress edition”, it has identical specs but a different body. It is sometimes listed with a different model number (example).

References

The most important doc is the Design Guide, it’s very well written.
Then of course, you need the CPU manual. It’s also excellent!
The toolchain also has a thorough manual.
Finally, the complete API reference lists the services provided by the Kernel.

I found an interesting CPU bug.

Tavis Ormandy

Zenbleed

Tavis Ormandy

$Id: 7d85ef516b9682eaf0db09cc9c4e53a523486f3f $

If you remove the first word from the string "hello world", what should the result be? This is the story of how we discovered that the answer could be your root password!

Introduction

All x86-64 CPUs have a set of 128-bit vector registers called the XMM registers. You can never have enough bits, so recent CPUs have extended the width of those registers up to 256-bit and even 512-bits.

The 256-bit extended registers are called YMM, and the 512-bit registers are ZMM.

These big registers are useful in lots of situations, not just number crunching! They’re even used by standard C library functions, like strcmp, memcpy, strlen and so on.

Let’s take a look at an example. Here are the first few instructions of glibc’s AVX2 optimized strlen:

(gdb) x/20i __strlen_avx2
...
   <__strlen_avx2+9>:   vpxor  xmm0,xmm0,xmm0
...
   <__strlen_avx2+29>:  vpcmpeqb ymm1,ymm0,YMMWORD PTR [rdi]
   <__strlen_avx2+33>:  vpmovmskb eax,ymm1
...
   <__strlen_avx2+41>:  tzcnt  eax,eax
   <__strlen_avx2+45>:  vzeroupper
   <__strlen_avx2+48>:  ret

The full routine is complicated and handles lots of cases, but let’s step through this simple case. Bear with me, I promise there’s a point!

The first step is to initialize ymm0 to zero, which is done by just xoring xmm0 with itself¹.

    > vpxor xmm0, xmm0, xmm0
      vpcmpeqb ymm1, ymm0, [rdi]
      vpmovmskb eax, ymm1
      tzcnt eax, eax
      vzeroupper

Here rdi contains a pointer to our string, so vpcmpeqb will check which bytes in ymm0 match our string, and stores the result in ymm1.

As we’ve already set ymm0 to all zero bytes, only nul bytes will match.

      vpxor xmm0, xmm0, xmm0
    > vpcmpeqb ymm1, ymm0, [rdi]
      vpmovmskb eax, ymm1
      tzcnt eax, eax
      vzeroupper

Now we can extract the result into a general purpose register like eax with vpmovmskb.

Any nul byte will create a 1 bit, and any other value will create a 0 bit.

      vpxor xmm0, xmm0, xmm0
      vpcmpeqb ymm1, ymm0, [rdi]
    > vpmovmskb eax, ymm1
      tzcnt eax, eax
      vzeroupper

Finding the first zero byte is now just a case of counting the number of trailing zero bits.

That’s a common enough operation that there’s an instruction for it - tzcnt (Trailing Zero Count).

      vpxor xmm0, xmm0, xmm0
      vpcmpeqb ymm1, ymm0, [rdi]
      vpmovmskb eax, ymm1
    > tzcnt eax, eax
      vzeroupper

Now we have the position of the first nul byte, in just four machine instructions!

You can probably imagine just how often strlen is running on your system right now, but suffice to say, bits and bytes are flowing into these vector registers from all over your system constantly.

Zeroing Registers

You might have noticed that I missed one instruction, and that’s vzeroupper.

      vpxor xmm0, xmm0, xmm0
      vpcmpeqb ymm1, ymm0, [rdi]
      vpmovmskb eax, ymm1
      tzcnt eax, eax
    > vzeroupper

You guessed it, vzeroupper will zero the upper bits of the vector registers.

The reason we do this is because if you mix XMM and YMM registers, the XMM registers automatically get promoted to full width. It’s a bit like integer promotion in C.

This works fine, but superscalar processors need to track dependencies so that they know which operations can be parallelized. This promotion adds a dependency on those upper bits, and that causes unnecessary stalls while the processor waits for results it didn’t really need.

These stalls are what glibc is trying to avoid with vzeroupper. Now any future results won’t depend on what those bits are, so we safely avoid that bottleneck!

The Vector Register File

Now that we know what vzeroupper does, how does it do it?

Your processor doesn’t have a single physical location where each register lives, it has what’s called a Register File and a Register Allocation Table. This is a bit like managing the heap with malloc and free, if you think of each register as a pointer. The RAT keeps track of what space in the register file is assigned to which register.

In fact, when you zero an XMM register, the processor doesn’t store those bits anywhere at all - it just sets a flag called the z-bit in the RAT. This flag can be applied to the upper and lower parts of YMM registers independently, so vzeroupper can simply set the z-bit and then release any resources assigned to it in the register file.

A register allocation table (left) and a physical register file (right).

Speculation

Hold on, there’s another complication! Modern processors use speculative execution, so sometimes operations have to be rolled back.

What should happen if the processor speculatively executed a vzeroupper, but then discovers that there was a branch misprediction? Well, we will have to revert that operation and put things back the way they were… maybe we can just unset that z-bit?

If we return to the analogy of malloc and free, you can see that it can’t be that simple - that would be like calling free() on a pointer, and then changing your mind!

That would be a use-after-free vulnerability, but there is no such thing as a use-after-free in a CPU… or is there?

Spoiler: yes there is 🙂

This animation shows why resetting the z-bit is not sufficient.

Vulnerability

It turns out that with precise scheduling, you can cause some processors to recover from a mispredicted vzeroupper incorrectly!

This technique is CVE-2023-20593 and it works on all Zen 2 class processors, which includes at least the following products:

AMD Ryzen 3000 Series Processors
AMD Ryzen PRO 3000 Series Processors
AMD Ryzen Threadripper 3000 Series Processors
AMD Ryzen 4000 Series Processors with Radeon Graphics
AMD Ryzen PRO 4000 Series Processors
AMD Ryzen 5000 Series Processors with Radeon Graphics
AMD Ryzen 7020 Series Processors with Radeon Graphics
AMD EPYC “Rome” Processors

The bug works like this, first of all you need to trigger something called the XMM Register Merge Optimization², followed by a register rename and a mispredicted vzeroupper. This all has to happen within a precise window to work.

We now know that basic operations like strlen, memcpy and strcmp will use the vector registers - so we can effectively spy on those operations happening anywhere on the system! It doesn’t matter if they’re happening in other virtual machines, sandboxes, containers, processes, whatever!

This works because the register file is shared by everything on the same physical core. In fact, two hyperthreads even share the same physical register file.

Don’t believe me? Let’s write an exploit 🙂

Exploitation

There are quite a few ways to trigger this, but let’s examine a very simple example.

    vcvtsi2s{s,d}   xmm, xmm, r64
    vmovdqa         ymm, ymm
    jcc             overzero
    vzeroupper
overzero:
    nop

Here cvtsi2sd is used to trigger the merge optimization. It’s not important what cvtsi2sd is supposed to do, I’m just using it because it’s one of the instructions the manual says use that optimization³.

Then we need to trigger a register rename, vmovdqa will work. If the conditional branch⁴ is taken but the CPU predicts the not-taken path, the vzeroupper will be mispredicted and the bug occurs!

Optimization

It turns out that mispredicting on purpose is difficult to optimize! It took a bit of work, but I found a variant that can leak about 30 kb per core, per second.

This is fast enough to monitor encryption keys and passwords as users login!

We’re releasing our full technical advisory, along with all the associated code today. Full details will be available in our security research repository.

If you want to test the exploit, the code is available here.

Note that the code is for Linux, but the bug is not dependent on any particular operating system - all operating systems are affected!

Discovery

I found this bug by fuzzing, big surprise 🙂 I’m not the first person to apply fuzzing techniques to finding hardware flaws. In fact, vendors fuzz their own products extensively - the industry term for it is Post-Silicon Validation.

So how come this bug wasn’t found earlier? I think I did a couple of things differently, perhaps with a new perspective as I don’t have an EE background!

Feedback

The best performing fuzzers are guided by coverage feedback. The problem is that there is nothing really analogous to code coverage in CPUs… However, we do have performance counters!

These will let us know when all kinds of interesting architectural events happen.

Feeding this data to the fuzzer lets us gently guide it towards exploring interesting features that we wouldn’t have been able to find by chance alone!

It was challenging to get the details right, but I used this to teach my fuzzer to find interesting instruction sequences. This allowed me to discover features like merge optimization automatically, without any input from me!

Oracle

When we fuzz software, we’re usually looking for crashes. Software isn’t supposed to crash, so we know something must have gone wrong if it does.

How can we know if a a CPU is executing a randomly generated program correctly? It might be completely correct for it to crash!

Well, a few solutions have been proposed to this problem. One approach is called reversi. The general idea is that for every random instruction you generate, you also generate the inverse (e.g. ADD r1, r2 → SUB r1, r2). Any deviation from the initial state at the end of execution must have been an error, neat!

The reversi approach is clever, but it makes generating testcases very complicated for a CISC architecture like x86.

A simpler solution is to use an oracle. An oracle is just another CPU or a simulator that we can use to check the result. If we compare the results from our test CPU to our oracle CPU, any mismatch would suggest that something went wrong.

I developed a new approach with a combination of these two ideas, I call it Oracle Serialization.

Oracle Serialization

As developers we monitor the macro-architectural state, that’s just things like register values. There is also the micro-architectural state which is mostly invisible to us, like the branch predictor, out-of-order execution state and the instruction pipeline.

Serialization lets us have some control over that, by instructing the CPU to reset instruction-level parallelism. This includes things like store/load barriers, speculation fences, cache line flushes, and so on.

The idea of a Serialized Oracle is to generate a random program, then automatically transform it into a serialized form.

`movnti [rbp+0x0],ebx`	`movnti [rbp+0x0],ebx`
	`sfence`
`rcr dh,1`	`rcr dh,1`
	`lfence`
`sub r10, rax`	`sub r10, rax`
	`mfence`
`rol rbx, cl`	`rol rbx, cl`
	`nop`
`xor edi,[rbp-0x57]`	`xor edi,[rbp-0x57]`

: A randomly generated sequence of instructions, and the same sequence but with randomized alignment, serialization and speculation fences added.

These two program might have very different performance characteristics, but they should produce identical output. The serialized form can now be my oracle!

If the final states don’t match, then there must have been some error in how they were executed micro-architecturally - that could indicate a bug.

This is exactly how we first discovered this vulnerability, the output of the serialized oracle didn’t match!

Solution

We reported this vulnerability to AMD on the 15th May 2023.

AMD have released an microcode update for affected processors. Your BIOS or Operating System vendor may already have an update available that includes it.

Workaround

It is highly recommended to use the microcode update.

If you can’t apply the update for some reason, there is a software workaround: you can set the chicken bit DE_CFG[9].

This may have some performance cost.

Linux

You can use msr-tools to set the chicken bit on all cores, like this:

# wrmsr -a 0xc0011029 $(($(rdmsr -c 0xc0011029) | (1<<9)))

FreeBSD

On FreeBSD you would use cpucontrol(8).

Others

If you’re using some other operating system and don’t know how to set MSRs, ask your vendor for assistance.

Note that it is not sufficient to disable SMT.

Detection

I am not aware of any reliable techniques to detect exploitation. This is because no special system calls or privileges are required.

It is definitely not possible to detect improper usage of vzeroupper statically, please don’t try!

Conclusion

It turns out that memory management is hard, even in silicon 🙂

Acknowledgements

This bug was discovered by me, Tavis Ormandy from Google Information Security!

I couldn’t have found it without help from my colleagues, in particular Eduardo Vela Nava and Alexandra Sandulescu. I also had help analyzing the bug from Josh Eads.

George Lucas once sued someone over a spreadsheet.

Tavis Ormandy

Templates of Doom!

Tavis Ormandy

$Id: a07cf90837a3c4373b82d6724b97593810766af7 $

Introduction

In the late 1980s there was - believe it or not - a popular monthly magazine dedicated to spreadsheets. It had a respectable eight year run, along with two companion books.

The entire back catalog is on archive.org, although most issues were scanned from microfiche so there’s no color.

I’m interested in Lotus 1-2-3 history, so I’ve been slowly reading through back issues. It’s actually great, there are some really approachable articles on linear regression, forecasting, calculating bond yields, etc… mixed in with some retro computing news!

There’s also advice, tips, reviews, reader questions, and so on.

Why 1-2-3?

I’m sometimes asked what it is I find appealing about Lotus 1-2-3. It’s from an era where there was an expectation that “powerful” software would be designed around keystroke macros. I think I just really appreciate that.

The moment that Vim first “clicked” for me is when I wrote my first macro. I suddenly understood why there is this huge selection of commands, and why that is so powerful.

In vim you can record a keystroke macro to a register called x with qx, then run it with @x. In 1-2-3 you can write a keystroke macro to a range called \x, and run it with Alt-x. Not so different!

Lotus

I was reading the December 1987 issue and read this review of “Templates of Doom”, a game written in Lotus macros! Hilarious, a real commercially released game you could actually buy for $69.95! Of course, I had to try it.

As far as I can tell, the full game has not been archived anywhere, but I was able to find a demo version from an old shareware archive. It seems to work okay in the UNIX version of 1-2-3.

I contacted the original author and asked him if he still had any old copies. He did, he had some old 5.25” disks but didn’t have any hardware to read them.

Well, I happen to know an expert in 5.25” disks, and with his help I was able to get a disk image!

History

The package I got in the mail looked like this, but there was this page to stuck to the front that was peeling off.

The glue had deterioated after 30 years, but there was clearly something underneath…it was loose anyway, so I removed it revealing different cover art.

It turns out that in December 1988 Lucasfilm sued the author over similarities to “Indiana Jones and the Temple of Doom”, and the author had to change the cover art to appease Lucas’s lawyers!

Gameplay

So how does the game work! It’s a set of 24 puzzles that encourage you to learn how to use 1-2-3 features through problem solving.

You’re timed when you start each puzzle, and scored based on how long it took you to solve and how many hints you needed.

It has a boss key (Alt+B) and a hint system (Alt+H). When you’ve solved the puzzle, you hit Alt+A and choose the right answer - if you win it takes you to the score screen.

If you want to see me solve one of the puzzles, there’s a video here.

Playing

If you want to play this yourself…

Install 123elf.
Set your global directory to where the game is, /Worksheet Global Default Dir
/File Retrieve STARTPRG.WKS
You might get the error File and/or extension converted. This is just 1-2-3 informing you it translated something from DOS to UNIX. Just hit Enter.
When the file is loaded, press Alt+S to start.

Note

You don’t have to, but the game assumes you’re using 80x25 (the standard DOS resolution). Some cells you’re not supposed to see will be visible if you don’t change your terminal to match that.

Now just follow the prompts!

Remember to press Alt+S when a new level loads.

I’m looking for a new wrist watch.

Tavis Ormandy

Quest for my perfect watch.

Tavis Ormandy

$Id: a07cf90837a3c4373b82d6724b97593810766af7 $

Intro

I love my Casio Protrek watch, I won’t leave the house without it. I’ve owned it for years, as well as some earlier Protrek models. Those earlier models didn’t break — these watches are basically indestructible — I just liked them so much I went and bought the new revisions!

It has the standard digital watch features (alarms, timers, stopwatch, world time, calendar, backlight, etc), but this watch is really intended for hikers — it has an altimeter, barometer, thermometer and digital compass built in. I’m not outdoorsy — The only hiking I’ll do is to Starbucks — but I still love this watch. I think the reason is it’s reliable — I never worry that it won’t work.

The standard watch modes displayed in the manual.
I think I’ve looked at the barometric pressure graph a total of two times over the years. It is pretty cool though, the theory is you can predict an upcoming weather change based on the direction the graph is trending. If the graph is flat, the current weather is likely to hold for a few hours.

Maybe you’re thinking “hah, unless the battery needs to be replaced” — wrong! The watch face is literally a solar panel, it sips so little power (we’re talking μamps) that a few cumulative minutes exposure to daylight is enough to keep the watch running for months.

I would be astonished if I woke up tomorrow and it wasn’t working, and it’s rare I can say that about any other tech I use. If an unattended update hosed my laptop, or an app update breaks my workflow — oh well, that’s par for the course. Sure, a digital watch is simple in comparison, but it’s nice to have something I can trust to do it’s job.

Smartwatch

Still, I am a nerd — and I hear the siren call of smartwatches. I think I would get frustrated quickly with an Android or iOS smartwatch, but is there some middle ground? A simple, reliable, programmable digital watch?

It doesn’t have to be zero-maintenance, I can tolerate annual battery replacements… but daily or weekly recharging and any dependence on online services really seems unbearable to me after years with my Casio.

The main feature I would like is some sort of sync capability. Ideally, I would like the alarms and reminders from my UNIX ~/.reminders file to appear on my watch, perhaps over RF or BLE. I don’t mind writing code to make this happen, so some customizability, automation or programmability is essential.

What options are there for someone like me?

Go Homebrew?

There are a few homebrew smartwatches that appeal to me.

Sensorwatch

Jose Castillo has built SensorWatch, a fully programmable replacement board for the Casio F-91W watch.

This almost meets all my needs, it lasts a full year on a single battery and I could write an app to handle my reminders. The only issue is that sending data to the watch would be difficult once it’s assembled, as only the buttons can be used for input.

There is an add-on board that adds GPIO pins, the idea is you can add your own additional sensors… if you can make them fit. This might be beyond my hardware skills, but perhaps exposing the pins externally to add some form of i2c might be good enough — at the expense of water resistance.

I could ask Jose how he would add sync, perhaps he knows of some small sensor like a magnetometer or IR photodiode that I could use to build a low-bandwidth communication channel.

Goodwatch

Travis Goodspeed (unfortunately for him, people sometimes confuse us because of our similar names!) has a very impressive project called goodwatch for amateur radio enthusiasts. It’s another board swap using the MSP430, but for a different Casio shell with far more inputs and a more capable segment display. Travis says a battery can last months, perhaps longer depending on usage. That’s a little lower than I would like, but it sounds tolerable.

I don’t have much amateur radio knowledge, but it seems plausible I could add some low-bandwidth sync capability with a cheap SDR dongle and a weekend hacking MSP340 assembly. I can absolutely imagine using this, although Travis is a notable hardware hacker and I can barely solder! Assembly looks a little more involved than the sensorwatch, so I worry this might be out of my league!

Go Hybrid?

There is an interesting Sony product line called Wena, primarily sold in the Japanese market. Wena is not a smartwatch, it’s a “smartband” that replaces the band on your existing watch. This seems moderately interesting, it needs regular recharging but degrades gracefully into a regular watch if the battery dies.

Of course, there is no built-in functionality to use it for what I want, but it seems totally plausible I could add it — decompiling the companion APK if necessary to figure out the API.

It is not cheap, and I would have to import one to the US. Frankly, online reviews are not favourable at all — but as I would write my own software, I think I could make it work.

There are two interesting models, the Wena Pro and the Wena3. The Wena3 has a heartrate monitor, bigger screen and USB-C, but I think I prefer the simpler Wena Pro design.

Go Retro?

There have been smartwatches around since the 80s, and the specifications I’m interested in are hardly cutting edge! Perhaps the closest device to what I’m looking for is actually from 2003, the Timex m851.

It’s a programmable digital watch based on the 8-bit SEIKO 88349 CPU, and 2kb of RAM. Here is the developer manual, and here is the CPU manual. Essentially, you can write “wrist apps” in S1C88 assembly, and sync it over USB with a cable.

There is apparently also a C compiler and simulator available, although I suspect writing in C might not be an optimal use of the constrained system resources!

The simulator seems capable, here are screenshots from the manual.

The documentation says the battery needs to be replaced every 2 years, which is totally acceptable.

There was also a modest enthusiast community at some point, who wrote some interesting apps. Amusingly, this watch uses the same CPU as a popular retro gaming device (the Pokemon mini), so retro game hackers have already documented the toolchain and device quirks, and even built emulators I might be able to repurpose!

Conclusion

Well, I’ve ordered a Wena Pro, a Sensor Watch and a (preowned) Timex M851.

I’ll try to get all three working and post a part two with my developer review!

Putting debugging symbols back into stripped binaries.

Tavis Ormandy

Unstripping Stripped Binaries

Tavis Ormandy

$Id: a07cf90837a3c4373b82d6724b97593810766af7 $

Intro

I’ve written before about how much I enjoy vintage software. Lately I’ve been tinkering with WordPerfect for UNIX.

It’s working great, combined with Lotus 1-2-3 you can have a full-featured office suite in an xterm! 😂

Debugging

These are 30 year old stripped binaries that I’ve somehow managed to patch into a working state. As you can imagine, when something doesn’t work, tracking down what went wrong can be a real challenge.

What I really want is to take the names and types I’ve figured out from my disassembler, and make them visible to gdb.

I’ve found a simple solution to this that’s been working well for me, here are some notes on it.

Stabs

You’re probably familiar with DWARF, the debugging format used everywhere in Linux. DWARF is really neat, it’s capable of expressing the most complex locations and types possible.

In this context, “location” means explaining to a debugger how to find a variable. In some cases that’s really easy, but it gets complicated fast. Perhaps the variable moves in and out of registers or requires complex address calculation logic (e.g. a bitfield struct member stored in a stack frame).

Figuring out these locations requires the debugger to execute little bytecode programs called DWARF expressions – crazy stuff!

Before DWARF, STABS (Symbol TABle Strings), was the predominant debugging format. STABS takes simple locations and types, encodes them into strings, and then stuffs them into the symbol table. It’s not very elegant, but it worked.

STABS is entirely obsolete, DWARF is superior in every way. However… STABS does have one benefit that DWARF doesn’t – expressing simple stuff (very simple) is so easy you can even do it by hand…

Tools

All the GNU tools still have native support for STABS. In fact, I’ve noticed the GNU assembler even has pseudo-instructions that you can write manually:

https://sourceware.org/binutils/docs/as/Stab.html:

There are three directives that begin ‘.stab’. All emit symbols (see Symbols), for use by symbolic debuggers. Up to five fields are required:

string
This is the symbol’s name.

type
An absolute expression. The symbol’s type is set to the low 8 bits of this expression. Any bit pattern is permitted, but ld and debuggers choke on silly bit patterns.

[…]

value
An absolute expression which becomes the symbol’s value.

If you just want basic native types and simple locations, nothing could be simpler than this.

Examples

Let’s say I’ve figured out there is a function at 0x8005bba like this:

void example(unsigned int *foo, unsigned long bar);

The stab to declare that is just this:

.stabs "example:f-11", N_FUN, 0, 0, 0x8005bba

Here f means this is a function, and -11 is the pre-defined type for void. If you only want to use basic types, you don’t even have to define them!

Here is a list of some of the predefined types that GDB recognizes:

Num	Type
-1	int, 32 bit signed integral type.
-2	char, 8 bit type holding a character.
-4	long, 32 bit signed integral type.
-5	unsigned char, 8 bit unsigned integral type.
-6	signed char, 8 bit signed integral type.
-7	unsigned short, 16 bit unsigned integral type.
-8	unsigned int, 32 bit unsigned integral type.
-9	unsigned, 32 bit unsigned integral type.
-10	unsigned long, 32 bit unsigned integral type.
-11	void, type indicating the lack of a value.
-31	long long, 64 bit signed integral type.
-32	unsigned long long, 64 bit unsigned integral type.

There are some confusing choices in there, but there are about 30 predefined types that GDB knows about.

Notice I said “knows about”, that’s because there is no STABS standard, just analyses of what crazy incompatible things all the 90s UNIX vendors were doing!

This cygnus document on stabs is great; it’s well written and thorough, but occasional glimpses of frustration with Sun and IBM for their incompatible undocumented extensions seep through.

https://sourceware.org/gdb/onlinedocs/stabs.pdf

Parameters

Okay, functions are working, what about function parameters? If they’re one of the predefined types and this is a standard cdecl function, that’s easy too!

.stabs "foo:p*-8", N_PSYM, 0, 0, 8
.stabs "bar:p-10", N_PSYM, 0, 0, 12

This means there is a parameter foo, a pointer to an unsigned int at bp+8 and an unsigned long called bar at bp+12.

I wrote some gas macros to make this less laborious, and now I can just write this:

function main, 0x8128000, %int
    param argc, %int
    param argv, ** %char
    param envp, ** %char

Pretty neat!

You don’t even need to specify the offset – gas macros can store counters between invocations, so I just keep incrementing it for each new parameter, then reset it when you start a new function!

They’re really simple, they look like this (some code ommitted):

.macro function name, address, ptr=, type
    .set _arg, 0
    .stabs "\name:f\ptr\type", N_FUN, 0, 0, \address
.endm

.macro param name, ptr=, type
    .set _arg, _arg + 1
    .stabs "\name:p\ptr\type", N_PSYM, 0, 0, PARAM_SIZE+_arg*PARAM_SIZE
.endm

It works great, here is a sample gdb session. You can see I set breakpoints, examine values, print types, and so on.

(gdb) add-symbol-file symbols.dbg
(gdb) pt rddec
type = boolean (char **, unsigned short *)
(gdb) x/i rddec
0x81dac90 <rddec>:   push   ebp
(gdb) b rddec
Breakpoint 1 at 0x81dac96
(gdb) c
Breakpoint 1, 0x081dac96 in rddec (numstr=0xffffc044, num=0x8350dbe)
(gdb) pt numstr
type = char **
(gdb) p *numstr
$1 = 0xffffc048 "06"

Even conditional breakpoints on parameter values work, it’s just like unstripping the binary.

In order to get a symbol file, I assemble them like this:

$ as --32 -gstabs -o wp.o wp.s
$ as --32 -gstabs -o types.o types.s
$ ld -m elf_i386 -shared -Tdata=082d7938 -Ttext=0804a5f0 -Tbss=083377c0 -o wp.dbg wp.o types.o
$ strip --only-keep-debug wp.dbg

It’s important to have the sections lined up with the target binary, or gdb will get confused.

Now you can just do this:

(gdb) add-symbol-file wp.dbg
add symbol table from file "wp.dbg"
Reading symbols from wp.dbg...

I haven’t tried it, but I bet objcopy --add-gnu-debuglink would work too!

Usage

I can write these symbols manually when I need to, but also wrote a quick script to export these from my disassembler.

The output is just thousands of lines like this:

function g_init, 0x0815EFA0, %int
function dflt_init, 0x0815F1A0, %int
function tool_init, 0x0815F620, %int
function g_close, 0x0815F630, %int
function g_inits, 0x0815F710, %int
function g_dint, 0x0815F720, %int
function g_dot, 0x0815F730, %int
function sub_8160290, 0x08160290, * %void

Putting it all together, my stripped binary now has symbols and parameter information in gdb, woohoo!

Breakpoint 2, 0x0815f1b9 in dflt_init ()
(gdb) bt
#0  0x0815f1b9 in dflt_init ()
#1  0x0815f02a in g_init ()
#2  0x0815acfc in int_dsp_xxx ()
#3  0x0814f8ad in gshow_init ()

Symbol Porting

I actually have a huge advantage here that I didn’t mention.

After a few of hours digging around on archive.org, I found a binary for an older version of WordPerfect that wasn’t stripped! It must have been a mistake while building the final RTM binaries.

$ ls -l wp
-rw------- 1 taviso taviso 4.5M Jul 31  1996 wp
$ file wp
wp: ELF 32-bit MSB executable, SPARC, version 1 (SYSV), dynamically linked, interpreter /usr/lib/ld.so.1, not stripped

Unfortunately it’s not only an older version, but also for a different architecture and operating system, Solaris SPARC. BinDiff does do pretty well at matching these symbols to my i386 binary.

I’ve found that people are sometimes surprised this works! Most of the clever tricks BinDiff uses to match functions between two binaries are actually architecture neutral. That means that If you have a stripped binary for ARM and an unstripped version of the same binary for MIPS, BinDiff can figure out which functions are which for you.

String References

Some of the techniques used are easy to understand, like string reference matching. If only a single function references the string “error in function foo”, then it doesn’t matter if it’s SPARC or x86, clearly this is the same function, so you just learned a symbol name!

CFG

It’s rare it’s that easy though, and BinDiff is full of insanely clever tricks. Many of them involve CFG matching.

If you generate a graph of all the function calls in two similar binaries, there should be lots of matches.

If function foo calls function bar, which calls function baz and quux, then you don’t need to know what architecture this is or what the functions do. You can simply find the same unique graph and be confident these are the same functions!

DOT Code…

digraph {
    rankdir="LR"
    sub_123 -> sub_456
    sub_456 -> sub_789
    sub_456 -> sub_abc

    foo -> bar
    bar -> baz
    bar -> quux
}

This is just a trivial example, you can read about some more of the BinDiff matching strategies here.

These tricks combined have let me debug and track down some pretty gnarly issues!

Code

If the idea of using stabs to debug stripped code has piqued your interest, you can take a look at my macros and code on github here.

Hopefully this could be a useful base for someone elses project!

I solved a mystery in an old UNIX print ad.

Tavis Ormandy

WordPerfect for UNIX

Tavis Ormandy

$Id: a07cf90837a3c4373b82d6724b97593810766af7 $

Intro

I found this interesting print-ad for WordPerfect for UNIX, apparently the best UNIX product of 1992! (Very) Belated congratulations Corel!

However, this section intrigued me:

But what if you don’t go for the GUI? Just choose the Character version, and
depending on your hardware, you’ll get great-looking features rarely seen on a
character terminal. Like document preview, column layout, and other important
features such as easy-to-use drop down menus and easy-to-understand help
screens.

I understand “column layout” and “drop down menus”, but what exactly does “document preview” mean?

Mystery

Well, a weekend of hacking later and I have it working on modern Linux!

The answer is it can generate Print Previews using Sixels!

I find this hilarious.

It requires a DEC VT340 compatible terminal (XTerm works fine). I think it needs some tweaking to scale the image, but it basically works!

Here is a screenshot of the Reveal Codes view that WordPerfect users always talk about, and the generated preview!

Conclusion

It took a lot of tweaking to get it to work on modern Linux, I’ve put my patches online here.

I rediscovered some lost UNIX software.

Tavis Ormandy

Lotus 1-2-3 For Linux

Tavis Ormandy

$Id: a07cf90837a3c4373b82d6724b97593810766af7 $

Intro

It’s an exciting time in the Lotus 1-2-3 enthusiast community – that was a joke, there is no enthusiast community, it’s just me! 😆

It really is an exciting time though – that part isn’t a joke!

There have been some major developments in the last few weeks, and I guess that’s pretty unusual for 30 year old abandonware.

I’ll cut to the chase; through a combination of unlikely discoveries, crazy hacks and the 90s BBS warez scene I’ve been able to port Lotus 1-2-3 natively to Linux – an operating system that literally didn’t exist when 1-2-3 was released!¹

If you want to hear how a proprietary application could be ported to new operating systems 30 years after release, read on.

Background

I really like Lotus 1-2-3, I even maintain a driver to make sure it works well on modern systems. I had to reverse engineer the driver api to make that happen, but it works beautifully.

Getting that driver working was quite an adventure, but there is still one piece of the puzzle missing: add-ins. 1-2-3 was designed to be extensible with plugins (or “add-ins”) – in theory you could add support for modern spreadsheet functions or integrate with Google Finance or something!

The problem is add-ins had to be written in a special language called LPL, and unfortunately the compiler and SDK have been lost. This is not really surprising, this was a niche product and Lotus didn’t just give the SDK away – they charged $395 for it. I can’t imagine many people have a copy just lying around.

There were lots of commercial plugins available in the heyday, some were really impressive. I emailed a few developers who worked on some to see if they had any old backups, the answer was always the same – 1-2-3 was the biggest name in software, nobody thought it was going anywhere, so why would they keep backups?

There were also two third-party books written about LPL, I managed to track them down from old libraries. I like the cover on this one, very serious business. It’s so frustating, I can see screenshots of the debugger, compiler output and sample add-ins - but just can’t do anything with it without the SDK!

Progress

Fast forward a year or two, and I found someone who used to be a sysop in the ’90s BBS scene. Check out his website, he still has a catalogue of NFOs and a telnet BBS online if you want to see some rad ANSI art!

He had kept tape backups from old BBS systems, and was able to recover a warez copy of the SDK – unbelievable! It actually worked, I was able to build a few sample plugins!

You can download the ADK right here, and here is a sample LPL program.

This would already have been enough to keep me entertained for a while, but it doesn’t stop there. It turns out that the BBS also had a warez copy of Lotus 1-2-3 for UNIX. This was widely thought to be lost – I’m told it couldn’t compete with a more popular UNIX office suite called SCO Professional, so there were not many copies sold.

I didn’t really have any use for it, but I’m definitely curious enough to poke around on the installation media to see what’s in it!

Lotus 1-2-3 for UNIX

The warez release was a bunch of TD0 files, that’s a format I’ve never seen before – apparently it’s an old compressed disk image format from the 80s. I found this page which recommends using samdisk to convert it to a raw disk image.

$ ls
123UNIX1.TD0  123UNIX2.TD0  123UNIX3.TD0  123UNIX4.TD0  123UNIX5.TD0  LEGAL.NFO  WHATITIS
$ file *.TD0
123UNIX1.TD0: floppy image data (TeleDisk)
123UNIX2.TD0: floppy image data (TeleDisk)
123UNIX3.TD0: floppy image data (TeleDisk)
123UNIX4.TD0: floppy image data (TeleDisk)
123UNIX5.TD0: floppy image data (TeleDisk)

That seemed promising, samdisk builds and seems to work! I’ve uploaded the full images onto the Internet Archive in case anyone else is curious enough to take a look.

$ samdisk info 123UNIX1.TD0
[123UNIX1.TD0]
 Type:   TD0
 Size:   80 Cyls, 2 Heads

 created : 1991-06-22 20:24:04
Lotus 1-2-3 for UNIX System V
$ for i in *.TD0; do samdisk copy ${i} ${i/.TD0/.RAW}; done
Wrote 80 cyls, 2 heads, 18 sectors,  512 bytes/sector = 1474560 bytes
Wrote 80 cyls, 2 heads, 18 sectors,  512 bytes/sector = 1474560 bytes
Wrote 80 cyls, 2 heads, 18 sectors,  512 bytes/sector = 1474560 bytes
Wrote 80 cyls, 2 heads, 18 sectors,  512 bytes/sector = 1474560 bytes
Wrote 80 cyls, 2 heads, 18 sectors,  512 bytes/sector = 1474560 bytes

$ file *.RAW
123UNIX1.RAW: tar archive
123UNIX2.RAW: ASCII cpio archive (pre-SVR4 or odc)
123UNIX3.RAW: ASCII cpio archive (pre-SVR4 or odc)
123UNIX4.RAW: ASCII cpio archive (pre-SVR4 or odc)
123UNIX5.RAW: ASCII cpio archive (pre-SVR4 or odc)

I know that UNIX software was usually distributed as raw archives, and you were expected to insert the diskette and run something like tar -C / -xvf /dev/fd0, so these files look right. I think this is smart, why waste precious bytes on a filesystem?

$ tar xf 123UNIX1.RAW
$ for i in 123UNIX{2..5}.RAW; do cpio -id < $i; done
1555 blocks
2606 blocks
2510 blocks
2481 blocks

Full Directory Listing

$ tree --prune -Dh
.
├── [ 512 May 20 13:53]  lotus
│   ├── [ 512 May 20 13:53]  123.v10
│   │   ├── [ 512 May 20 13:53]  cbd
│   │   │   ├── [2.4K May 20 13:53]  dfbcp437.bun.z
│   │   │   ├── [2.4K May 20 13:53]  dfbcp850.bun.z
│   │   │   ├── [3.0K May 20 13:53]  l13cUSA3.cbd
│   │   │   ├── [3.0K May 20 13:53]  l13cUSF3.cbd
│   │   │   ├── [3.0K May 20 13:53]  l13cUSL3.cbd
│   │   │   ├── [1.6K May 20 13:53]  l_ascii.bun.z
│   │   │   ├── [1.8K May 20 13:53]  l_latin1.bun.z
│   │   │   └── [ 874 May 20 13:53]  l_lrf.bun.z
│   │   ├── [ 512 May 20 13:53]  fonts
│   │   │   ├── [4.6K Sep 20  1990]  aa012laa.lrf.z
│   │   │   ├── [4.3K Sep 20  1990]  aa012lfa.lrf.z
│   │   │   ├── [5.4K Sep 20  1990]  aa012lha.lrf.z
│   │   │   ├── [5.9K Sep 20  1990]  aa017laa.lrf.z
│   │   │   ├── [7.8K Sep 20  1990]  aa017lia.lrf.z
│   │   │   ├── [7.8K Sep 20  1990]  aa024lfa.lrf.z
│   │   │   ├── [ 11K Sep 20  1990]  aa024lha.lrf.z
│   │   │   ├── [8.8K Sep 20  1990]  aa025laa.lrf.z
│   │   │   ├── [ 11K Sep 20  1990]  aa030laa.lrf.z
│   │   │   ├── [6.6K Sep 20  1990]  aa030lga.lrf.z
│   │   │   ├── [ 12K Sep 20  1990]  aa033laa.lrf.z
│   │   │   ├── [ 18K Sep 20  1990]  aa033lia.lrf.z
│   │   │   ├── [ 22K Sep 20  1990]  aa050laa.lrf.z
│   │   │   ├── [ 29K Sep 20  1990]  aa060laa.lrf.z
│   │   │   ├── [ 15K Sep 20  1990]  aa060lga.lrf.z
│   │   │   ├── [ 53K Sep 20  1990]  aa100laa.lrf.z
│   │   │   ├── [5.4K Sep 20  1990]  ac016lfa.lrf.z
│   │   │   ├── [7.1K Sep 20  1990]  ac016lha.lrf.z
│   │   │   ├── [6.1K Sep 20  1990]  ac017laa.lrf.z
│   │   │   ├── [8.1K Sep 20  1990]  ac022laa.lrf.z
│   │   │   ├── [ 11K Sep 20  1990]  ac022lia.lrf.z
│   │   │   ├── [ 13K Sep 20  1990]  ac033laa.lrf.z
│   │   │   ├── [ 16K Sep 20  1990]  ac040laa.lrf.z
│   │   │   ├── [9.4K Sep 20  1990]  ac040lga.lrf.z
│   │   │   ├── [ 34K Sep 20  1990]  ac066laa.lrf.z
│   │   │   ├── [4.7K Sep 20  1990]  ai012laa.lrf.z
│   │   │   ├── [4.3K Sep 20  1990]  ai012lfa.lrf.z
│   │   │   ├── [5.5K Sep 20  1990]  ai012lha.lrf.z
│   │   │   ├── [6.1K Sep 20  1990]  ai017laa.lrf.z
│   │   │   ├── [8.0K Sep 20  1990]  ai017lia.lrf.z
│   │   │   ├── [7.9K Sep 20  1990]  ai024lfa.lrf.z
│   │   │   ├── [ 11K Sep 20  1990]  ai024lha.lrf.z
│   │   │   ├── [9.0K Sep 20  1990]  ai025laa.lrf.z
│   │   │   ├── [ 11K Sep 20  1990]  ai030laa.lrf.z
│   │   │   ├── [6.5K Sep 20  1990]  ai030lga.lrf.z
│   │   │   ├── [ 12K Sep 20  1990]  ai033laa.lrf.z
│   │   │   ├── [ 19K Sep 20  1990]  ai033lia.lrf.z
│   │   │   ├── [ 23K Sep 20  1990]  ai050laa.lrf.z
│   │   │   ├── [ 29K Sep 20  1990]  ai060laa.lrf.z
│   │   │   ├── [ 15K Sep 20  1990]  ai060lga.lrf.z
│   │   │   ├── [ 55K Sep 20  1990]  ai100laa.lrf.z
│   │   │   ├── [5.5K Sep 20  1990]  aj016lfa.lrf.z
│   │   │   ├── [7.1K Sep 20  1990]  aj016lha.lrf.z
│   │   │   ├── [6.2K Sep 20  1990]  aj017laa.lrf.z
│   │   │   ├── [8.1K Sep 20  1990]  aj022laa.lrf.z
│   │   │   ├── [ 11K Sep 20  1990]  aj022lia.lrf.z
│   │   │   ├── [ 13K Sep 20  1990]  aj033laa.lrf.z
│   │   │   ├── [ 17K Sep 20  1990]  aj040laa.lrf.z
│   │   │   ├── [9.4K Sep 20  1990]  aj040lga.lrf.z
│   │   │   ├── [ 35K Sep 20  1990]  aj066laa.lrf.z
│   │   │   ├── [3.6K Sep 20  1990]  bc008lba.lrf.z
│   │   │   ├── [4.0K Sep 20  1990]  bc010lba.lrf.z
│   │   │   ├── [5.3K Sep 20  1990]  bc012lea.lrf.z
│   │   │   ├── [5.5K Sep 20  1990]  bc014lda.lrf.z
│   │   │   ├── [5.8K Sep 20  1990]  bc016lba.lrf.z
│   │   │   ├── [5.8K Sep 20  1990]  bc016lca.lrf.z
│   │   │   ├── [5.7K Sep 20  1990]  bc016lda.lrf.z
│   │   │   ├── [6.1K Sep 20  1990]  bc018lba.lrf.z
│   │   │   ├── [6.8K Sep 20  1990]  bc021laa.lrf.z
│   │   │   ├── [ 12K Sep 20  1990]  bc031lca.lrf.z
│   │   │   ├── [ 12K Sep 20  1990]  bc031lda.lrf.z
│   │   │   ├── [ 14K Sep 20  1990]  bc042laa.lrf.z
│   │   │   ├── [1.7K Sep 20  1990]  bo008lda.lrf.z
│   │   │   ├── [2.6K Sep 20  1990]  bo014lca.lrf.z
│   │   │   ├── [2.6K Sep 20  1990]  bo014lda.lrf.z
│   │   │   ├── [3.1K Sep 20  1990]  bo016lba.lrf.z
│   │   │   └── [3.5K Sep 20  1990]  bo019laa.lrf.z
│   │   ├── [ 512 May 20 13:53]  hlp
│   │   │   └── [ 512 May 20 13:53]  USA-English
│   │   │       ├── [407K May 20 13:53]  123_sysV.hlp.z
│   │   │       ├── [7.3K May 20 13:53]  keyedit.hlp.z
│   │   │       └── [8.2K May 20 13:53]  setup123.hlp.z
│   │   ├── [ 512 May 20 13:53]  keymaps
│   │   │   ├── [ 512 May 20 13:53]  a
│   │   │   │   └── [3.7K May 20 13:53]  AT386.z
│   │   │   ├── [ 512 May 20 13:53]  p
│   │   │   │   └── [2.7K May 20 13:53]  PC-NFS.z
│   │   │   ├── [ 512 May 20 13:53]  s
│   │   │   │   ├── [2.6K May 20 13:53]  sco386-101.z
│   │   │   │   ├── [3.7K May 20 13:53]  sun.z
│   │   │   │   ├── [3.7K May 20 13:53]  sysV386-101.z
│   │   │   │   └── [5.7K May 20 13:53]  sysV386-xterm
│   │   │   ├── [ 512 May 20 13:53]  v
│   │   │   │   ├── [4.7K May 20 13:53]  vt100.z
│   │   │   │   └── [ 10K May 20 13:53]  vtxxx.z
│   │   │   └── [ 512 May 20 13:53]  w
│   │   │       ├── [3.2K May 20 13:53]  wyse-pce.z
│   │   │       ├── [ 14K May 20 13:53]  wyse50-lts123
│   │   │       └── [5.9K May 20 13:53]  wyse50.z
│   │   ├── [ 512 May 20 13:53]  pbd
│   │   │   ├── [ 604 Sep 20  1990]  egas25ca.vbd.z
│   │   │   ├── [ 604 Sep 20  1990]  egas25ci.vbd.z
│   │   │   ├── [ 605 Sep 20  1990]  egas25cs.vbd.z
│   │   │   ├── [ 602 Sep 20  1990]  egas25la.vbd.z
│   │   │   ├── [ 602 Sep 20  1990]  egas25li.vbd.z
│   │   │   ├── [ 604 Sep 20  1990]  egas25ls.vbd.z
│   │   │   ├── [ 574 Sep 20  1990]  egas25ma.vbd.z
│   │   │   ├── [ 574 Sep 20  1990]  egas25mi.vbd.z
│   │   │   ├── [ 576 Sep 20  1990]  egas25ms.vbd.z
│   │   │   ├── [ 592 Sep 20  1990]  egas25pa.vbd.z
│   │   │   ├── [ 592 Sep 20  1990]  egas25pi.vbd.z
│   │   │   ├── [ 593 Sep 20  1990]  egas25ps.vbd.z
│   │   │   ├── [ 604 Sep 20  1990]  egas43ca.vbd.z
│   │   │   ├── [ 604 Sep 20  1990]  egas43ci.vbd.z
│   │   │   ├── [ 605 Sep 20  1990]  egas43cs.vbd.z
│   │   │   ├── [ 574 Sep 20  1990]  egas43ma.vbd.z
│   │   │   ├── [ 574 Sep 20  1990]  egas43mi.vbd.z
│   │   │   ├── [ 575 Sep 20  1990]  egas43ms.vbd.z
│   │   │   ├── [ 588 Sep 20  1990]  egas43pa.vbd.z
│   │   │   ├── [ 588 Sep 20  1990]  egas43pi.vbd.z
│   │   │   ├── [ 589 Sep 20  1990]  egas43ps.vbd.z
│   │   │   ├── [2.4K Sep 20  1990]  epdtfx.pbd.z
│   │   │   ├── [2.4K Sep 20  1990]  epdtmx.pbd.z
│   │   │   ├── [3.7K Sep 20  1990]  eplql15p.pbd.z
│   │   │   ├── [3.3K Sep 20  1990]  eplql25m.pbd.z
│   │   │   ├── [4.2K Sep 20  1990]  eplql25p.pbd.z
│   │   │   ├── [3.4K Sep 20  1990]  eplql2mc.pbd.z
│   │   │   ├── [4.3K Sep 20  1990]  eplql2pc.pbd.z
│   │   │   ├── [2.9K Sep 20  1990]  eplql80m.pbd.z
│   │   │   ├── [3.6K Sep 20  1990]  eplql80p.pbd.z
│   │   │   ├── [1020 Sep 20  1990]  gc_bs.pbd
│   │   │   ├── [ 754 Sep 20  1990]  gc_nobs.pbd
│   │   │   ├── [3.9K Sep 20  1990]  hplj1flo.pbd
│   │   │   ├── [2.6K Sep 20  1990]  hplj1jlo.pbd.z
│   │   │   ├── [2.6K Sep 20  1990]  hplj1nol.pbd
│   │   │   ├── [5.2K Sep 20  1990]  hplj1zlo.pbd.z
│   │   │   ├── [4.3K Sep 20  1990]  hplj2fhi.pbd.z
│   │   │   ├── [3.4K Sep 20  1990]  hplj2flo.pbd.z
│   │   │   ├── [4.5K Sep 20  1990]  hplj2jhi.pbd.z
│   │   │   ├── [3.6K Sep 20  1990]  hplj2jlo.pbd.z
│   │   │   ├── [3.9K Sep 20  1990]  hplj2noh.pbd.z
│   │   │   ├── [3.0K Sep 20  1990]  hpljnolo.pbd.z
│   │   │   ├── [4.3K Sep 20  1990]  hpljpfhi.pbd.z
│   │   │   ├── [3.4K Sep 20  1990]  hpljpflo.pbd.z
│   │   │   ├── [3.7K Sep 20  1990]  hpljpjhi.pbd.z
│   │   │   ├── [2.9K Sep 20  1990]  hpljpjlo.pbd.z
│   │   │   ├── [3.1K Sep 20  1990]  hpljpnoh.pbd.z
│   │   │   ├── [2.3K Sep 20  1990]  hpljpnol.pbd.z
│   │   │   ├── [6.3K Sep 20  1990]  hpljzhi.pbd.z
│   │   │   ├── [5.4K Sep 20  1990]  hpljzlo.pbd.z
│   │   │   ├── [2.9K Sep 20  1990]  hppjpapr.pbd.z
│   │   │   ├── [2.9K Sep 20  1990]  hppjtran.pbd.z
│   │   │   ├── [ 976 Sep 20  1990]  hrcf25ma.vbd.z
│   │   │   ├── [ 976 Sep 20  1990]  hrcf25mi.vbd.z
│   │   │   ├── [ 977 Sep 20  1990]  hrcf25ms.vbd.z
│   │   │   ├── [1006 Sep 20  1990]  hrcs43ma.vbd
│   │   │   ├── [1006 Sep 20  1990]  hrcs43mi.vbd
│   │   │   ├── [1006 Sep 20  1990]  hrcs43ms.vbd
│   │   │   ├── [7.6K Sep 20  1990]  ps00alw0.pbd.z
│   │   │   ├── [ 12K Sep 20  1990]  ps00alwp.pbd.z
│   │   │   ├── [ 604 Sep 20  1990]  vgas25ca.vbd.z
│   │   │   ├── [ 604 Sep 20  1990]  vgas25ci.vbd.z
│   │   │   ├── [ 606 Sep 20  1990]  vgas25cs.vbd.z
│   │   │   ├── [ 574 Sep 20  1990]  vgas25ma.vbd.z
│   │   │   ├── [ 573 Sep 20  1990]  vgas25mi.vbd.z
│   │   │   ├── [ 576 Sep 20  1990]  vgas25ms.vbd.z
│   │   │   ├── [ 601 Sep 20  1990]  vgas34ca.vbd.z
│   │   │   ├── [ 601 Sep 20  1990]  vgas34ci.vbd.z
│   │   │   ├── [ 602 Sep 20  1990]  vgas34cs.vbd.z
│   │   │   ├── [ 575 Sep 20  1990]  vgas34ma.vbd.z
│   │   │   ├── [ 575 Sep 20  1990]  vgas34mi.vbd.z
│   │   │   ├── [ 575 Sep 20  1990]  vgas43ms.vbd.z
│   │   │   ├── [ 602 Sep 20  1990]  vgas60ca.vbd.z
│   │   │   ├── [ 602 Sep 20  1990]  vgas60ci.vbd.z
│   │   │   ├── [ 603 Sep 20  1990]  vgas60cs.vbd.z
│   │   │   ├── [ 575 Sep 20  1990]  vgas60ma.vbd.z
│   │   │   └── [ 575 Sep 20  1990]  vgas60mi.vbd.z
│   │   ├── [ 512 May 20 13:53]  ri
│   │   │   └── [ 512 May 20 13:53]  USA-English
│   │   │       ├── [1001 May 20 13:53]  123.ri.z
│   │   │       ├── [ 602 May 20 13:53]  dr123txt.ri
│   │   │       ├── [1.9K May 20 13:53]  inst_dl.ri
│   │   │       ├── [5.3K May 20 13:53]  keyedit.ri.z
│   │   │       ├── [2.0K May 20 13:53]  l123smp3.ri
│   │   │       ├── [ 49K May 20 13:53]  l123txt3.ri.z
│   │   │       ├── [ 844 May 20 13:53]  license.ri.z
│   │   │       ├── [6.9K May 20 13:53]  printer.ri.z
│   │   │       ├── [2.6K May 20 13:53]  prsetup.ri.z
│   │   │       └── [ 23K May 20 13:53]  setup123.ri.z
│   │   ├── [ 512 May 20 13:53]  smpfiles
│   │   │   ├── [1.3K May 20 13:53]  ACCTG.WK3.z
│   │   │   ├── [1.5K May 20 13:53]  CONSOL.WK3.z
│   │   │   ├── [1.7K May 20 13:53]  DATA.WK3.z
│   │   │   ├── [3.2K May 20 13:53]  DBT13S.WK3.z
│   │   │   ├── [3.2K May 20 13:53]  DBT14S.WK3.z
│   │   │   ├── [1.3K May 20 13:53]  INC10S.WK3.z
│   │   │   ├── [3.0K May 20 13:53]  INC11S.WK3.z
│   │   │   ├── [4.1K May 20 13:53]  INC12S.WK3.z
│   │   │   ├── [3.7K May 20 13:53]  INC16S.WK3.z
│   │   │   ├── [ 619 May 20 13:53]  INC2S.WK3
│   │   │   ├── [ 965 May 20 13:53]  INC4S.WK3
│   │   │   ├── [ 712 May 20 13:53]  INC5S.WK3.z
│   │   │   ├── [1.8K May 20 13:53]  INC6S.WK3
│   │   │   ├── [1.2K May 20 13:53]  INC7S.WK3.z
│   │   │   ├── [1.4K May 20 13:53]  INC8S.WK3.z
│   │   │   ├── [1.7K May 20 13:53]  INC9S.WK3.z
│   │   │   ├── [3.8K May 20 13:53]  MAC17S.WK3.z
│   │   │   ├── [2.0K May 20 13:53]  MFG.WK3
│   │   │   ├── [1.8K May 20 13:53]  SALES.WK3
│   │   │   ├── [4.7K May 20 13:53]  SAMPMACS.WK3.z
│   │   │   ├── [1.5K May 20 13:53]  SHOES.WK3.z
│   │   │   ├── [2.6K May 20 13:53]  SMPSCHED.WK3.z
│   │   │   ├── [1.3K May 20 13:53]  SUM1988S.WK3.z
│   │   │   ├── [1.5K May 20 13:53]  SUMMARY.WK3
│   │   │   ├── [2.2K May 20 13:53]  TABLES.WK3.z
│   │   │   ├── [6.4K May 20 13:53]  consale.wk1.z
│   │   │   ├── [6.3K May 20 13:53]  consale.wk3.z
│   │   │   ├── [6.8K May 20 13:53]  consaler.wk1.z
│   │   │   ├── [7.2K May 20 13:53]  income.wk3.z
│   │   │   ├── [8.7K May 20 13:53]  income2.wk3.z
│   │   │   ├── [8.7K May 20 13:53]  income3.wk3.z
│   │   │   ├── [8.7K May 20 13:53]  income4.wk3.z
│   │   │   ├── [7.2K May 20 13:53]  incomer.wk3.z
│   │   │   ├── [ 512 May 20 13:53]  intro
│   │   │   │   ├── [ 874 Sep 20  1990]  chicago.wk3.z
│   │   │   │   ├── [1.8K Sep 20  1990]  debugmac.wk3
│   │   │   │   ├── [ 886 Sep 20  1990]  houston.wk3.z
│   │   │   │   ├── [ 799 Sep 20  1990]  less1ex1.wk3
│   │   │   │   ├── [ 565 Sep 20  1990]  less1ex2.wk3.z
│   │   │   │   ├── [ 581 Sep 20  1990]  less2ex1.wk3.z
│   │   │   │   ├── [ 877 Sep 20  1990]  less2ex2.wk3
│   │   │   │   ├── [4.1K Sep 20  1990]  less3ex2.wk3.z
│   │   │   │   ├── [ 706 Sep 20  1990]  less4ex1.wk3.z
│   │   │   │   ├── [ 834 Sep 20  1990]  less4ex2.wk3.z
│   │   │   │   ├── [ 798 Sep 20  1990]  less5ex1.wk3.z
│   │   │   │   ├── [ 811 Sep 20  1990]  less5ex2.wk3.z
│   │   │   │   ├── [ 931 Sep 20  1990]  less5ex3.wk3.z
│   │   │   │   ├── [ 690 Sep 20  1990]  less6ex1.wk3.z
│   │   │   │   ├── [ 781 Sep 20  1990]  less6ex2.wk3.z
│   │   │   │   ├── [ 839 Sep 20  1990]  less7ex1.wk3.z
│   │   │   │   ├── [ 856 Sep 20  1990]  less7ex2.wk3.z
│   │   │   │   ├── [ 896 Sep 20  1990]  less8ex2.wk3.z
│   │   │   │   ├── [1.7K Sep 20  1990]  less9ex1.wk3
│   │   │   │   ├── [1.8K Sep 20  1990]  macros.wk3
│   │   │   │   ├── [1.9K Sep 20  1990]  menuexam.wk3
│   │   │   │   ├── [ 774 Sep 20  1990]  menumac.wk3
│   │   │   │   ├── [ 647 Sep 20  1990]  mis.wk3.z
│   │   │   │   ├── [ 939 Sep 20  1990]  mlibrary.wk3
│   │   │   │   ├── [ 890 Sep 20  1990]  nyc.wk3.z
│   │   │   │   ├── [1.8K Sep 20  1990]  parts.wk3.z
│   │   │   │   ├── [ 698 Sep 20  1990]  planning.wk3.z
│   │   │   │   ├── [2.0K Sep 20  1990]  q2graph.wk3
│   │   │   │   ├── [1.3K Sep 20  1990]  q2graph2.wk3.z
│   │   │   │   ├── [ 690 Sep 20  1990]  q2inc.wk3
│   │   │   │   ├── [ 763 Sep 20  1990]  q2inc2.wk3.z
│   │   │   │   ├── [ 770 Sep 20  1990]  q2mac.wk3.z
│   │   │   │   ├── [1.2K Sep 20  1990]  q2macro.wk3.z
│   │   │   │   ├── [1.3K Sep 20  1990]  q2price.wk3.z
│   │   │   │   ├── [ 674 Sep 20  1990]  research.wk3.z
│   │   │   │   ├── [1.9K Sep 20  1990]  salary.wk3
│   │   │   │   ├── [ 878 Sep 20  1990]  sanfran.wk3.z
│   │   │   │   └── [2.0K Sep 20  1990]  supplier.wk3.z
│   │   │   ├── [ 12K May 20 13:53]  menu2.wk3.z
│   │   │   ├── [ 12K May 20 13:53]  menu3.wk3.z
│   │   │   ├── [ 11K May 20 13:53]  menu4.wk3.z
│   │   │   └── [ 512 May 20 13:53]  nf
│   │   │       ├── [1.4K Sep 20  1990]  acctg.wk3.z
│   │   │       ├── [5.3K Sep 20  1990]  actual.wk3.z
│   │   │       ├── [ 871 Sep 20  1990]  asset.wk3.z
│   │   │       ├── [1023 Sep 20  1990]  bud91.wk3
│   │   │       ├── [5.0K Sep 20  1990]  budget.wk3.z
│   │   │       ├── [1.5K Sep 20  1990]  calcex.wk3.z
│   │   │       ├── [2.9K Sep 20  1990]  consol.wk3.z
│   │   │       ├── [3.2K Sep 20  1990]  data.wk3.z
│   │   │       ├── [4.9K Sep 20  1990]  dataex.wk3.z
│   │   │       ├── [4.8K Sep 20  1990]  datatab.wk3.z
│   │   │       ├── [ 948 Sep 20  1990]  dattim.wk3.z
│   │   │       ├── [ 677 Sep 20  1990]  dept_emp.wk3.z
│   │   │       ├── [ 690 Sep 20  1990]  deptexp.wk3.z
│   │   │       ├── [ 852 Sep 20  1990]  deptname.wk3
│   │   │       ├── [4.4K Sep 20  1990]  dfillex.wk3.z
│   │   │       ├── [6.9K Sep 20  1990]  graphex.wk3.z
│   │   │       ├── [ 588 Sep 20  1990]  hotel90.wk3.z
│   │   │       ├── [2.0K Sep 20  1990]  hotel_q1.wk1.z
│   │   │       ├── [2.0K Sep 20  1990]  hotel_q2.wk1.z
│   │   │       ├── [2.0K Sep 20  1990]  hotel_q3.wk1.z
│   │   │       ├── [2.0K Sep 20  1990]  hotel_q4.wk1.z
│   │   │       ├── [2.4K Sep 20  1990]  joinex.wk3.z
│   │   │       ├── [2.5K Sep 20  1990]  macex.wk3.z
│   │   │       ├── [3.4K Sep 20  1990]  macros.wk3.z
│   │   │       ├── [1.4K Sep 20  1990]  mfg.wk3.z
│   │   │       ├── [1.8K Sep 20  1990]  mis.wk1.z
│   │   │       ├── [1.8K Sep 20  1990]  planning.wk1.z
│   │   │       ├── [8.4K Sep 20  1990]  print.wk3.z
│   │   │       ├── [8.4K Sep 20  1990]  print_bw.wk3.z
│   │   │       ├── [1.3K Sep 20  1990]  printex.wk3.z
│   │   │       ├── [1.3K Sep 20  1990]  q2price.wk3.z
│   │   │       ├── [1.8K Sep 20  1990]  research.wk1.z
│   │   │       ├── [3.1K Sep 20  1990]  rev-1.wk3.z
│   │   │       ├── [3.1K Sep 20  1990]  rev-2.wk3.z
│   │   │       ├── [1.9K Sep 20  1990]  salary.wk3
│   │   │       ├── [1.3K Sep 20  1990]  sales.wk3.z
│   │   │       ├── [8.4K Sep 20  1990]  shoes.wk3.z
│   │   │       ├── [8.4K Sep 20  1990]  shoes_bw.wk3.z
│   │   │       ├── [ 842 Sep 20  1990]  summary.wk3.z
│   │   │       ├── [3.3K Sep 20  1990]  tables.wk3.z
│   │   │       ├── [3.5K Sep 20  1990]  temp.wk3.z
│   │   │       ├── [ 985 Sep 20  1990]  util.wk3.z
│   │   │       └── [4.2K Sep 20  1990]  world.wk3.z
│   │   └── [ 512 May 20 13:53]  sysV386
│   │       ├── [ 512 May 20 13:53]  bin
│   │       │   ├── [ 924 May 20 13:53]  123.sh.z
│   │       │   ├── [1.1M May 20 13:53]  123_exe.z
│   │       │   ├── [8.6K May 20 13:53]  inst_dl.z
│   │       │   ├── [ 96K May 20 13:53]  keyedit.z
│   │       │   ├── [ 14K May 20 13:53]  l13pbanr.z
│   │       │   ├── [1.4K May 20 13:53]  l13pupif.z
│   │       │   ├── [9.5K May 20 13:53]  prsetup123.z
│   │       │   └── [ 95K May 20 13:53]  setup123_exe.z
│   │       ├── [ 512 May 20 13:53]  dvr
│   │       │   └── [ 97K May 20 13:53]  l13pepdt.dvr.z
│   │       ├── [ 512 May 20 13:53]  lib
│   │       │   ├── [1.1M May 20 13:53]  123.o.z_1
│   │       │   ├── [717K May 20 13:53]  123.o.z_2
│   │       │   ├── [ 913 May 20 13:53]  dl_init.o.z
│   │       │   ├── [ 76K May 20 13:53]  ld.z
│   │       │   ├── [ 14K May 20 13:53]  mkdlobj.z
│   │       │   ├── [ 649 May 20 13:53]  stub.o
│   │       │   ├── [ 328 May 20 13:53]  tail.o
│   │       │   └── [1.3K May 20 13:53]  wyse50-lts123
│   │       └── [  56 May 20 13:53]  lotus.bcf
│   └── [ 512 May 20 13:53]  man
│       ├── [ 512 May 20 13:53]  cat.LOCAL
│       │   ├── [3.0K May 20 13:53]  123.1.z
│       │   ├── [3.1K May 20 13:53]  123.LOCAL.z
│       │   ├── [ 738 May 20 13:53]  inst_dl.1.z
│       │   ├── [1.2K May 20 13:53]  inst_dl.LOCAL
│       │   ├── [ 706 May 20 13:53]  keyedit.1.z
│       │   ├── [1.1K May 20 13:53]  keyedit.LOCAL
│       │   ├── [1.5K May 20 13:53]  setup123.1.z
│       │   └── [2.7K May 20 13:53]  setup123.LOCAL
│       └── [ 512 May 20 13:53]  man1
│           ├── [2.6K May 20 13:53]  123.1.z
│           ├── [ 931 May 20 13:53]  inst_dl.1
│           ├── [ 946 May 20 13:53]  keyedit.1
│           └── [2.0K May 20 13:53]  setup123.1
├── [ 512 May 20 13:53]  tmp
│   ├── [ 512 May 20 13:53]  _lbl
│   │   └── [ 512 May 20 13:53]  prd=123
│   │       └── [ 512 May 20 13:53]  typ=n386
│   │           └── [ 512 May 20 13:53]  rel=1.0
│   │               └── [   0 Sep 20  1990]  vol=01
│   ├── [ 178 Sep 20  1990]  init.123
│   └── [ 512 May 20 13:53]  perms
│       └── [ 985 Sep 20  1990]  123
└── [ 512 May 20 13:53]  usr
    └── [ 512 May 20 13:53]  tmp
        └── [ 512 May 20 13:53]  lotus_install
            └── [ 512 May 20 13:53]  123
                ├── [ 19K Sep 20  1990]  Install
                ├── [ 860 Sep 20  1990]  banner
                ├── [ 20K Sep 20  1990]  chkdskno
                ├── [ 25K Sep 20  1990]  combine_exe
                ├── [ 208 Sep 20  1990]  copyright.d
                ├── [ 167 Sep 20  1990]  filelist.1
                ├── [4.8K Sep 20  1990]  filelist.2
                ├── [ 351 Sep 20  1990]  filelist.3
                ├── [ 294 Sep 20  1990]  filelist.4
                ├── [ 257 Sep 20  1990]  filelist.5
                ├── [ 12K Sep 20  1990]  messages
                └── [ 25K Sep 20  1990]  mk_banner

35 directories, 338 files

A Mystery File

While poking around, this one directory caught my eye, what exactly is this?

$ ls -l
total 2.0M
-rw-r--r-- 1 taviso taviso 1.2M May 20 13:53 123.o.z_1
-rw-r--r-- 1 taviso taviso 717K May 20 13:53 123.o.z_2
-rw------- 1 taviso taviso  913 May 20 13:53 dl_init.o.z
-rwx------ 1 taviso taviso  77K May 20 13:53 ld.z*
-rwx------ 1 taviso taviso  14K May 20 13:53 mkdlobj.z*
-rw------- 1 taviso taviso  649 May 20 13:53 stub.o
-rw------- 1 taviso taviso  328 May 20 13:53 tail.o
-rw-r--r-- 1 taviso taviso 1.4K May 20 13:53 wyse50-lts123

That 123.o file is huge, even compressed it had to be split across two disks. Let’s take a closer look…

$ cat 123.o.z_? | gzip -d > 123.o
$ file 123.o
123.o: Intel 80386 COFF object file, not stripped, 5 sections, symbol offset=0x1efbdc, 19755 symbols, optional header size 28

Yikes - it’s an original unstripped object file from 1-2-3. There are nearly 20,000 symbols including private symbols and debug information.

Why would Lotus ship this? It’s so big it must have required them to phyiscally ship an extra disk to every customer? Could it have been a mistake, accidentally left on the final release image?

I had so many questions, but I’m not old enough to have any experience with SysV, so I asked the greybeards on alt.folklore.computers if they had seen this before and why this might have happened.

The answer was that this is probably deliberate - dlopen() was not widely available on UNIX in the early 90s, so there was no easy way to load native plugins or extensions. To solve this, vendors would ship a bunch of partially linked object files with a script to relink them with your extensions – Clever!

Hacking

I can’t tell you how useful this discovery was – the debug information answered so many questions I had about Lotus 1-2-3 internals! This was a direct source port from DOS, so it mostly worked the same way but now I had debugging data. For example, I really wanted to hook into the rasterizer in my driver so that I could improve the appearance of graphs in the terminal… but it was just too complex to understand without documentation.

I now know that the rasterizer dynamically generates little bytecode programs that are interpreted by the graphics engine. Now that I know what the opcodes are I can disassemble and change them in my driver to improve the output!

GNU objcopy

Okay… but there is one more big question, I know that objcopy can convert COFF object files to ELF, the format used by Linux. It seems like a long shot, but is it possible I could link this into a native Linux program? 🐧

$ objcopy -I coff-i386 -O elf32-i386 123.o 123elf.o
$ file 123elf.o
123elf.o: ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped

Hilariously, the first version of Linux hadn’t even been released when this object file was compiled – but I think this is possible! If you want to hear about the technical challenges, read on!

$ objdump -p 123.o  | grep Date
Time/Date               Sat Sep  8 06:23:50 1990

Porting Problems

System Calls

The first problem is that Linux and UNIX do not use a compatible system call interface. UNIX uses the lcall7 interface, so we need to find those calls and fix them up. Here is how this object file calls open():

$ objdump -M intel --disassemble=open 123elf.o

123elf.o:     file format elf32-i386


Disassembly of section .text:

000e20d4 <open>:
   e20d4:   b8 05 00 00 00          mov    eax,0x5
   e20d9:   9a 00 00 00 00 07 00    call   0x7:0x0
   e20e0:   0f 82 c6 01 00 00       jb     e22ac <_cerror>
   e20e6:   c3                      ret    
   e20e7:   90                      nop

That call instruction is what’s known as a callgate, which isn’t supported on Linux², it will just crash. I want to remove this and route all calls through glibc instead. My first thought was just to mark these symbols as undefined, and then let the linker fix that up by importing a replacement symbol from glibc.

Relocations

Nothing is ever easy, it turns out that won’t work! If we try, objcopy will simply refuse:

$ objcopy -I coff-i386 -O elf32-i386 --strip-symbol open 123.o 123elf.o
objcopy: not stripping symbol `open' because it is named in a relocation

What is objcopy trying to tell us here?

This is a relocatable object file, which simply means it can be loaded at any address and still work. That’s possible because it contains all the necessary information – the relocations – to adjust it.

Relocations are really simple, the compiler just records the name of the symbol and the references to it. Now the linker can just walk through and patch each reference to point to the new location – easy.

So objcopy is saying that you can’t remove this symbol, because the linker won’t know what to patch in when it moves it. Fair enough – but, just because objcopy won’t do it doesn’t mean it’s impossible! We could just fix the relocations too, right?

I don’t know of any tool that can do that, but COFF is not a complicated format – I’ll write one!

Introducing coffsyrup, a tiny little tool that will remove those pesky COFF symbols even if objcopy refuses!

$ coffsyrup 123.o 123new.o open
MATCH open
RELOC rel open @0x180fa ~0xc9ede
RELOC rel open @0x4c9a1 ~0x95637
RELOC rel open @0x4d348 ~0x94c90
RELOC rel open @0x4ec13 ~0x933c5

Incompatible Functions

Now that we can reroute functions, we have to worry about incompatible functions.

Lots of standard UNIX functions are source but not binary compatible, this is because nobody promises that structures are the same size or layout across UNIX versions. The obvious example is struct stat.

For example, this code is likely to work on any UNIX-like system you can compile it on:

struct stat sb;

if (stat("/etc/passwd", &sb) == 0) {
    printf("Size: %u\n", sb.st_size);
}

However, the resulting object file is not likely to work on any other system. That’s because the size of struct stat and the offset of st_size will be different – it will probably just corrupt your stack and crash!

Luckily there are not really that many functions like this in UNIX. In fact, the number is small enough that I can probably write wrappers to translate them. The important ones are stat(), times(), uname(), fcntl(), ioctl() and so on.

All I have to do is rename those symbols with objcopy, then mark them undefined with coffsyrup. Now I can write a little wrapper that translates a Linux struct stat to a UNIX struct stat and it should work!

Termios

Well…I said “little” wrappers – but there are some big incompatabilities in places. One big nightmare was termios. Go ahead and take a look at the termios(3) man page, pretty complex right? Well, everything here works differently in subtle, incompatible, and difficult to debug ways on every UNIX system.

Licensing

Incredibly, after a bunch of hacking it actually runs without crashing!

…and refuses to work without a license, damn! Well, I am a legitimate licensed 1-2-3 owner with a boxed copy of 1-2-3, and this is 32 year old abandonware. I think Mitch Kapor will forgive me for bypassing this check.

I can see from breaking on exit() that there is an internal symbol called lic_init() responsible for checking for a valid license. I looked at the code in IDA, and figured out the logic.

It is simply looking for a file called LICENSE.000, which contains an expiry date, username and systemname. If that all matches what the system reports, the check passes! 🏴‍☠️

Result

That’s it, Lotus 1-2-3 has been ported to a new operating system. There are a few kinks that need to be ironed out, and I need to port over my terminal driver, but it is 100% usable. At the moment, the DOS version running under emulation looks better - but this can be fixed!

I hope you enjoyed reading about this, in the highly unlikely event you actually want to try it yourself – all of my code is on github.

I wrote down some thoughts about Password Managers.

Tavis Ormandy

Password Managers.

Tavis Ormandy

$Id: fc2269c2b7523b7a939e3f444343abf5c27bd089 $

Introduction

I’ve spent a lot of time trying to understand the attack surface of popular password managers. I think I’ve spent more time analyzing them than practically anybody else, and I think that qualifies me to have an opinion!

First, let’s get a few things out of the way. For some reason, few subjects can get heated faster than passwords. Maybe politics and religion, but that’s about it. It’s okay if you don’t like my opinion.

Second, everyone needs to be using unique passwords. You don’t have to use a password manager to do that, whatever system works for you is fine. If you want to use a notebook in a desk drawer, that’s totally acceptable.

Okay, let’s begin.

If you just want some advice on what to use, skip to the conclusion.

Problems

Conceptually, what could be simpler than a password manager? It’s just a trivial key-value store. In fact, the simplest implementations are usually great. Good examples of simple and safe password managers are keepass and keepassx, or even pass if you’re a nerd.

Things start to go wrong when you want integration with other applications, or when you want data synchronized by an untrusted intermediary. There are safe ways to achieve this, but the allure of recurring subscription fees has attracted businesses to this space with varying degrees of competence. I’m generally skeptical of these online subscription password managers, and that’s going to be the focus of the rest of this article.

Bad Advice

I often say that “use a password manager” is bad advice. That’s because it’s difficult to tell the difference between a competent implementation and a naive one. The tech press can review usability and onboarding experience, but can’t realistically evaluate any security claims, so how do you propose users tell the difference? For that reason, I think “use a password manager” is so vague that it’s dangerous.

A good analogy is telling someone with a headache to pop any pills they find in the medicine cabinet. Maybe they’ll get lucky and find an aspirin, or maybe they won’t and you’ll be making a call to poison control.

Advice on this topic needs to be specific. It’s better to recommend implementations that are well designed, rather than general product categories. This position is surprisingly contentious, many people argue any password manager is acceptable and that I’m sowing fear by actually evaluating vendor claims. I remain unconvinced.

Attack Surface

My primary area of interest is how remote attackers can interact with your password manager.

I’m not interested in things like testing how resistant encrypted blobs are to offline cracking. This might be a valid concern for some, but in most cases if an attacker is in a position to access or tamper with encrypted state, then you were in trouble whether you used a password manager or not.

There are two common issues I run into, the first is that trusted user interface elements are injected into potentially hostile websites. The second is that different components ipc over web-accessible channels (e.g. WebSockets, postMessage, etc.) without adequate mutual authentication.

Lets discuss user interface elements first.

Hostile Environments

Most online password managers use content scripts, javascript that is inserted into every website you visit. It’s really easy to write content scripts, but really tough to make them tamper resistant. That’s kind of a problem, because they’re going to be hosted in hostile environments.

A good example of the kind of subtle ways this can go wrong is a bug like this. How isolated worlds interact is complicated enough, but password managers make matters even worse by blurring the distinction between user interface and content.

Chrome vs Content

There are two primary components that make up your browser interface, the chrome (confusingly, the term has nothing to do with Google Chrome) and the content area. The chrome contains things like the address bar, tabs and back button. These components can be trusted, and websites can’t interfere with them. Conversely, anything inside the content area can be controlled by the website and therefore it can’t be trusted.

Most password managers blur this distinction by drawing their UI in the content area. There is just no way to do this safely, and it’s trivial to make demos that show why.

Frames

Okay, but it’s not unusual to have different parts of the content area with different privileges, that’s basically how iframes work. That doesn’t make it safe though, and you’re probably familiar with X-Frame-Options, it’s a way for websites to opt-out of being framed. Most security conscious websites will do exactly that, because it’s just too easy for attackers to trick you into interacting with frames in unintended ways.

These are sometimes called redress attacks or clickjacking (groan, I hate that term). There’s a thorough analysis of the attack in the Browser Security Handbook.

If redress attacks are problematic enough to make everyone disable framing, should the software tasked with protecting your passwords also opt-out? That was a trick question, they can’t!

A Brief Illustration

As I’m writing this, I happened to see a commercial for a new password manager called Nordpass. Let’s use it as a testcase and take a look. All online password managers work in a similar way, this isn’t specific to any one implementation.

Note: I’m deliberately trying to avoid finding specific vulnerabilities so that I don’t have to start the tedious formal disclosure process. This is just a general illustration of weaknesses inherent in these designs.

After installing the extension I immediately see that it uses content scripts to add interface elements to login forms.

In general, websites can interact with that UI, and quick testing confirms that is true here. It looks like I can style those elements too, that means trivial redress is possible.

This problem is pervasive among online password managers, you can never be sure if you’re interacting with a website or your password manager. Let’s make a quick demo.

Here is a quick example, you can try it out yourself if you have Nordpass installed, please excuse my terrible design skills.

This is a fundamental and unfixable problem with designs like this.

I’ve found many real, exploitable vulnerabilities in the way these UI elements work. Trivial ones like this, or this. Sometimes interaction isn’t even necessary.

Interprocess Communication

We’ve already established that one component of online password managers must be injected into potentially hostile environments. How can those components communicate with other components?

One naive solution would be to just use XHR or WebSockets to a local HTTP endpoint. This sounds appealing to developers, they’re the native way to communicate on the web. The problem with this solution is it’s very difficult to differentiate between your content script, and a hostile script running on the same page but a different world.

Essentially every implementation I’ve looked at has got this wrong, resulting in critical game-over vulnerabilities. Some worst offenders are bugs like this, this.

Vendors come up with all kind of hacky solutions to this, often involving inherently racy background scripts that try to verify a tabs origin.

Sandboxes

Another gripe I have with online password managers is that they render browser sandboxes less effective. Modern browsers use a sandbox architecture to isolate components that can go wrong.

The problem is that online password managers effectively inject privileged components into these sandboxed processes with extensions. The purpose of sandboxing is to isolate potentially compromised components from each other, but if you stuff all your most valuable secrets inside the sandbox - then what’s the point?

I worry that people don’t understand the tradeoff they’re making here.

Vendor Claims

Despite what your vendor says, if their network is compromised, the attacker can read your passwords. Here are some selected marketing claims from password manager vendors:

“No one apart from you, not even [us], has access to your passwords.”
“[We keep] your information private, secure and hidden (even from us).”
“Your data is secured in a way that only you can view and manage it. [Our] employees can’t”
etc, etc.

These claims are all nonsense. An attacker (or malicious insider) in control of the vendor’s network can change the code that is served to your browser, and that code can obviously access your passwords. This isn’t farfetched, altering the content of websites (i.e. defacement) is so common that it’s practically a sport.

The reality is that you have to trust your vendor to maintain their infrastructure and keep it safe. The existence of encryption (“bank grade” or not) does not alter this.

Perhaps you think this isn’t a big deal, you already trusted them when you installed their software. Fine, but these claims are front and center in all marketing, so vendors must believe their customers care about it. I think these claims are bending the truth to assuage legitimate concerns.

More Claims

It’s easy to poke holes in marketing fluff, but here are some other fun ones I noticed from real password manager vendors.

Keystroke encryption protects everything you type from being read by cybercriminals. Oh, okay.
Many of the .NET assemblies […] are obfuscated, so even using a disassembler users are unable to view critical areas of methods/functions/classes. Well, I certainly feel safer.
etc, etc.

Conclusion

If you want to use an online password manager, I would recommend using the one already built into your browser. They provide the same functionality, and can sidestep these fundamental problems with extensions.

I use Chrome, but the other major browsers like Edge or Firefox are fine too. They can isolate their trusted UI from websites, they don’t break the sandbox security model, they have world-class security teams, and they couldn’t be easier to use.

No doubt there will be many people reading this who don’t like this advice. All I can say is I’ve heard all the arguments, and stand by my conclusions.

I wrote a display driver for an old DOS program.

Tavis Ormandy

Lotus 1-2-3

Tavis Ormandy

$Id: a07cf90837a3c4373b82d6724b97593810766af7 $

Intro

I tend to use a lot of retro software, partly because I’m a big nerd but also because I just prefer to work in a terminal. There were a ton of major commercial software products for MS-DOS that were highly polished, and can still be used effectively in an xterm today.

I do like using Lotus 1-2-3, I even have a boxed copy of the last DOS version released, version 4!

Problem

The truth is I’m cheating, it does work, but it only supports a few standard text mode resolutions. If your terminal is not exactly 80 columns wide, it just makes a big ugly mess on your screen.

There’s a workaround, just type stty cols 80, and it will be confined to a portion of your terminal, looking a bit sad. There is no way to display more columns, and maximizing your terminal will do nothing.

Research

I wondered if other people still uses 123, and if they had found a solution. I enjoy using it, but being limited to such a small work area is a real nuisance.

I spent an hour reading vintage computing forums and old USENET posts.

Unfortunately, the answer is that nobody has a solution to my resolution problem, but I did find something interesting…

Endeavor

I found this old manual for a 1993 PC Workstation called the Epson Endeavor, which claimed to be distributed with a utilities disk that enabled a 100x31 text mode in 1-2-3.

How could such a thing be possible?

I was able to find the drivers on an old ftp site. Sadly they didn’t work, presumably they needed a real Cirrus Logic VGA card, but the fact that these drivers exist at all made me curious. It seems plausible that 123 can internally generate an arbitrary sized canvas, and it simply asks the display driver what resolution it wants?

I know that dosemu can simulate arbitrary text mode resolutions, and some DOS software is able to use it. Is it possible that I could write a dosemu display driver, and 123 would just work in a fullscreen xterm?

Reverse Engineering

The display driver API is unfortunately quite complicated, I think it might take a while to figure out with just a disassembler and a debugger. Further complicating matters, 123 uses a technique called overlays (like software bank switching) which confuses the debugger I’m using as it loses breakpoints whenever a overlay switch occurs!

I figured that Lotus must have sent Cirrus Logic some documentation to write the driver, perhaps that still exists somewhere and would save me some time.

I spent some time looking, but came up empty.

I learned a lot about the Lotus add-in ecosystem, and how they worked. I figured I would download all the drivers and add-ins I could find, I’ll start reversing the simplest one, and move on until I know enough to write my own.

If you want to read about what commercial addins were available, this article is a fun read.

Hello World

With this data, and a weekend in IDA I was able to figure out a large amount of the API. It was painful getting a toolchain that worked, but I was eventually able to produce something that Lotus could load.

I think I must be the first person to write a Lotus 1-2-3 driver in thirty years!

My driver didn’t actually do anything, but when I responded to device information requests claiming to support 256 columns, I saw in my log that it sent me an LMBCS string to decode like this:

ENTER WriteLmbcsStringWithAttributes, caller 15f:6c7d
lotdemu.c:291: TRACE byteslen 238
lotdemu.c:293: TRACE attrs 000
lmbcs.c:104: TRACE translate_lmbcs
lmbcs.c:106: TRACE maxdst 256
lmbcs.c:107: TRACE maxsrc 238
000000: 20 20 20 20 41 20 20 20 20 20 20 20 20 42 20 20      A        B
000010: 20 20 20 20 20 20 43 20 20 20 20 20 20 20 20 44        C        D
000020: 20 20 20 20 20 20 20 20 45 20 20 20 20 20 20 20          E
000030: 20 46 20 20 20 20 20 20 20 20 47 20 20 20 20 20   F        G
000040: 20 20 20 48 20 20 20 20 20 20 20 20 49 20 20 20     H        I
000050: 20 20 20 20 20 4a 20 20 20 20 20 20 20 20 4b 20       J        K
000060: 20 20 20 20 20 20 20 4c 20 20 20 20 20 20 20 20         L
000070: 4d 20 20 20 20 20 20 20 20 4e 20 20 20 20 20 20  M        N
000080: 20 20 4f 20 20 20 20 20 20 20 20 50 20 20 20 20    O        P
000090: 20 20 20 20 51 20 20 20 20 20 20 20 20 52 20 20      Q        R
0000a0: 20 20 20 20 20 20 53 20 20 20 20 20 20 20 20 54        S        T
0000b0: 20 20 20 20 20 20 20 20 55 20 20 20 20 20 20 20          U
0000c0: 20 56 20 20 20 20 20 20 20 20 57 20 20 20 20 20   V        W
0000d0: 20 20 20 58 20 20 20 20 20 20 20 20 59 20 20 20     X        Y
0000e0: 20 20 20 20 20 5a 20 20 20 20 20 20 20 20             Z
translated characters is 238
lotdemu.c:209: TRACE WriteStringToFramebuffer

It’s asking me to write 26 column headers to the framebuffer!

This validated the idea, Lotus 1-2-3 does internally support arbitrary resolutions!

Results

A ton of hacking later, and I do now have a usable driver for dosemu that supports arbitrary resolutions, just look at all those columns!

I know enough about the API that I think I can probably add libcaca support so that text mode graphs work, but that’s a project for another weekend!

Conclusion

My work-in-progress source code is available on github.

I’ve put all of my notes online so that in the event that anybody else wants to make 123 DOS work somewhere new, they won’t have to repeat all my work!

Update

I got ascii art graphs working in a terminal too! Here is a 123 plotting a sine wave in an xterm.

You can make popup ads in XTerm with Tektronix mode.Linking a binary with two versions of the same library.

Tavis Ormandy

Tektronix Popups.

Tavis Ormandy

$Id: a07cf90837a3c4373b82d6724b97593810766af7 $

Intro

xterm can emulate a Tektronix 4010 terminal, It’s actually pretty neat, it’s enabled by default and applications can plot simple graphics (lines, curves, etc) or use it to show graphs.

It randomly occurred to me that windows unexpectedly opening is a bit like a popup browser ad.

Well, it turns out that Tektronix emulation in xterm is so fast that you can draw monochrome bitmaps. Nobody ever asked for or wanted this, but here is a README file that abuses this to create a popup ad when you cat it in an xterm. 😈

Recipe

Here is how I made this:

First I created a simple popup ad in GIMP, and saved it as a 1bit png image, then converted that to raw RGB with ImageMagick.

$ convert -flip -colors 2 xterm-popup.png rgb:xterm-popup-400x200.rgb

Then I wrote this quick C program to generate a plot.

#include <stdio.h>
#include <stdlib.h>

// Convert RGB into monochrome points for gnuplot
// Usage: ./genplot <width> <height> < image.rgb > plot.dat
// -- taviso@gmail.com December, 2020

int main(int argc, char **argv)
{
    int r, g, b;
    int x, y;
    int width  = atoi(argv[1]);
    int height = atoi(argv[2]);

    for (x = y = 0;; x++) {

        x %= width;
        y += !x;

        r = getchar();
        g = getchar();
        b = getchar();

        if (feof(stdin))
            break;

        if (!!r || !!g || !!b)
            continue;

        printf("%u %u\n", x, y - 1);
    }
    return 0;
}

Now gnuplot can parse that file and generate Tetronix commands that you can just append to a text file. I needed to experiment with the pt and ps options to find the best look, but it worked out pretty well. You’re welcome!

$ gnuplot -e "set terminal xterm;
>  unset key; unset ytics; unset xtics;
>  plot 'plot.dat' pt 2 ps 1" >> README.txt

Don’t worry, if this is ever abused, you can always do alias ucat-origin="cat -v" 😂

I reviewed some spreadsheets that work in the terminal.

Tavis Ormandy

Console Spreadsheets.

Tavis Ormandy

$Id: f0cc50a6fc6a5dd652c2b96ca9c1779f763fd6b1 $

Intro

I’m not a heavy spreadsheet user, just keeping track of expenses, household inventory, investments, that kind of thing. I mostly just use Google Sheets.

However, I really prefer to stay in a terminal if I can, so decided to review some of the options. My only requirements are that it runs in a terminal and works on Linux, but bonus points if it’s extensible (scripts, macros, plugins, etc).

I’ve tested two F/OSS and two commercial console spreadsheets. Here’s the big caveat, it might surprise you to learn (haha) that there isn’t a huge demand for console productivity software. That means these programs havn’t been actively developed for a very long time, but they were once incredibly popular.

Their userbase may now be extinct, but they’re all still mature, stable and functional.

Spreadsheets

Spreadsheet Calculator (sc)

sc is the classic UNIX spreadsheet, written by James Gosling. It’s public domain, and available in most linux distributions.

Perhaps the most interesting thing about sc is that spreadsheets are saved as sequences of commands, and a .sc file is a human readable script. Here’s an example, if I dump the saved file from the screenshot above, you can see it looks like a scripting language:

$ cat test.sc
# This data file was generated by the Spreadsheet Calculator.
# You almost certainly shouldn't edit it.

rightstring B0 = "Cost"
rightstring C0 = "Count"
rightstring D0 = "Total"
rightstring A1 = "Screws"
let B1 = 0.02
let C1 = 23
let D1 = B1*C1
rightstring A2 = "Plates"
let B2 = 0.44
let C2 = 8
let D2 = B2*C2
label D3 = "------------"
let D4 = @sum(D0:D2)
goto D6 A0

You don’t have to enter these commands directly, as you move around and press keybindings, sc populates the command entry for you - you simply add the options and parameters. I found a particularly good introduction to using sc here. It turns out sc doesn’t have any way to import data, so I had to write a quick script to do that - but once you do get data into sc, all the standard spreadsheet features are there.

The syntax for formulas, cell references and ranges is all familiar. When working with ranges, you can select them in an interactive visual mode and then name them for use in formulas.

That said, while formula syntax, cell references/ranges (e.g. C3, $A$1:B2, etc) and basic navigation are all familiar, the list ends there.

You’re going to have to read the documentation to do anything more advanced than editing cells. The commands mostly make sense, but you’re not prompted for any parameters, you have to memorize the options you need from the documentation before you enter command editing mode. Good luck getting that right.

sc does support macros, so you can create a macro for commonly used options (a macro is essentially just a sc command script), and even bind it to a function key.

You can’t rebind standard movement or navigation keys, but most functionality is accessed via commands, and you can define your own.

Summary

License	Public Domain
Platform	Portable, C
Last Release	2002
Website	None, probably comp.sources.unix

Pros

Portable, Public Domain.
Macro support.
Most standard spreadsheet features and functions.

Cons

Steep learning curve.
No graphs, although you could pipe data into gnuplot.
No undo.
No import.
Limited formatting options.
Keys can’t be rebound.

SC-Improved (sc-im)

I really don’t think I could use sc regularly, it seems like a considerable time investment to master basic worksheet manipulation, and I just don’t use a spreadsheet enough to justify that.

However, hope is not lost. There’s a fork of sc with the interface reworked to make it familiar to Vim users. Sc-im also adds desperately needed features like undo/redo, rebindable keys, and import from xls and csv files.

The main attraction for me is that I was able to guess how to do some operations without looking it up, because the bindings and commands are all inspired by Vim. Things like visual mode, yanking and pasting, searching, undo and redo, even :wq! just work.

In the demo above, you can see me using visual mode to select a column, yanking and pasting it somewhere else, and then adjusting widths. You can also generate a simple ASCII graphs using gnuplot. To generate the graph in the screenshot I used v to enter visual mode, moved around to choose the data, then :plot bar.

An especially welcome feature is the formatting options, headers can be bold, negative values can be red, and so on. That makes reading a big list of numbers easier.

Summary

License	As-is
Platform	Portable, C
Last Release	2017
Website	https://github.com/andmarti1424/sc-im

Pros

Familiar bindings and commands for Vim users.
Import and export data from CSV, XLS, etc.
Integrated gnuplot support.
Extensible, supports scripting and plugins.
Still under development.
Undo/Redo support.
Serviceable online help.

Cons

I ran into a few bugs while trying it out.

GNU Oleo

The original GNU spreadsheet. I had to fix some bitrot to get it to compile on a modern system. The code was mostly well written, just some outdated C syntax.

Oleo uses r1c1 cell references by default. The only reason I know about r1c1 is because I saw a fun talk by Joel Spolsky who said anything else is “baby mode” 😂.

It’s worth watching the talk, most of what he says applies to any spreadsheet. Still, it feels pretty foreign if you’re not used to it, so luckily you can disable it. I guess I’m a spreadsheet baby! 👶

Where sc-im navigation feels familiar, Oleo was clearly designed with Emacs users in mind. For example, disabling r1c1 mode requires M-x set-option a0. Still, the bindings are configurable with keymaps and macros, so you can adjust it.

There is builtin graphing, but by default it uses xterm’s tektronix mode - I don’t think any other terminals support that except real xterm. If you’re not using xterm, Oleo can save an image file that can be viewed externally. There is a simple in-cell bar chart format though.

Worksheets can update automatically, so you can make a live dashboard that ingests stock prices or other data. I think this is surprisingly unique, the other spreadsheets I’ve tested only do recalculation when you make a change. Oleo has an option called ticks that lets you specify an automatic update frequency.

Here’s a silly demo that updates with random numbers, but I can imagine how I might make a spreadsheet that fetches live data for a dashboard.

Oleo has some major missing features, as far as I can tell there is no ability to sort a range, and there are very limited formatting options. That said, the features that are implemented are quite polished.

Summary

License	GPL-3
Platform	Portable, C
Last Release	2001
Website	https://www.gnu.org/software/oleo/

Pros

Optional live recalculation.
Optional mouse support.
Extensive keymap rebinding.
Built in graphing, including simple in-cell graphs.
Native import support.
Macros and scripting.

Cons

Default keybindings only an Emacs user could love.
Steep learning curve.
Some missing standard spreadsheet features.
Limited formatting options.
Terrible online help, and cryptic error messages.

Lotus 1-2-3

The original IBM PC killer application, with millions of active users throughout the 80s and 90s.

I was able to find a boxed copy on eBay with the original shrinkwrap for around $20, and installed it on Linux under DOSEMU2. Using ancient software with emulation won’t appeal to everyone, but I quite enjoy it. DOSEMU2 let’s old DOS software access the filesystem, run host commands as if they’re DOS commands, and makes them look indistinguishable from native ncurses applications.

Note: 1-2-3 is available from reputable historical software archives like the Internet Archive and WinWorld. Original copies are available on eBay for around $5-$20 depending on condition. I paid a little more to get a boxed copy with a printed manual.

While the UNIX spreadsheets all have a bare bones interface, the MS-DOS alternatives have very polished usability. 1-2-3 has context-sensitive help with detailed examples, menus, and dialogs to setup complex functions. If a formula or command generates an error, pressing F1 will usually give you a detailed description of the error, suggest causes and solutions.

If you’ve used any spreadsheet before, you don’t really need to read any documentation to get started. Perhaps the only thing you need to know is that the 1-2-3 menu is activated with the / key. Interestingly, that menu was the subject of a major lawsuit in the 90s.

I don’t mind learning the common commands, but if I only want something occasionally I know I just won’t bother using it if I have to lookup the syntax every time. Here’s a good example, sc and Oleo both support column freezing (that’s where you make a column always visible as your browse around, so you can always see the headings). I could live without it, but it’s so easy to find in 1-2-3 that it’s no effort at all.

1-2-3 can import from CSV and a few other formats but with serious limitations: No line can exceed 512 characters, which seems ridiculously small to me. Worksheets can contain thousands of columns though, so I suppose you just need to import a larger CSV dataset a few columns at a time.

There are extensive formatting options, but very few of them are actually displayed in text mode. For example, you can make a label bold, but unless you print it or switch to graphical mode, it doesn’t look bold. Some options work though, like setting negative values red.

Graphs in 1-2-3

You can create a graph and give it a name (e.g. EXPENSES), then assign cell ranges to axes interactively. There are dialogs and menus to help choose display options, things like graph type (e.g. bar/pie/line), colors, and so on. The process is easy, mostly just following prompts.

You can insert the graph into the sheet, but only a placeholder is inserted - if you want to view it you need to send it to the printer or save to a file. This actually works pretty well in DOSEMU2, I can hit a key and the graph opens in my browser.

Here’s an example graph, if you want to see it.

Using Macros

To see how macros worked, I tried automatically updating a spreadsheet with stock prices so I wouldn’t have to enter them manually. This wasn’t too difficult with DOSEMU, first I created a quick shell script to print stock prices to stdout.

#!/bin/bash

declare api="https://query1.finance.yahoo.com/v7/finance/quote"
declare filter=".quoteResponse.result[].regularMarketPrice"

curl -s "${api}?symbols=${1:-^GSPC}" | jq -ar "${filter}"

Now you can call it from within DOSEMU using the unix command, like this:

C:\>unix stocks.sh GOOGL
1764.13

So an example 1-2-3 macro would look like this, where C1 is the cell I want the result stored.

{SYSTEM "UNIX stocks.sh GOOGL > %TEMP%\STOCKS.TXT"}
{OPEN "C:\TMP\STOCKS.TXT", "r"}
{READLN C1}
{CLOSE}

In the screenshot you can see I’ve called this macro {GETSTOCKS}, and I’ve configured it to run automatically when I open the file. Note that you can hide the macros if you don’t want them to clutter up your spreadsheet.

I think this worked pretty well, who would have thought 30 year old MS-DOS abandonware could use a JSON API 😂

Summary

License	Abandonware, Originally $495
Platform	MS-DOS
Last Release	1994 (for MS-DOS)
Website	https://web.archive.org/web/19961221020320/http://www.lotus.com/corpcomm/2612.htm

Pros

Excellent documentation and online help.
Menu system makes finding features easy.
Minimal need to memorize all but the most common commands.
Highly configurable graphs.
Imports from CSV, but with some size limitations.
Excellent macro support, with record, replay, and debugging.
Undo support.
Optional mouse support.

Cons

~~Only supports VGA text resolutions.~~ I fixed this!
Most cell formatting options are not displayed.
Proprietary.
Non-native.

Borland Quattro Pro

Perhaps 1-2-3’s biggest competitor, Quattro Pro was later acquired by Corel who still develop it today. The last console version is from 1993, and I was able to find a new-old-stock copy on eBay for around $30.

Note: Quattro Pro is available from reputable historical software archives like the Internet Archive and WinWorld. Original copies are available on eBay for around $5-$20 depending on condition. I paid a little more to get a boxed copy with a printed manual.

Apparently I got some kind of limited edition copy given to the development team…? ¯\_(ツ)_/¯

Like Lotus 1-2-3, I use it in Linux using DOSEMU2. I think my first impression of Quattro is how modern it looks, the Turbo Vision interface has aged remarkably well. Menus are organized how you would expect them today, things like File/Open, Edit/Copy, etc.

Quattro Pro is the only spreadsheet I’ve tested that supports box drawing characters, so you can draw cell borders. More formatting options are always welcome, and the graphs in Quattro also look pretty good. Here is the same example from Lotus 1-2-3.

All the standard features work well, and are easy to discover through the menus. The interface is very configurable, you can rearrange and edit menus and keybindings. If you wanted to use it with a mouse you can even make a toolbar.

In general, Quattro Pro has feature parity with 1-2-3, but just feels more polished. Perhaps one exception, you can import data from CSV, but astonishingly there is a maximum line length limit of just 254 characters. That just seems ridiculous.

I browsed around old usenet archives and found that Borland released an official workaround, a macro called DELREAD in their Quattro Pro Utilities collection. I guess even in 1993 people thought that limit was crazy.

Summary

License	Abandonware
Platform	MS-DOS
Last Release	1993 (for MS-DOS)
Website	https://web.archive.org/web/19961221133010/http://www.borland.com/

Pros

Possibly the easiest to use, familiar menu layout.
Highly configurable. Menus, colors, toolbars and keys can all be redefined.
Optional mouse support.
More formatting options that display in the console.
Excellent online help, including context sensitive help.
Excellent macro support, even better debugger than 1-2-3.
Nicer looking graphs than 1-2-3.
Very polished usability, dialogs and menus are easy to navigate.
Import and export to various other formats.

Cons

Only supports VGA text resolutions.
Proprietary.
Non-native.

Conclusion

I’m a big enough nerd that I rather enjoy using retro software, and I think I’m going to start using Lotus 1-2-3. There’s no question the MS-DOS packages are the more polished of the console spreadsheets I’ve tested.

I really do like sc-im, and I’m glad it exists. I think as a vim user I could be quite efficient at basic manipulation. That said, I think there are features I like but just wouldn’t use because I’d have to look up how to use them every time. Quattro Pro is accessible enough that I can poke around in menus when I need to use a feature I’m not familiar with.

Perhaps I’ll update this article in 3 months with my experiences and describe any limitations I ran into… or if I gave up and switched to something else. 😅

Read about me trying out some old MS-DOS software, Lotus Agenda.

Tavis Ormandy

Lotus Agenda.

Tavis Ormandy

$Id: a07cf90837a3c4373b82d6724b97593810766af7 $

Intro

I never used Lotus Agenda, but I’m told it was a popular productivity tool for MS-DOS in the late 80s. I’ve been on a retro software rediscovery kick lately, so I’ve decided to give it a whirl and write about my experiences. There is something that appeals to me about using long-abandoned software. Perhaps it’s update fatigue, there’s certainly no need to dread a major update breaking something!

Regardless, I’ve always enjoyed finding new productivity tools to try out, and I’m not afraid of steep learning curves or getting my hands dirty. I’ll usually choose powerful and flexible software over simplicity.

At the moment I mostly use taskwarrior, but I’ve lost count of all the others I’ve tried!

About Agenda

Agenda is a PIM, a Personal Information Manager. That term has fallen out of fashion, I think a quick summary might be “anything that manages those small pieces of information we all deal with”. Things like contacts, todo lists, notes, and so on.

I found a 1989 episode of the TV show Computer Chronicles that discussed how people thought about PIMs at the time.

At the 21-minute mark there’s a demonstration of Lotus Agenda, but it’s not easy to follow, watch the clip and you’ll see what I mean. Still, you do see some interesting features:

Some form of NLP is used to extract dates and names from free-form text.
Categories are a little like what we would call tags today.
You can create custom dashboards called views.

Apparently this was an $800 software package (That’s $395 adjusted for inflation from 1989), yikes! You don’t have to pay that, Lotus made it available for free when development ceased.

In preparation for trying out Agenda, I found a copy of the original manuals on eBay for a few dollars. Just look at this monster, the user guide alone is over 700 pages, that’s not including the supplementary guides. The supplements I have are Working with Macros, Working with Definition Files, Setting up Agenda, and a few miscellaneous leaflets.

I guess that’s my bedtime reading taken care of for a while. I actually received the macro reference still in the original shrink wrap, it almost seems a shame to open it!

Getting Started

I’m using DOSEMU2 to run Agenda, which also works with WSL on Windows. The primary benefit of DOSEMU over other options is that it has a terminal mode that translates VGA text mode to terminal escape sequences. This makes old MS-DOS applications look and feel like native ncurses applications.

Terminal tools like GNU screen work, you can copy and paste between native applications, and you can use it over ssh. Naturally, it also supports filesystem and shell redirection, so you can access host directories, and execute host commands as if they were DOS commands.

Here’s how Agenda looks when it’s first installed.

You’ve probably noticed the empty screen regions, well, that’s because Agenda only supports standard VGA Text Modes. I usually use a 100x30 terminal, so that leaves some blank space. If you use GNU screen you can just put something else in there, how about ncal!

I configured DOSEMU to map a few directories so I can save my files in the usual places, you just need to add something like lredir I: \\linux\fs\home\taviso\Documents to autoexec.bat.

Learning Agenda

I started making my way through the Quick Start guide, I found a scan online if you want to take a look. The guide has you use Agenda as a task planner to enter upcoming events, like “Call Sarah this Friday to give her feedback on her proposal”, and demonstrates how Agenda can extract dates and activities from text you enter.

Here is a gif to demonstrate, I enter a task and it correctly populates some of the other columns. Playing around with different ways of writing tasks, I find the metadata extraction works quite well. Time specification parsing in particular is quite polished, things like “Check data retention policy every four months starting Tuesday” just work. I’m also relieved that it appears to be Y2K compliant (haha!).

The pitch in the Quick Start Guide is that if you have a call with Tom, you can quickly generate a report for all the things you wanted to discuss. If you’re going to a specific location, you can list all the things you needed to do there, and so on.

Agenda is flexible software that can be used for more than just task management, but to keep this article brief I’m sticking to using it as a planner.

Using The Planner

I’ve entered enough tasks now to get a feel for usage, and all the standard stuff works well. Things like notes, setting alarms, adding contacts, assigning priorities, searching and sorting, and so on. The sort of things that get tedious can be automated, like marking all the results of a search, then bulk assigning or marking done.

Agenda is pretty stable, I guess some of those $395 dollars went towards QA. I did find one crash, I was trying to make an assignment action for a pseudo category in the show view! I think I’m the first person to discover this, I can’t find any other references to it!

I had to change my workflow a bit, I’m used to organizing my tasks into subtasks, so for example I might have a to-do list like this:

Deploy Death Star

Choose Target

Verify Design

Check for weak points

Call maintenance

Assign Staff

Call Mom

That isn’t really how Agenda works, instead you would create each of these tasks and assign them to the same project. This is a shift for me, but it has benefits, such as allowing a task to be assigned to multiple projects.

You can see in this clip I’ve changed the layout, you’re free to add or remove columns and change their order, width, and so on. You can also add filters, so you might want to only see Tasks due today, for example.

You can save an unlimited number layouts (Agenda calls these Views). You can switch to the view that suits your current task, preferred level of information density, and so on.

You might have a “Chores” view while at home that just shows the things you need to do at home, or a “Development” view while working on code. This can let you display your tasks in the most appropriate way.

Honestly, I think Agenda does this quite well, and getting a finely tuned view with sections and filters set up precisely was quite satisfying. There are different kinds of views as well, such as datebook view (like a calendar), expense view (for tallying expenses associated with a project), and so on.

In my main dashboard I have three sections:

Things I have due within a week	Tasks `[-Done,When(<->A week from today)]`
Things that dont have a due date	Tasks `[-Done,-When]`
Things I’ve completed recently	Done `[Done(2 days ago<->)]`

I actually wrote the filter as “day before yesterday”, and it automatically rewrote it to 2 days ago. Neat. I have a few more views with more columns and project filters that I switch to as I need them. You can either press F8 to browse the list of views, or use Alt-N (Next) and Alt-P (Prev) to cycle between them.

Entering Notes

Agenda does have a built-in editor for keeping notes. It’s okay. It supports a few basic editing options and has rebindable keys and macros. Luckily it supports importing external files, so you can just use Vim.

In fact, I wrote a macro to switch to vim whenever I type Ctrl-F5 in the editor. Here is a clip of me opening the Macro Manager, showing that I have an “Editor” macro bound to {CtlF5}, and then editing a note in Vim.

I wish I could say the macro language is pretty, but it is not. I think it was inspired by the macro language from 1-2-3, it’s geared towards reading whats on the screen and simulating keystrokes. It has basic control flow, string manipulation, and so on. It’s functional.

Here is that macro you just saw in the editor clip.

{Editor}
{LET;%FILE;C:\TMP\NOTE.TXT}
{WINDOWSOFF}
{F10}E{TYPE;%FILE}
{ENTER}
{ENTER}
{SELECTION;File already exists}
{IF;(#PROMPT <> File already exists:);EDITOR}
{F3}Replace{ENTER}
{LABEL;EDITOR}
{F5}
{F10}ULVIM {TYPE;%FILE}
{ENTER}
{F5}
{F10}C
{ENTER}
{F10}I{TYPE;%FILE}
{ENTER}
{ENTER}
{WINDOWSON}

I can say a few positive things about the macro language, the manual is pretty good and it comes with lots of samples. It can automate all parts of Agenda, but I’ve mostly just used it to rebind keys.

In fact, in the 1980s third party developers wrote commercial add-ons for Agenda. Apparently there was a popular package called President’s Planner. I haven’t tried it, but I suppose I have a morbid curiousity what an application written in this language can do!

Importing and Exporting

You probably need to use other applications or services, and sync your data with your phone. Writing and reading files from outside DOSEMU is no problem, so if you just want to sync files this is no problem.

As it’s a terminal application you can also just SSH in and run it.

You probably also want to have your appointments sync with your calendar or something.

Export

There are two ways to export data from Agenda. If you have a commandline tool that you can pass arguments to, then you can write a macro that will invoke it.

Otherwise, you can export your data to a file.

Files

Agenda can export items to a format called STF, Structured Text File. The specification for that format is (mostly) documented in the manual, but it didn’t catch on.

I wrote a quick parser that can convert it to JSON, so now you can use modern tools like jq to manipulate and transform the data however you wish.

You can download it here, here are some examples.

Print a list of all items.
- $ ./stfjson < transfer.stf | jq '.[].items[].text'
Show all items with a due date.
- $ ./stfjson < transfer.stf | jq '.[].items[] | select(.categories[].name=="\\When")'

And so on, there are more examples in the README. If you can exchange data with other apps, you can now use stfjson to generate the correct format.

You can automate exports, Agenda has “Special Actions” in the category options. Alternatively, if it’s just a one off or for a macro, you can use the Transfer > Export command.

Commands

In DOSEMU, the UNIX command will invoke a shell command on the host.

C:\>unix uname
Linux

If there is a commandline tool that will import data, e.g. a TaskWarrior user might use task add drop off laundry at dry cleaners, then you can create a macro in Agenda that simply launches that command.

You can use something like {F10}ULUNIX task {TYPE;%TASKTEXT}.

Import

Surprisingly, Agenda supports importing arbitrary text data. One of the manuals that came with agenda was Working with Definition Files, which explains how to write a configuration file that allow Agenda to parse anything.

It even has a Regular Expression tutorial, pretty impressive for a 1980s consumer product.

Conclusion

I quite like Agenda. It does many things well, but it’s absolutely true you could replicate most of it’s functionality with modern tools. However, I do enjoy using it, and I’m a big enough nerd that I quite like the challenge of using retro software.

I think the closest modern equivalent to Agenda would be taskwiki. It’s not a perfect match, but if you liked some of what you saw here but are not interested in retro software, try it out!

I’m still using Agenda after two weeks, and about 40% of the way through the manual 😂

Resouces

Bob Newell’s Agenda Page has lots of old documentation. In particular, I found this old Lotus Support FAQ from 1990 fascinating.

Troubleshooting

Anyone trying to earnestly use 30 year old software is bound to encounter a few hiccups. Here are some of the problems I ran into. I also have an FAQ.

Q. The installer complains about registering disks.
A. The free disk images from Lotus include some form of copy protection. You can’t just extract the files - you need to emulate a floppy drive. If you don’t want to bother with that, there’s a preinstalled ZIP available here.

Q. Some of the default keystrokes are difficult to enter today, like Alt-F4.
A. You can rebind keys using a Macro.
1. Open the Macro Manager, Alt-F3.
2. Press Ins to create a new Macro, call it “Macro”
3. Press F2 to open the editor, and enter this:
```
{Macro}
{AltF4}
```
1. Press F5 to save it, then F6 to open the macro properties.
2. Select “Attach to key”, Press F2 and enter {CtlF4}.
3. Now you can use Ctrl-F4 instead of Alt-F4. You can do the same thing to other bindings that you don’t like.

I made Ctrl-S Save, and Ctrl-F Find. The defaults are Alt-W (??) And Alt-F6 (???!).

Q. I can’t enter Alt-F3 in my terminal, so I can’t rebind keys!
A. Download SCANCODE.COM from here, it’s a DOS TSR that can simulate keystrokes. Once you have it installed:
1. Press F10, Select Utilities, then Launch
2. Type SCANCODE Delay 3,Alt-F3 (If SCANCODE is not in your %PATH%, type the full path, e.g. C:\SCANCODE\SCANCODE.COM.
3. After a few seconds, the Macro Manager will appear. I would suggest rebinding it to Ctrl-F3, so you don’t have to do that everytime!

Q. When I exit agenda, try to print something, or run external commands I get an error like EInternal Error.
A. Add the line $_umb_a0 = (off) to your .dosemurc.

Q. When I run Agenda, it looks garbled like this.
A. You are using DOSEMU with a non-standard text mode. Here are some possible solutions, pick whichever one you prefer.
- Before starting DOSEMU, type stty cols 80.
  - If you use an unusual terminal (like Windows Terminal), you might get an error like Operation not supported. A workaround is to install xtermcontrol and type xtermcontrol --geometry=80x${LINES}.
- Use GNU screen, and create a layout for DOS applications (this is what I do, see the next question).

Q. Can I see your GNU screen configuration to get me started?
A. Sure, here’s my main screenrc, and here’s my dosemu specific one.

I use alias agenda="screen -c ${HOME}/.screenrc.dosemu dosemu -term -E agenda" in my .bashrc to start agenda.

Q. Can I see your DOSEMU configuration to get me started?
A. Sure, here.

Q. Agenda uses 100% CPU time while idle, which is stopping my CPU from entering low power states. How do I fix that?
A. Update DOSEMU, I filed an issue and a fix was committed.

Q. Agenda doesn’t use 100% CPU time, but it still uses more than I would like.
A. DOSEMU simulates timer interrupts using setitimer(), and by default it requests them as quickly as possible. If the DOS application does a lot of work on every interrupt (Agenda queries the time), then it can use some CPU.

There’s a tradeoff here, you can simulate a slower interrupt timer but responsiveness will go down (i.e. how quickly the application responds to keyboard input). You can experiment with the numbers and find the optimal balance.

If you were playing a DOS game, you would probably want it high, but for an office application like Agenda you can turn responsiveness waaaaay down and it’s still totally acceptable.

To do that, set the timer option. The default value is 18, but I find 8 to be totally acceptable, maybe even 6. You can only change the timer setting on the commandline, not in .dosemurc.

$ dosemu -term -I "timer 8" -E "C:\AGENDA\AGENDA.EXE"

The lower you can set this value, the lower CPU time agenda will use.

Q. I get an error like “Error: A.EXE not found”.
A. Agenda is not in your %PATH%, add a line like path %PATH%;C:\AGENDA to one of your initialization files, like fdppauto.bat.

Q. I’m using GNU screen, and some function keys produce garbage.
A. S-Lang tries to check if you’re using an xterm compatible terminal by doing strncmp($TERM, "xterm", 5);. GNU screen is xterm compatible, but it sets the terminal to screen.xterm-256color.

A workaround is to just make an alias for this termcap entry that starts with “xterm”. First, get a copy of the termcap entry, e.g.
```
$ TERM=screen.xterm-256color infocmp > termcap.txt
```
Now, add an alias at the top of the file that starts with xterm, e.g. I made mine look like this:
```
screen.xterm-256color|xterm-256color.screen|GNU Screen with xterm using 256 colors,
```
Now recompile it, tic termcap.txt. tic automatically installs that entry in your $HOME/.termcap directory, so it should be available immediately. Now add the line term xterm-256color.screen to your .screenrc, and the problem should be solved.

Q. I’m using GNU screen, it works but the Escape key seems slow, unreliable, or needs to be pressed twice?
A. First add the line maptimeout 50 to your screenrc, and see if that helps. If it feels better but not fixed, try tweaking the number.

If it didn’t help at all, then I think your termcap definition is wrong. Perhaps you are using a terminal that claims to be xterm compatible, but is not.

Try these steps:
1. Decompile your current termcap entry, $ infocmp -1 > termcap.txt
2. Find the line that starts kmous= and remove it, that’s mouse event reporting. I’ve found this unreliable in some vte-based terminals.
3. Recompile it, $ tic termcap.txt.
See if that fixes the problem. tic places compiled definitions in your $HOME/.termcap directory, you can remove them if you want to go back.

Q. How can I print something from Agenda?

A. Add the line $_lpt1 = "dosprint" to your .dosemurc, then create this shell script in your $PATH called dosprint, you can adjust to your liking. The ps2pdfwr utility is part of ghostscript.

  #!/bin/bash
  declare tmpfile=$(mktemp --tmpdir=${HOME}/Downloads --suffix=.pdf printjob.XXX)
  if ps2pdfwr - ${tmpfile} > /dev/null; then
      xdg-open "${tmpfile}"
  else
      rm -f ${tmpfile}
  fi

Q. Can I see a sample of Agenda generated PostScript and/or a sample PDF?
A. Sure, here is some PostScript, and here is a PDF.