Blog

In a post in my BG3 series, I observed that, for reasons detailed in that post, I’d taken over five thousand screenshots from Baldur’s Gate 3. Through a rather tedious process, those images were transferred to my Photos Library on Mac OS.

Let’s set aside the question of whether that number of photos was excessive. What I want to focus on is how Photos’ face-recognition algorithm handled that ensemble of pictures.

Your privacy

I have no idea of how Apple’s face-recognition algorithms work. In particular, I don’t know if the facial pattern-matching data is stored locally on my computer or if it’s being transmitted to Apple’s servers.

With that in mind, I make no effort to identify Real Humans in my Photos library using the techniques I describe below. I don’t want to add you to some kind of database that might be used against you in the future.

However, I’ve got no problems with adding fictional characters to any face-recognition database. If Astarion the Vampire Ascendant walks among us, I think we’d all be more comfortable if the FBI identified him through traffic cameras.

My images

When I skim my Photos library, I see have relatively few pictures of Real Humans anyway. According to the statistics in Photos, I’ve got 22,000+ images (and 200+) videos. I scrolled to some random location with my entire Photos library, and saw this:

Here are some tidbits about the contents of my Photos library:

• Already noted: 22,000+ images
• When I look at my “PS App” folder (by default, all images I import from my PS5 go there), it says it contains 6000+ photos. When I add up the total number of photos from individual playthroughs, I get 5000+. There must be about a thousand pictures I took on the PS5 that were not filed as part of a single BG3 playthrough.
• 4000+ photos are in an album labeled “Crafts”. This includes all the photos I made for my now-closed jewelry shop.
• 1400+ photos are cat pictures. This is as it should be.
• When I skim my Photos library for images of Real Humans, I see few new pictures since my Murder Mystery Masquerade in 2019. Whether this is normal or sad I leave to you.

It seems that the majority of my photos reflect my hobby interests at the time. The BG3 pictures are easy to take: just hold down the “Create” button on the controller. For my hobby photos, I took some effort with each picture, moving either the camera, the subject, or the lighting. I took more time with the jewelry photos, which is why there are fewer of them than video-game screen captures.

This makes me feel a bit better about the number of BG3 photos in my library.

Of course, those BG3 photos provide a interesting data sample for understanding face recognition.

Data collection

The photos you see in my previous BG3 posts are all transfers from my PS5.

All the photos you see in this post come from my taking partial screenshots on my desktop Mac, by pressing Command+Shift+4.

This matters because when I export an image from Mac OS Photos, or simply drag it into a blog or social-media post, the image you see isn’t the image I see. Consider the picture at the top of this post. I see “unnamed” because I never assigned a name to that character’s face. If I simply copied that image from Photos to this post, you would not have seen that text.

How well does Photos face recognition work?

I went through my BG3 images in Photos and began to supply names to faces. For example, if I saw this:

I would type the name “Gale” into that text field. The result was this:

It’s actually a bit more complicated than that. If I just kept typing “Gale” every time I saw “unnamed” under Gale’s face, what would happen is that Photos would assume that each of them was a different Gale.

What I discovered I had to do is to wait a bit after I identified a new face for the first time. After a while, when I started typing Gale’s name, by the time I got to “ga” there’d be a pop-up menu that would include the name “Gale” as an option. This meant that Photos now included a new category of Faces called “Gale”. If I clicked on that name in the pop-up menu, the photo would be added to the “Gale” collection.

By doing this repeatedly for a large number of photos, I’d build up a collection of names in the window I’d see when I selected Photo’s View->Photos->People & Pets menu:

Here’s another example. Photos didn’t know who Sorcerer Sabrina was:

Here it is after I filled it in:

The reason why you see a circle in those images is that, at the time I did the screen capture, I had the mouse hover over Sorcerer Sabrina’s nose. It shows the region of the image that Photos identified as being a face.

After a time (usually overnight), I’d begin to see Photos identify the face on its own. It became interesting when, even after I’d tagged a face dozens of times, Photos would still fail in a puzzling way:

The same thing happens with Wyll:

In my current playthrough, I’m playing as a custom character named Amaranth. Of course, Photos can’t identify her until I tag her in several photos. We start with something like this:

I filled in the name in that photo and others:

Then I came across pictures in which Photos couldn’t identify that a face was present, even though (to a Human) such an identification seems easy:

I’ve just started Amaranth’s playthrough and I have only a couple dozen taggable photos of her. But apparently that’s enough for Photos to mis-identify her:

For the record, this is Naoise Nallinto. You can judge how close her facial features are to Amaranth’s.

Another example of mis-identification. Nubaldin is not a significant character in the game, and I haven’t bothered to tag him in the few pictures I took of him. Barcus Wroot has an actual story arc, and I took and tagged many images of him. Yet still:

For the record, this is Barcus Wroot:

Similarly, I tagged both Minthara and Nym Orlith in many pictures. Photos reached the point where it could identify Minthara:

But it flubbed the identification of Nym Orlith:

Here’s the same thing with Astarion and Larroakan:

Photos learned to recognize Jaheira:

That recognition continued even when her face had a different perspective. But consider the rest of this photo:

In the above photo, Karlach’s face was recognized. But Shadowheart’s was not, even though I tagged her in roughly the same number of photos. “Tav” is unnamed because, at the time I saved this image, I made the choice not to tag her in Photos.

What does this mean?

Absolutely nothing!

I don’t know how Apple’s face-recognition is related to the face-recognition algorithms used by law-enforcement agencies. It might be the same, or not. The success rate of the law-enforcement software might be better than Apple’s, or not.

It has left me with skepticism on whether automated face recognition is a valid strategy in security and law enforcement. If there are these kinds of issues with the hyper-idealized images in a video game, what would the issues be for real-world faces? Whether this skepticism is justified or the result of my ignorance I leave to experts.

What I personally conclude is that law-enforcement agencies might be better served by playing Baldur’s Gate 3. It could produce better results than engaging in some of their morally-dubious tactics, even if Astarion the Vampire Ascendant walks among us. They might pick up the Wizard Lorroakan instead.