iOS thumbnails wrong orientation issue

Took me 2 days to get through this ticket so I think it might be worthy to mention it here :)

Symptom

QA reported that a report generated from an iOS device had its image thumnails in wrong orientation. See below for an example.

Image taken with the home button of an iPad pointing towards right:

Its thumbnail extracted from the EXIF information: (notice how it got upside down)

Attempts

Currently in our project, thumbnails are generated by calling an API that is maintained by another team. Since nobody on our team has access to the other project, we would prefer to solve this in our scope only.

Take 1: flip the original image

Because QA suggested that this issue only occurs when a user attaches an image taken from an iOS device that has its home button on the right, I thought that it was just an isolated case that can be solved by flipping the original image. Basically, if an image is taken by iOS device and has its orientation to be landscape, we would flip the image by 180 degree then pass to the external service call.

Result: thumbnails were indeed fixed… with the following byproduct: Every image, regardless of its original orientation, was rotated 180 degree if its manufacturer is Apple. All images were upside down when requesting a report with full-size attachments. Plus, images taken with the home button of an iPad pointing towards left have their thumbnails in the correct position. By fixing the “home button right” case this miserably failed.

Realizing that this can’t be solved from our end, I went ahead and requested access to modify source codes in the backend API project.

Take 2: flip the thumbnail in the API

Condition: if an image is in landscape and taken by an Apple device, we would flip the thumbnail after getting it through the GetThumbnailImage method. Image manufacturer could be obtained through the Image.GetPropertyItem() method, with Id = 271. (C# code sample is wrong here and I’ve opened an issue for that)

Result: thumbnails were fixed again; BUT I didn’t test all 4 possible orientations, so I messed up images taken with the home button of an iPad pointing towards left AGAIN.

Take 3: fix the original image EXIF information in the API

I kind of lost hopes at this point, because I found that images taken with the home button on the left/right BOTH have their orientation = landscape when reading the EXIF information. That way I can’t tell if an image and its thumbnail share the same orientation. But what I did know that this is an issue with iOS device only (ref). Checked this with my Android, thumbnails work just fine. Plus, thumbnails created through Image.GetThumbnailImage() do not always have the orientation EXIF informaton. Thankfully I found another solution here. The idea is to clear selected EXIF informaton (orientation of the Image (0x0112), orientation of the thumbnail (0x5029), and thumbnail bytes (0x501B)) of the original image first, then extracted its thumbnail. Note that image needs to be saved again once we processed its EXIF information. This post-processing is only required for images taken by Apple cameras.

Result: checked with images taken from iPad Air in all orientations, thumbnails were saved correctly.

Code snippet

C# code below demonstrates how to get a thumbnail of size 150x150 of an image in the correct orientation.

using System.Drawing;
using System.Drawing.Imaging;

string filepath = @"\Downloads\aright.jpg";
string thumbpath = @"\Downloads\thumb.jpg";

Bitmap img = new Bitmap(Image.FromFile(filepath));
int thumbnailHeight = 150; // desired thumbnail image size = 150 x 150
int thumbnailWidth = 150;
Image thumbnail = null;
Bitmap fixedImg = null;
private readonly int orientationId = 0x0112; //Image orientation
private readonly int thumbnailOrientationId = 0x5029; //Thumbnail orientation
private readonly int thumbnailBytes = 0x501B; //Thumbnail bytes

var encoding = new System.Text.ASCIIEncoding();
try
{
	string manufacturer = encoding.GetString(img.GetPropertyItem(271).Value);
	if (manufacturer.Contains("Apple"))
	{
		// clean up EXIF info for images taken by iOS devices.
		if (img.PropertyIdList.Contains(orientationId))
		{
			img.RemovePropertyItem(orientationId);
		}
		if (img.PropertyIdList.Contains(thumbnailOrientationId))
		{
			img.RemovePropertyItem(thumbnailOrientationId);
		}
		if (img.PropertyIdList.Contains(thumbnailBytes))
		{
			img.RemovePropertyItem(thumbnailBytes);
		}

		using (var imgStream = new MemoryStream())
		{
			img.Save(imgStream, ImageFormat.Jpeg);
			fixedImg = new Bitmap(Image.FromStream(imgStream));
		}
	}
}
catch (ArgumentException ex)
{
	// if the image has no property item, skips and proceeds to the next step.
}

if (img.Width > img.Height)
{
	thumbnail = (fixedImg ?? img).GetThumbnailImage(
		(int)Math.Round((double)((img.Size.Width * thumbnailHeight) / img.Size.Height)),
		thumbnailHeight,
		null,
		IntPtr.Zero);
}
else
{
	thumbnail = (fixedImg ?? img).GetThumbnailImage(
		thumbnailWidth,
		(int)Math.Round((double)((img.Size.Height * thumbnailWidth) / img.Size.Width)),
		null,
		IntPtr.Zero);
}

using (var original = new Bitmap(thumbnail))
{
	int factor = 0;
	if (thumbnail.Width > thumbnailWidth)
	{
		factor = (int)Math.Round((double)((thumbnail.Width - thumbnailWidth) / 2));
		thumbnail = original.Clone(new Rectangle(factor, 0, 150, 150), original.PixelFormat); // Get centered bitmap.
	}
	if (thumbnail.Height > thumbnailHeight)
	{
		factor = (int)Math.Round((double)((thumbnail.Height - thumbnailHeight) / 2));
		thumbnail = original.Clone(new Rectangle(0, factor, 150, 150), original.PixelFormat); // Get centered bitmap.
	}
}

thumbnail.Save(thumbpath);

After thoughts

At the beginning I thought it’s a bug in the backend API, but it turns out to be an issue ONLY for iOS devices. Images uploaded from desktop (Windows) and Android seem to be fine. A quick search online shows that people were already complaining about weird Apple thumbnails issue, and previously the code seems to just extract thumbnail directly from the EXIF of the original image. (So it’s an issue with Apple!) I was still unable to have iOS connected to the localhost server running on my Windows machine :( Plus it’s fun to play around with interactive c#… somehow. I could tell how I am spoiled by IPython haha. The sample code above was modified by the senior dev R. My original solution was not as succinct :P still a long way to go (but still way better than debugging data table column width with CSS, that’s driving me NUTS)