Tag Archives: binary

Getting into a 64 bits world: What Apple spotted and Microsoft missed

The user comes first

You might be a large corporation or a freelance developer, but it you’re in the software business you’re working for the end-user even if you don’t even know a single one of them. And let me tell you something about them: they’re stupid, but they hate to realize.

You want to make profit of your work, you have to make your user feel clever… or at least, don’t make him feel stupid or frustrated. A lot of users change powerful software for a crappy one that’s easy. I’m not saying technical quality is not important, I’m saying that if it two different applications can do the job, users will use the one which makes them feel clever by doing the job with less irritating error messages or cryptic questions.

Why so many users recommend using 32 bits?

Because they’re scared. They don’t know what 64 bits is apart from being the double of 32 bits… what they know is that some of their friends were using 64 bits and had to fallback to 32 because their applications weren’t working or the computer was unstable or… who knows what, but something wasn’t working.

But, is that true or is just a legend?

I’m afraid both. If we’re talking about windows, 32 bits applications will work on a 64 bits Windows without issues… but (there’s always a but) 16 bits applications will not work neither 32 bits drivers.

This means that if you’re running very old applications (MS-DOS and Windows 3.11 mainly) and you switch to 64 bits, you’re going to run into troubles. This is indeed a problem for business, but normal end-users do not rely on ancient software. However, they rely on something equally bad: cheap and crappy hardware.

Windows provides drivers for most common hardware, but it is likely that some bits of your system won’t be covered by the catalog windows offers. There is where hardware vendors step in providing you drivers for your operating system. A lot of 64-bits windows users found out, sadly, that their hardware vendors won’t provide 64 bits drivers (welcome to the Linux world, where your hardware vendor will tell you “I don’t care”).

32, 64 bits… who cares?

Let’s be honest. Users don’t care if your application is using 64, 32 bits or a giant wheel powered by a hamster. They want to do their job as fast as possible, knowing the minimum about the tool (ideally nothing) so they can move to important things (like watching big brother, or reading The Sun, but that’s another story).

However, as engineers we know that 64 bits is the right choice because we’re using more and more memory every day (chrome is actually eating memory) and relying on “tricks” like PAE is not a solution but a workaround.

The question is not if we should move to 64 bits, but how.

Universal Binaries: The Holy Grail

The solution to having to know about the architecture and choosing the right binary is: Universal binaries. Apple also used this solution when they changed their PowerPCs for Intel microprocessors.

The basic idea is to create a binary that would work on both, 32 and 64 bits by including both versions of the binary merged. Besides this, due to the particular way applications are packaged in MacOS X, applications have only one download for all architectures and all languages.

This is great for the user. It doesn’t matter if he’s using one or another architecture, in English or in Russian…. the same file is going to work smoothly and in the correct language.

The downside

Universal Binaries are the holy grail because the user doesn’t need to know anything. However, they achieve this by including binaries for all supported architectures and languages within the same distribution. Which means that the size of the application is going to be multiplied by the number of architectures supported and increased by the amount of languages included.

I’ve a 64 bits Intel Processor and I use everything in English. Why should I download application three times larger than expected (some of them have 64 and 32 bits versions along with a PowerPC version) and with support for German if I don’t have those processors and I can’t read German?

As an example, Google’s uploader for Google Music, called Music Manager, weights 45MB. After removing languages other than English and unused architectures (in this case, PowerPC support) the size is 11.7MB. In other words, I’ve an application that sizes nearly 4 times what it should.

Fortunately enough, there is software, like Xslimmer to strip out the unnecessary architecture and language support from our applications. But take this with caution, because this procedure could cause some updating methods to fail. Moreover, some applications won’t survive the process because realizing their size has changed, they refuse to work assuming they’re corrupted or tampered. You’re warned.

Upload any kind of file into Google Music: Google Data Uploader

A few days ago I posted a small tool to download songs from Google Music. The main reason was to allow me backup my data, but let’s be honest, I love to do this kind of stuff, and I wasn’t going to stop just there.

Google Music allows you to store up to 7.000 songs. That’s it. No size limit, just 7.000 songs. Wouldn’t it be great if, somehow, you could upload a regular files instead of just songs? Think about it, 7.000 files, of any size, stored for free!

Today I’m releasing another tool that makes regular files appear as songs to Google Music.


How does it work?

To put it simple, files have two main sections, header and content. The header is a bunch of metadata that describes what kind of information the file contains (music, video, document…). This tool takes a real mp3 and uses its header, along with a little bit of its music, and then appends the file you want to store, making Google Music believe it’s a real music file.

Doesn’t it make my file bigger?

Yes, it does. Exactly 100kb more.

But, if the file seems to be an mp3, how the hell am I gonna use it afterwards?

Do not worry, this tool is also capable of “un-hiding” the mp3, so you’ll get the original file 😉

Will I have my data files and music files mixed in Google Music?

Yes, and no. You will see all your uploaded data files inside the album “Data” of the artist “Google Data Upload”. You will be able to create playlist that mix data a real music files… but that will be a really weird thing to do, won’t it?

How do I use the tool?

It’s a command line tool. If you know nothing about command line you can either learn it or wait until I create a graphical tool to do it… which might be available tomorrow or never.

To hide a file: GoogleMusicDataUploader.exe –merge <real_mp3_file> <file>

To reveal a file: GoogleMusicDataUploader.exe –unmerge <hidden_file>

How do I upload the hidden file?

Use the same method you’ve used with the rest of your real mp3 files. Google Music will not notice the difference and will upload it 😉

Why are you developing this?

Because I want to. Some people have fun solving puzzles… I do this.

Where is the source code?

You can find the code at my GitHub space.

I want to know more about the technical details.

Fair enough. Let’s start saying that my first attempt was to make use of the ID3Tags by injecting the desired file as the cover of the mp3. I’ve used that trick in different occasions to defeat some monitoring tools, however Google Music looks to this kind of metadata and attempts to resize the cover. I’m guessing they try to do this to save space. Obviously it can’t resize a binary file, so instead of ignoring it, it deletes the metadata it does not understand and thereby making this trick useless to my end.

The current implementation relies on the fact that file headers can be stored at the beginning of the file (that’s why it’s a header) but also at the end of it. The vast majority of them have their headers at the beginning, but some of them, like zip files, have it at their end.

Google Data Uploader takes a regular mp3 file and takes it’s header and some of its content (the first 100kb) and then it adds the target file compressed in zip format. This makes the file structure look like:

The tool also adds some information to the ID3Tags. To be more accurate, it sets the Album to “Data”, the artist to “Google Data Uploader” and the title to whatever the name of the file was. This makes it easy for you to find your files within Google Music.

Google Music expects to find an MP3 file, so starts looking at the beginning of the file. It finds a valid MP3 header, and therefore decides to upload the whole thing.

Once the file is in Google Music, you can “play it”. It will go for a few seconds and then suddenly ends as soon as it runs out of song and reaches the End of file (which is in fact the end of file of the zip file).

If you download the file (using the tool I posted before) you’ll find yourself with an mp3 slightly different from the one you’ve uploaded. Google Music sends you the file without the ID3Tags. That’s not a problem cause we don’t need it anymore, all the information we care about from now on is inside the zip file which remains unaffected.

You can use this tool (–unmerge option) to “split” the zip file from the mp3 and extract all its files or you can just rename the file to “.zip” and open it with a zip program that doesn’t care about malformed files (like 7-Zip). This works because, under the assumption that the file is a ZIP, the program should start reading from its end, finding a valid header, continuing with the content and reaching the “End of file” preventing it from access the mp3 data.

So you’re just taking advantage of a naïve implementation of Google’s uploader, aren’t you?

Definitely. This is an extremely easy trick that works just because Google is not checking the content of the whole file assuming that if the header says the file is an mp3, it has to be an mp3.