Step 2: MKV to MP4
In this step, we'll convert our MKV file to MP4. We don't need to re-encode the file, as we'll basically extract the H.264 and AAC tracks from the MKV file using MKVExtractGUI and then multiplex the two streams into a MP4 container.
MKVExtractGUI requires MKVToolnix to be installed first, so go and download and install Windows version of MKVToolnix. This should, by default, install MKVToolnix to "C:\Program Files\MKVtoolnix". Now download the MKVExtractGUI ZIP package and extract it to the directory where MKVToolnix is installed. This should create a "MKVExtractGUI" sub-folder under the main MKVToolnix folder, with 4 files. Move these 4 files down to the main MKVToolnix directory, so that MKVextractGUI.exe and mkvextract.exe are in the same folder.
Start MKVExtractGUI by opening the executable (MKVextractGUI.exe).
Press the "..." button next to the "Input" text box to load in your input MKV file. The "Content" text box should now show the two tracks present in the MKV file (what will be listed as "V_MPEG4/ISO/AVC" and "A_AAC"). Check the two checkboxes for the two streams and press the "Extract" button to extract these two streams as separate files. The stream files will be saved in the same folder as your input file by default if you haven't changed the output location. You can close MKVExtractGUI when the extraction process is finished.
You should now have two extra files, a .h264 file and a .aac file.
Now we'll put these two streams into a MP4 container. There are many tools to do this, but since I've already written some brief instructions for YAMB, I'll use this tool. The instructions are here, just follow the "MP4 Multiplexing" instructions in the middle of the page. Note with .h264 files, you'll have to manually enter the framerate. This should be the same value as your original input file - you can find the framerate of video files using various tools, such as MediaInfo. Once YAMB finishes, you should now have a MP4 file.
This file will be ready to play on the Xbox 360 and PS3 without further modification. To play the MP4 file on your Xbox 360, please follow the instructions on this page. The PS3 equivalent of the instructions can be found here. And we're finished!